AI Agents for Non-Techies

Lesson 6. Quality: How to Build an Agent Users Like#

Goal: understand what makes an agent "good" from the user's perspective.

Agent Quality Criteria#

1. Usefulness

The agent solves a real user problem.

Bad:
"Bot greets the user but does nothing else."

Good:
"Bot helps find products, place orders, answer questions."

2. Speed

The agent responds quickly (under 5 seconds for text).

Bad:
User waits 30 seconds → leaves.

Good:
Agent responds in 2–3 seconds → user is satisfied.

3. Clarity

The agent speaks the user's language, without jargon.

Bad:
"Error 500 occurred. Stack trace: ..."

Good:
"Sorry, the service is temporarily unavailable. Please try again in 5 minutes."

4. Reliability

The agent doesn't crash, lose data, or throw errors.

Bad:
Every other request ends in error.

Good:
99% of requests succeed.

5. Personalization

The agent uses the user's name, remembers context, suggests relevant options.

Bad:
"Hello, user. How can I help?"

Good:
"Hello, Ivan! You asked about returns. Here's the guide."

How to Measure Quality#

Quality metrics:

CSAT (Customer Satisfaction Score) — user satisfaction

Question: "How helpful was the agent? (1 to 5)"

Target: average score > 4.0
Resolution Rate — percentage of requests resolved by the agent

Formula: (Resolved requests / Total requests) × 100%

Target: > 80%
Escalation Rate — percentage of requests handed to an operator

Formula: (Handed to operator / Total requests) × 100%

Target: < 20%
Average Handling Time — average time to process a request

Target: < 5 seconds

How to Improve Quality#

1. Collect feedback

After the conversation, ask:

"How helpful was the bot? (1–5 stars)"
"What could be improved?" (text field)

Implementation:

at the end of the dialog send: "Rate my work: ⭐️ ⭐️⭐️ ⭐️⭐️⭐️ ⭐️⭐️⭐️⭐️ ⭐️⭐️⭐️⭐️⭐️"
save the rating in Google Sheets
check average CSAT weekly

2. Analyze requests the agent didn't understand

log all requests where the agent replied "I didn't understand"
review these requests weekly
add new scenarios / questions to the knowledge base

3. Optimize speed

use caching (when data doesn't change often)
use faster models (GPT-4o mini instead of GPT-5.2, Gemini 3 Flash or other fast models)
remove unnecessary steps in the workflow

4. Improve prompts

if the agent gives irrelevant answers → improve the prompt (add examples, clarify the role)
test different prompts and pick the best one

5. Add personalization

save the user's name (ask at the start of the dialog)
save request history (when the user returns → show context)
suggest relevant actions (based on previous requests)

Module Practice#

Practice Task 1: Create a Security Checklist#

Task: create a checklist for verifying your agent's security.

Requirements:

use the template from Lesson 1 (Personal Data Security)
add items for logging, monitoring, fault tolerance
adapt to your project (remove irrelevant items, add project-specific ones)

Format: Google Docs, Notion, or PDF.

Checklist:

Practice Task 2: Set Up Logging for Your Agent#

Task: add logging to your agent (from previous modules).

Requirements:

create a "Logs" table in Google Sheets or Airtable
add columns: Timestamp, UserID, Action, Details, Status, ErrorMessage
configure the workflow so every important action is logged

Checklist:

"Logs" table created
Columns added
Incoming request logging configured
Agent action logging configured
Error logging configured
Test: sent a request → record appeared in the table

Practice Task 3: Set Up Monitoring#

Task: configure monitoring for your agent.

Option 1 (simple):

use built-in platform notifications (Zapier Email Alerts, n8n Error Workflow)
configure Telegram notification on error

Option 2 (advanced):

use UptimeRobot to monitor webhook availability
configure a workflow to check error count per hour (if errors > 10 → notification)

Checklist:

Monitoring configured
Telegram / Email notifications work
Test: created an artificial error → received notification

Bonus Task: n8n Practice — Error Workflow and Monitoring#

Task:
Create an n8n Error Workflow that tracks errors and sends notifications in all your workflows.

What to do:

Create an Error Workflow in n8n:
- Open n8n
- Create a new workflow: "Error Handler"
- Add an Error Trigger node
Add error handling:
- Node 1: Error Trigger (fires on any error)
- Node 2: Set (extract error data):
  - workflow_name: {{ $json.workflow.name }}
  - error_message: {{ $json.error.message }}
  - timestamp: {{ $now }}
- Node 3: IF (check error type):
  - If error contains "Rate limit" → branch A (wait and retry)
  - Else → branch B (notify admin)
Branch A: Retry on Rate Limit
- Node: Function (wait 60 seconds)
- Node: Webhook (send signal for retry if possible)
Branch B: Admin notification
- Node: Telegram → Send Message
- Chat ID: your ID
- Text: `⚠️ Error in workflow "{{ $json.workflow_name }}"

Description: {{ $json.error_message }}

Time: {{ $json.timestamp }}`

Add logging to Google Sheets:
- Node: Google Sheets → Append
- Table: "Error Logs"
- Data: Timestamp, Workflow Name, Error Message
Save and activate:
- Save → Active ON
Test:
- Create a test workflow with an artificial error (e.g., HTTP Request to a non-existent URL)
- Run the workflow → error should occur
- Verify: did you get a Telegram notification? Did a record appear in Google Sheets?