Lesson 5. Errors and Protection

Lesson 5. Errors and Protection#

Why This Matters#

AI makes mistakes. That's normal. It's important to know where errors are typical and how to prevent them.

Key Idea#

Best protection = proper setup + control + Plan B

Errors are inevitable, but they can be minimized.

Types of Errors#

1. Hallucinations (Making Up Facts)

What: the model generates plausible but incorrect information.

Example: Question: "When is your sale?"
Answer: "March 15–20, 30% off" ← made it up

How to avoid:

• provide a knowledge base
• rule: "If you don't know — say so honestly"
• verify answers

Prompt: "If the information isn't in the knowledge base, reply: 'I couldn't find that information, I'll check with the manager'"

2. Misunderstanding the Request

What: the client writes unclearly, the model interprets incorrectly.

Example: Client: "Book me for tomorrow"
Agent: "Done, booked!" ← but didn't ask for the time

How to avoid:

• teach the agent to clarify
• show examples

Prompt: "If the client didn't specify time, always ask: 'What time?'"

3. Sensitive Data

What: the agent accidentally "remembers" and shows someone else's data.

Example: Client 1: "My order #123, phone +1-999-111-22-33"
Client 2: "What's the phone for order #123?"
Agent: "+1-999-111-22-33" ← data leak

How to avoid:

• don't store sensitive data in context
• use anonymization
• set rules: "Don't disclose personal data"

4. Inappropriate Responses

What: the agent is rude, jokes inappropriately, or ignores emotions.

Example: Client: "I'm unhappy with your delivery!"
Agent: "Sorry, but that's not my problem" ← bad

How to avoid:

• set tone in the prompt: "Always polite and empathetic"
• show examples of correct responses
• add rule: "If the client is upset, apologize and hand off to the manager"

Protection Rules#

1. Constraints in the Prompt

Rules:
- Answer only based on the knowledge base
- If you don't know — say: "I'll check with the manager"
- Don't make up prices, dates, or terms
- Don't disclose others' data
- If the client is aggressive, politely hand off to the manager

2. Test Set of Requests

Create 10–20 typical requests and check how the agent responds.

Examples:

• "How much is delivery?"
• "When is your sale?" (if there's no sale)
• "Book me for tomorrow" (no time)
• "I want a refund, you're scammers!" (emotional)

Criteria:

• is the answer correct?
• is the tone polite?
• did it make up data?
• did it clarify when needed?

3. Monitoring and Logs

Save all agent conversations. Once a week:

• review errors
• look for patterns
• improve the prompt and knowledge base

4. Plan B (Escalation)

There should always be a "contact a human" option.

When to hand off:

• agent didn't understand the request
• client is unhappy
• task is complex

Example: "Sorry, I can't answer your question precisely. Connecting you to a manager."

Security Checklist#

• Prompt contains constraint rules
• Knowledge base exists (agent doesn't rely on "general knowledge")
• Tested on 10+ requests
• "Contact a human" option exists
• All conversations are logged
• Sensitive data isn't shared without protection

Check Your Understanding#

1. What types of errors do agents have?
2. What is a hallucination and how do you avoid it?
3. Why is a test set of requests needed?
4. What is Plan B and when is it needed?

Module Practice#

Assignment 1: Three Versions of a Prompt#

Take one task (e.g., "answer a client's question about delivery") and write three versions of a prompt:

1. Basic (minimum information)
2. Medium (added context)
3. Full (role + task + data + format + rules)

Compare how the quality changes.

Assignment 2: Prompt Template for Your Task#

Choose a task from your niche (from Module 1) and write a full prompt using the template:

• role
• task
• context/data
• rules
• response format

Assignment 3: Test Set of Requests#

Create 5–10 typical requests for your agent. Include:

• simple questions
• questions where no data exists
• emotional requests
• ambiguous requests

Example:

1. "How much is a haircut?"
2. "When do you have discounts?" (if there are none)
3. "You're scammers, give me my money back!"
4. "Book me for tomorrow" (no time)

Review Questions#

1. What is context and why is it needed?
2. Why does the model "confidently" make mistakes?
3. What in the prompt affects quality?
4. How does memory differ from a knowledge base?
5. Why are tools needed if the model "already answers"?
6. What data should not be given to the model?
7. Why is a test set mandatory?
8. What is "response constraint"?

Answers:

1. Context is all the information the model "sees" when answering; needed for a correct answer
2. The model predicts words by patterns, can "make things up" if there's no data
3. Clarity of role, task, data, rules, and format
4. Memory is short-term (current conversation), knowledge base is long-term (documents)
5. Tools let the agent act (record, send) rather than just answer
6. Personal data, passwords, trade secrets without protection
7. To find errors before launch
8. A rule for the model: "what can and can't be said"

Module Summary#

You've learned:

• how AI works: context, tokens, prediction
• how to write effective prompts (role + task + data + format)
• what a knowledge base is and why it's needed
• how tools make an agent useful
• what errors occur and how to prevent them

Next step: Module 3 — overview of 2026 tools for earning and daily life.

Test Your Knowledge#

Quiz: LLM Basics#

Flashcards for Memorization#

Site Materials#