AI Agents
Module 07Lesson 4

Lesson 4. Testing and Improving Prompts

Hands-on: Zapier

Lesson 4. Testing and Improving Prompts#

Why This Matters#

The first version of a prompt is almost never perfect. Testing and iteration help improve quality.

Key Idea#

Prompt = living document

Test → analyze → improve → repeat.

How to Test a Prompt#

Step 1: Create a set of test queries

Query categories:

  1. Simple questions (in the knowledge base)

    • "How much does a haircut cost?"
    • "What are your hours?"

  2. Questions without answers (not in the base)

    • "When do you have discounts?" (if there are no promotions)
    • "How much does a massage cost?" (if you don't offer it)

  3. Vague requests

    • "I want to come in" (didn't specify for what)
    • "Book me for tomorrow" (didn't specify time)

  4. Emotional requests

    • "I'm unhappy with the quality!"
    • "You're scammers!"

  5. Actions

    • "I want to book a manicure"
    • "Cancel my appointment"

  6. Off-topic questions

    • "How do you make borscht?" (if it's a beauty salon bot)
    • "Who will win the election?" (if it's a store bot)

Step 2: Run queries through the agent

Send each query to the agent and record the responses.

Step 3: Evaluate the responses

Evaluation criteria:

✅ Good:

  • Response is correct and complete
  • Tone is appropriate
  • Didn't make up data
  • Asked for clarification when needed
  • Suggested the next step

❌ Bad:

  • Response is incorrect or incomplete
  • Rude or overly familiar tone
  • Hallucination (made up information)
  • Didn't ask for clarification when needed
  • Didn't use the tool (e.g., didn't record data)

Step 4: Improve the prompt

Common problems and solutions:

Problem 1: Agent makes up information

Solution:
Strengthen the limitation:

Rules:
- Use ONLY information from the knowledge base
- If the answer isn't there, say: "I didn't find that information, I'll check with the administrator"
- Do NOT make up prices, services, or promotions

Problem 2: Agent is too verbose

Solution:
Add a format rule:

Format:
- Keep answers brief: 2–3 sentences (maximum)
- If more information is needed, split into several messages

Problem 3: Agent doesn't ask clarifying questions

Solution:
Strengthen the clarification rule:

Rules:
- If the customer didn't specify [name/phone/date/time], be SURE to ask
- Don't move to the next step until all data is collected

Problem 4: Agent doesn't use the tool (e.g., doesn't record in Google Sheets)

Solution:
Add an explicit instruction:

Actions:
- After collecting all data (name, phone, service, date, time), use the Google Sheets tool to record it
- Tell the customer: "Done! I've booked you."

Problem 5: Agent is rude or argues

Solution:
Add a rule for emotional situations:

Rules:
- If the customer is unhappy or rude, stay polite and empathetic
- Don't argue
- Offer: "I understand. I'll pass your question to the manager, they'll reach out soon."

Step 5: Repeat testing

After edits, run the test queries again. If quality improved — the prompt is ready. If not — continue iterating.

Prompt Testing Checklist#

  • Simple questions — does it answer correctly?
  • Questions without answers — does it say "I don't know" or hand off to a specialist?
  • Vague requests — does it ask for clarification?
  • Emotional requests — is it polite and empathetic?
  • Actions (booking, sending) — does it perform them?
  • Off-topic questions — does it decline or redirect?
  • Doesn't make up data?
  • Tone appropriate?
  • Response format convenient?
  • Uses tools when needed?