How to improve AI quality
How to improve AI quality Iterative tuning: prompts, A/B, model selection, measurements. The “observe → hypothesis → adjustment → measurement” cycle is the only way to avoid turning tuning into random
How to improve AI quality
Iterative tuning: prompts, A/B, model selection, measurements. The “observe → hypothesis → adjustment → measurement” cycle is the only way to avoid turning tuning into random trial and error.
If AI is already enabled but responses are unstable or expensive, random edits without metrics usually make the situation worse.
Where to look for problems
Messenger — real conversations
Look for:
- where AI replies off topic.
- where the user gets confused.
- where an operator has to intervene.
- where the bot gets stuck in a loop.
- where AI writes too much or too formally.
Steps Conversion — drops in the flow
- at which
AI Replysteps leads stop responding.
- where conversion drops sharply.
- after which Routings leads are more likely to exit the scenario.
- If one AI step causes a significant drop, the problem is usually either in the Goal / Requirements or in the AI response text itself.
Billing — cost of AI
- cost per conversation.
- increase in expenses after changes.
- which models consume the most funds.
- whether there are any unnecessary AI calls.
Symptom → where to look
Symptom | Where the problem usually is |
|---|---|
AI makes up features, prices, guarantees | Restrictions, Reply Addition Info, Goal |
AI goes down the wrong branch | AI Routings or Goal |
Responses are too dry / too long | Response Style, creativity, reply model |
AI ignores the step context | Reply Addition Info |
Responses are in the wrong language | Response Style |
Too expensive | Expensive model, long prompts, unnecessary AI calls |
Tuning cycle
Observe
Review real conversations in Messenger. Find a specific symptom: hallucinations, wrong language, poor transition.Formulate a hypothesis
For example: “AI ignores the Goal — the wording needs to be clarified” or “Routings are too similar — they need to be differentiated.”Make one change
Change only one parameter per iteration: the prompt, model, or Routing settings. Otherwise, it is unclear what helped.Measure the result
After making changes: test in Demo space, wait for real conversations, compare metrics before/after.
When to change the model
Upgrade the router model
If:
- • AI Routings — Deep Dive make mistakes.
- the flow goes to the wrong branches.
Upgrade the reply model
If:
- the transition logic is already good.
- but the responses themselves look weak.
- AI sounds too robotic.
- responses do a poor job of keeping the conversation going.
Following instructions / creativity
The slider affects response variability:
- closer to 0 → responses are stricter and more predictable.
- closer to 1 → responses are more varied and free-form.
- For most production scenarios, medium values around 0.4–0.6 are typically used.
Which metrics to monitor
Escalations to an agent
How many conversations the bot closes without a human.AI step conversion
Check via Steps Conversion: the share of users who completed the Goal.Cost
Check in Billing and Resource usage.Quality signals
It is useful to take external signals into account: team responses and feedback from leads.
️ Important: There are no universal “correct percentages” for AI. Good values depend heavily on:
- niche.
- traffic quality.
- flow length.
- offer complexity.
- models used.
- Evaluate improvements against your own past data, not someone else’s numbers.
Frequently asked questions
Related
Was this article helpful?