E-commerce
June 28, 2026
A bot dashboard shows 68% deflection and a CSAT of 4.2. Then an agent reviews 30 conversations: incorrect return policy on 4 threads, two invented promo promises, one failed escalation on a dispute. The aggregated numbers were hiding high-impact errors.
The e-commerce chatbot audit is the systematic review of real conversations, scored on a grid, to detect what volume KPIs miss. FutureAGI reminds us in 2026 that evaluating a bot means six distinct problems (intent, retrieval, generation, tools, multi-turn, security), not a single score (FutureAGI, chatbot evaluation 2026).
This guide #143 outlines an operational audit method for DTC support teams. Distinct from response quality (#116) (continuous KPIs), hallucinations (#123) (technical prevention), and governance (#142) (rules and RACI): here is how to concretely audit every week.
Summary
Why audit chatbot responses if the KPIs seem good?
The chatbot response audit complements dashboard metrics with a human (or assisted) reading of real threads.
What KPIs hide
Deflection: avoided ticket ≠ resolved issue
Average CSAT: masks toxic intents (return, dispute)
Unmatched rate: does not measure false matched responses
Model confidence: internal score ≠ business accuracy
Business risk
A policy error on 2% of WISMO conversations, multiplied by 800 threads/month, generates returns, recontacts, and chargebacks. HelloRep recommends auditing conversations, not just dashboards: 20 to 30 threads/week minimum (HelloRep, measuring e-commerce bot accuracy).
DTC Example
Accessories brand, bot live 4 months, CSAT 4.3. Monthly audit of 50 threads: 7% error on international delivery times, 3% false promo post-Black Friday. Corpus correction + killer promo intent: WISMO recontact -22% in 6 weeks.
How does auditing differ from governance and quality measurement?
Four related disciplines, four different rhythms.
Governance (#142)
Governance (#142): permanent framework (RACI, say/do matrix, kill switch). The audit verifies that this framework is respected in production.
Response Quality (#116)
Quality (#116): continuous precision, tone, FCR, and CSAT KPIs. The audit = in-depth bot sampling that feeds these KPIs. Integrate into the weekly QA review (#277) with agent tickets.
Hallucinations (#123)
Hallucinations (#123): RAG guardrails, corpus, confidence. The audit detects residual hallucinations after guardrails are deployed.
Audit-Test-Govern Cycle
Alhena offers three layers: weekly audit of live threads, regression tests stemming from failures, and governance for ownership (Alhena, e-commerce chatbot QA). This guide details the audit layer.
Which six dimensions should be scored during a conversation audit?
Scoring separately prevents good fluidity from masking a false policy.
1. Capture intention
Did the bot understand the request? Confusion between return vs. exchange, WISMO vs. late delivery = failure even if the response is polite.
2. Retrieval (sources)
Are the RAG chunks relevant? Return policy retrieved for a return question? Promo chunk for a promo question? Check source logs if available.
3. Generation (accuracy)
Is the response grounded in the sources? Invented claim (delay, stock, ingredient)? Core of the factual audit.
4. Tool usage
Tracking URL correct? Return portal sent? Cancel order attempted even though fulfillment has passed? FutureAGI isolates tool use as a distinct layer in CI and production.
5. Multi-turn consistency
Does the bot retain context (order number, size) over 3+ messages? Does it contradict its previous response?
6. Security and refusal
Refusal calibrated for health, legal, prompt injection? Dispute escalation triggered? See chatbot limits (#124).
How do you build a weighted audit grid?
A standard chatbot audit grid standardizes scoring among auditors.
Criteria and weights (inspired by Alhena)
Factual accuracy (40%): policy, price, stock, lead time
Relevance (25%): answers the question asked
Brand voice (20%): tone, empathy, clarity
Completeness (15%): no partial answer that forces follow-up
1-5 scale per criterion
5 = perfect, 3 = acceptable with minor correction, 1 = serious error or risk of dispute. Conversation score = weighted average. Alert threshold: < 3.5 on accuracy or global score < 3.8.
Mandatory error tags
`policy_stale`, `wrong_intent`, `hallucination`, `bad_escalation`, `tone_off_brand`, `incomplete`, `tool_fail`. Each tag feeds the corrective action backlog. See voix marque bot.
Google Sheet Template
Columns: date | conversation_id | intent | accuracy score | relevance score | tone score | completeness score | overall score | tags | corrective action assigned | owner | deadline.
How do you sample the conversations to be audited?
A representative bot audit sample is better than 500 poorly chosen random threads.
Weekly DTC Volume
< 200 conv/month: 15 threads/week (60% of audited monthly volume)
200-800 conv/month: 25-40 threads/week
800+ conv/month: 50-80 threads/week + stratification
Stratification (mandatory)
Do not audit only easy WISMO. Distribute as follows:
40% top volume intents (WISMO, return, size)
20% high-risk intents (promo, regulated, dispute)
20% unmatched or low confidence
10% bot-to-human escalations
10% CSAT 1-2 or recontact within 48h
Gorgias filters / export
Tag `bot_handled`, 7-day date range, CSV transcript export. Include 5 threads with agent post-correction (bot override) to measure drift.
How to conduct a 60-minute audit session?
Ritualizing the weekly chatbot audit prevents rescheduling "when we have time".
Before the session (15 min prep)
Export stratified sample (section 5)
Open sources of truth: policy site, 2 test PDPs, promo calendar
Duplicate Sheet template row per conversation
During (45 min)
Auditor (support lead or senior) reads complete transcript. For each thread: score 4 criteria, tag error, write corrective note in 1 sentence ("sync chunk return 30 d Markets FR"). Do not fix the bot live during the audit: take notes first, process after.
After (15 min debrief)
Top 3 errors by frequency or severity
1 priority corpus fix (owner + 48 h deadline)
1 case added to the regression suite (section 8)
Log Slack summary #bot-audit
Rotate auditor every 4 weeks to reduce bias. See maintenance KB (#140).
How to complement human audit with LLM-as-a-judge?
The LLM-as-judge scales pre-triage, not the final verdict alone.
Pragmatic SME Use
A judge model (Claude, GPT) receives the transcript + pasted policy + a 1-5 rubric. Output: scores + justification + error flag. A human validates 100% of scores < 4 and 10% of scores ≥ 4 (auditor quality control).
Documented Limits
A 2026 conversational commerce study shows that the judge-conversion correlation varies by dimension: the judge sometimes overestimates surface fluency over policy accuracy (arXiv, validity of LLM-as-judge commerce 2026). Never replace human auditing on regulated intents or disputes.
Sample Judge Prompt
"You are a QA support auditor for e-commerce [Brand]. Policy: [pasted]. Accuracy score 1-5. Quote the incorrect bot sentence. If a claim is unverifiable in the policy/PDP, score 1." Temperature 0. Always log prompt + output for traceability (aligned with governance #142).
How to turn audit failures into regression tests?
Each audited error becomes a bot regression case to prevent recurrence.
Test Suite Structure
100-150 maintained cases (Alhena). Distribution:
25-30 product accuracy, price, shipping, returns
20 brand voice (calm vs. angry customer)
20 policy adherence (promo, accumulation, regulated)
10-15 escalation and context handoff
15-20 adversarial (injection, data extraction)
15-20 regression of fixed bugs
Bug-to-Test Workflow
Audited error tagged `policy_stale` on Germany Markets returns
Fix Germany returns corpus chunk
Add case: "Can I return in Germany after 25 days?" + expected response
Re-run full test matrix (#283) before intent go-live or post-deploy Friday
Test owner = bot admin. Quarterly review. See clean corpus (#103), prioritize automation (#120).
Which KPIs should be drawn from the audit, and where should they be published?
The audit feeds a separate bot quality dashboard that is independent of volume.
Primary KPIs (monthly)
Factual error rate: accuracy score threads < 3 / total audited. Target < 3%
Average overall score: target 4.2+ / 5
Hallucination rate: `hallucination` tag / audited total. Target < 1%
Incorrect escalation rate: unresolved dispute not escalated or useless escalation
Corrective delay: median hours audit → live corpus fix
Reporting segmentation
By intent, channel (widget vs WhatsApp), Markets language, corpus version (date chunk). A stable overall KPI can hide post-campaign promo intent degradation.
Link to business KPIs
Correlate month M: audit error rate vs 7-day recontact and "incorrect info" feedback. Completes chatbot KPIs (#11) and quality (#116).
Which audit mistakes are costly for support teams?
Avoid these audit bot anti-patterns that produce false comfort.
Auditing only easy threads
WISMO with clear tracking = systematic 5 score. Without risk stratification, you are auditing the bot on flat ground.
Single "quality" score
Charming bot + wrong policy = 4/5 global. Always alert if accuracy < 3 even if tone = 5.
Audit without tracked fix
Sheet filled, nobody fixes corpus. Rule: zero session without 1 owner + deadline.
Ignoring agent overrides
Agent corrects bot 40% of the time on intent size = priority audit signal, not operational noise.
One-off pre-launch audit only
Catalog, promos, and policies change. Audit = weekly ritual, not a one-shot project. See automation errors, DTC playbook.
How does Qstomy facilitate the auditing of chatbot responses?
Qstomy exposes the data that a rigorous audit needs, without laborious manual exports.
Audit features
Export transcript + sources: RAG chunks cited per answer
Confidence score: filter low-confidence sample
Intent tags: WISMO, return, promo stratification
Log override: agent corrects bot, traceable
Escalation filter: handoff threads for quality review
Quantified DTC Scenario
Skincare brand with 520 bot conv/month, quarterly ad hoc audit. Transition to Qstomy routine: 35 stratified threads/week, sources export, Sheet grid. After 4 months: factual error rate 6.8% → 2.1%, policy hallucinations -74%, average corrective delay 72h → 18h, bot CSAT 4.1 → 4.5. Regression set of 112 cases, re-run before each corpus update.
Explore AI support, Shopify, request a demo.
Which operational playbooks should be used to launch the bot audit?
Playbook 1: Sheet grid (1 h)
Create a tab with section 4 columns. Add the weighted global score formula. Share in #bot-audit. Deadline: 1 h.
Playbook 2: first sample of 25 threads
7-day export: 10 WISMO, 5 returns, 5 low confidence, 3 escalations, 2 low CSAT. 60-min audit session. Top 3 errors + 1 corpus fix. Deadline: 2 h.
Playbook 3: regression case since P2 error
Black Friday promo error detected. Chunk fix. Add test case + re-run 20 promo cases. Document in governance registry. Deadline: 45 min.
Playbook 4: post-launch intent audit
New "size exchange" intent: shadow for 2 weeks then 15 audit threads/week for 1 month. Go-scale threshold: factual error < 2% over 60 threads.
Playbook 5: monthly management review
1 slide: audit KPIs, 2 examples of corrected errors, 1 open risk. 15-min committee meeting. Governance link #142.
Useful linking
Auditing is not about doubting the bot: it is about giving it an improvement loop as demanding as the one for your human agents.
Enzo
June 28, 2026
