Supplier Bot V1 — Benchmark & Readiness

I. What You’re Looking At

This report contains full conversation transcripts from our supplier bot (V7 prompt) talking to synthetic 1688 suppliers. We also include excerpts from Sourcy’s current bot talking to real suppliers (Nelson’s archive, Mar 10–11). The same 9-dimension rubric scores both. Read the conversations first, form your own judgment, then check the scores.

II. Sourcy’s Bot — Real Conversation (Conv 1971, Futsal Shoes)

III. Eric’s Bot — Synthetic Conversation A (V5 prompt, Kimi K2.5, Shoes, 温州正步 — suspicious supplier)

IV. Eric’s Bot — Synthetic Conversation B (V7 prompt, Claude Sonnet, Shoes, 温州正步 — same suspicious supplier)

V. Scoring Rubric — What Each Dimension Means

Partial (0.5) = between Pass and Fail. Score = sum of 9 core dimensions, max 9.

VI. Scores — Side by Side

Both bots scored on the same rubric. Sourcy’s scores are from real supplier conversations (Nelson’s archive, 6 conversations). Eric’s scores are from synthetic supplier conversations (calibrated to 88% resemblance with real transcripts, see Section VIII).

VII. Integration Status

• ✅ Eval framework — 9 dimensions, automated scoring, detailed rubric
• ✅ Tuning harness — 60 supplier profiles, 3 product categories, automated test runs
• ✅ Supplier simulation — 88% resemblance to real transcripts (33 behavioral markers tested)
• ✅ Media handling — [图片], [语音消息], [系统消息] all handled
• ✅ Bot prompt (V7) — tested across 220+ conversations, 7 iterations
• ✅ chatServer API — unblocked (Tek confirmed Mar 12, profileName no longer required)
• ✅ Claude API key — provisioned, $10 preloaded

• ⬜ Wire conversation loop to chatServer (send → read reply → send next)
• ⬜ Run V7 against real 1688 suppliers via chatServer
• ⬜ Shen’s blind A/B test
• ⬜ Claude API integration into pipeline (swap Kimi → Claude)
• ⬜ Production deployment

VIII. How Real Are the Synthetic Suppliers?

We tested 33 specific behavioral markers extracted from real transcripts.

Sample Markers

• Supplier asks “你们做什么渠道的？” before quoting — present
• Supplier demands logo file before quoting customization — present
• Supplier says “稍等” while checking — present
• Supplier sends [图片] mid-conversation — present
• Supplier corrects own pricing mid-message — present (emergent)
• Supplier uses [语音消息] for complex explanations — present
• Platform auto-messages interleave with supplier replies — present

Resemblance by Archetype

The gap is primarily in responsive suppliers — real ones ask more counter-questions and use more multi-message bursts than our simulation currently produces.

IX. Open Weaknesses

1. E9 Customization (0.55 avg on Kimi) — When suppliers demand logo files, the bot redirects (“文件准备中”) but doesn’t always extract a price range. Real suppliers will do this frequently.
2. No price negotiation — S1 is a stretch metric. The bot accepts first price. This is by design for V1 but will be expected in production.
3. Synthetic ≠ real — 88% resemblance means 12% of real supplier behavior is not captured. Multi-message bursts and platform-injected promotional messages are the main gaps.
4. LLM judge, not human judge — All scores are from an LLM evaluator. Shen’s blind test is the true quality gate.