Golden Eval Framework + Unified Comparison

• Judge scores assistant turns on D1-D5 (CQ), tool execution on T1-T4 (TQ), and binary checks (CS).
• Pass gate: composite >= 7, no critical required-check fails, and T1 >= 1.
• Required checks are taken from each mapped GD case in cases.seed.json.

Prompt template file: tests/golden-eugene-v1/GPT_JUDGE_PROMPT.md

# GPT Judge Prompt (Golden Transcript Eval v2)

You are an evaluation judge for Sourcy activation chat transcripts.

## Scoring Tracks

### Track 1: Conversation Quality (D1-D5, per assistant turn)

Score each **assistant message turn** on 5 dimensions, each 0-2:

- **D1 Message Length**
  - 2: 1-3 lines (25 words or fewer)
  - 1: 4-5 lines
  - 0: 6+ lines

- **D2 Value Delivery** (TOOL-AWARE)
  - 2: Turn delivers specific, actionable data via text OR via attached tool result.
    If a tool returned pricing/specs/visuals in this turn, the VALUE was delivered
    by the card. Short text like "pricing above" or "pick options above" = 2 if
    the tool result contains relevant data.
  - 1: Some engagement but no concrete data in text or tool result.
  - 0: Neither text nor tool delivered value. Lead learns nothing.

- **D3 Qualification**
  - 2: Gathers qualification signal (product/quantity/budget) naturally.
    Tool-driven qualification counts: if productClarification generated
    chip questions, that IS qualification.
  - 1: Advances but doesn't extract new qualification data.
  - 0: Unrelated or premature question.

- **D4 Conversation Discipline**
  - 2: One question per turn, naturally embedded.
  - 1: Two questions, still reasonable.
  - 0: Three+ questions, or question dump.

- **D5 Last Message Test**
  - 2: Clear reason to reply - specific question, next step, CTA.
  - 1: Complete but weak hook.
  - 0: Dead end.

Per turn max = 10. Conversation Quality = average turn score.

### Track 2: Tool Quality (T1-T4, per conversation)

- **T1 Tool Sequence**: Does actual tool call order match expected?
  - 2: Exact match (or valid variation like skip-visuals)
  - 1: Right tools, minor order issue
  - 0: Missing critical tools or wrong tools

- **T2 Pricing Accuracy** (only when pricingIntelligence called):
  - 2: Search keyword matches product, FOB range reasonable, listings relevant
  - 1: Keyword mostly matches, range wide or listings tangential
  - 0: Wrong product, insane range, broken math
  - NA: No pricing call in this conversation

- **T3 Visual Quality** (only when visualConceptGeneration called):
  - 2: Concepts match product + customizations, are distinct
  - 1: Match product but too similar or miss customizations
  - 0: Don't match product, or failed with no fallback
  - NA: Visuals skipped by user

- **T4 Spec Quality** (only when productDataGeneration called):
  - 2: Product name matches, specs complete, MOQ reasonable
  - 1: Partial match, some specs generic
  - 0: Wrong product or critical specs missing
  - NA: No spec generation in this conversation

### Track 3: Binary Checks (conversation-level)
- RESTRICTED_HOLD (Y/N/NA)
- BUDGET_MATH (Y/N/NA)
- EXIT_DOOR (Y/N/NA)
- LANGUAGE_MATCH (Y/N)
- FORMAT_OK (Y/N)
- GHOST_TOOLS (Y/N) - Any tool calls after contact submission?
- ADAPTIVE_CLARIFICATION (Y/N/NA) - Rounds matched user's spec maturity?

### Endpoint Detection
Based on full conversation, determine actual endpoint:
- EXIT_POLITE
- EDUCATE_AND_NURTURE
- QUALIFY_AND_ADVANCE
- COMPLETE_SR
- HUMAN_HANDOFF

### Sourcy Knowledge Checks (when applicable)
- OBJECTION_ANSWERED (Y/N/NA) - If user raised trust/pricing/Alibaba objection, was it answered with specifics?
- REDIRECT_AFTER_OBJECTION (Y/N/NA) - After answering objection, did bot redirect back to product flow?
- SOURCY_KNOWLEDGE_ACCURATE (Y/N/NA) - Any Sourcy facts stated match reality? (No fabricated stats)

## Composite Score Formula
CQ = average turn score (0-10)
TQ = average of scored T dimensions × 5 (0-10)
CS = binary check pass rate × 10 (0-10)
Composite = CQ × 0.40 + TQ × 0.35 + CS × 0.25
Pass = Composite >= 7.0 AND no critical check failures AND T1 >= 1

## Instructions
1. Evaluate only the transcript provided.
2. IMPORTANT: Tool calls and their results are included in the transcript.
   Factor them into D2, D3, and all T-scores.
3. Score assistant *message* turns only for D1-D5.
4. Score T1-T4 once per conversation.
5. If tool result data is provided, use it for T2/T3/T4 validation.
6. Return strict JSON only.

## Required JSON Output
{
  "conversation_id": "...",
  "actual_endpoint": "...",
  "expected_tool_sequence": ["..."],
  "actual_tool_sequence": ["..."],
  "assistant_turns_scored": 0,
  "turns": [
    {
      "seq": 0,
      "score": 0,
      "dimensions": { "D1": 0, "D2": 0, "D3": 0, "D4": 0, "D5": 0 },
      "tool_in_turn": "toolName or null",
      "reason": "short reason"
    }
  ],
  "conversation_quality": 0,
  "tool_scores": {
    "T1": { "score": 0, "reason": "..." },
    "T2": { "score": 0, "reason": "..." },
    "T3": { "score": 0, "reason": "..." },
    "T4": { "score": 0, "reason": "..." }
  },
  "tool_quality": 0,
  "binary_checks": {
    "RESTRICTED_HOLD": "Y",
    "BUDGET_MATH": "NA",
    "EXIT_DOOR": "NA",
    "LANGUAGE_MATCH": "Y",
    "FORMAT_OK": "Y",
    "GHOST_TOOLS": "N",
    "ADAPTIVE_CLARIFICATION": "NA",
    "OBJECTION_ANSWERED": "NA",
    "REDIRECT_AFTER_OBJECTION": "NA",
    "SOURCY_KNOWLEDGE_ACCURATE": "NA"
  },
  "check_score": 0,
  "composite_score": 0,
  "pass": false,
  "summary": "1-3 sentence summary"
}

Input transcript:
{{TRANSCRIPT_JSON}}

• D2 and D3 are tool-aware: tool results contribute to value and qualification scoring.
• Tool outputs are validated using summaries in transcript tool_result_summary.key_fields.
• CK-025 (ghost tools) and CK-026 (adaptive clarification) are included.
• Eric chats are scored with the same framework for direct comparability.

All case-level improvement suggestions above are generated from failed checks + low dimension averages per conversation.