Diagram Generation Eval v3 — Ground Truth Comparison

Summary

Eval Cases

Visual PASS

Visual PARTIAL

Visual FAIL

What this report shows: Each case compares the actual HKDSE past paper diagram (left) against our generated SVG (right). Below each pair is a visual inspection with specific notes on what matches and what doesn't. Programmatic checks (style, labeling, edge count, orientation) run separately and are shown as badges.

Honest caveat: Both the SVGs and the validator were written by me in this session. The programmatic checks pass (82/82) but the visual inspection reveals issues the validator doesn't catch — label placement, projection proportions, missing construction lines. The real test is LLM-generated SVGs from just the prompt + spec.

Case-by-Case Comparison

2012 Q40 — Regular Tetrahedron

PARTIAL 9/9 PASS

Regular tetrahedron ABCD, side 2x. E is foot of perp from A to BC. Find angle AED.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: Structural geometry correct — all 6 tetrahedron edges present, BD correctly dashed. However, the ground truth shows internal construction lines (median from A to base center, midpoint lines within faces) that my SVG omits. These construction lines help students identify the dihedral angle. Missing feature: internal construction geometry.

2014 Q40 — Perpendicular to Plane

PASS 11/11 PASS

Triangle BCD (8-15-17 Pythagorean triple), AB=8m perpendicular to plane BCD. Find tan∠AEB.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: Good match. AB vertical pole, AC/AD oblique edges both present. CD correctly dashed as hidden edge. Dimension labels (8m, 15m, 17m) placed near correct edges. Right-angle mark at B. The base triangle BCD spreads below with perspective depth, matching ground truth orientation.

2016 Q39 — Rectangular Box

PARTIAL 9/9 PASS

Rectangular box ABCDEFGH. P midpoint of AC, Q on GH. Find sin∠PFQ.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: All 12 box edges present. 3 hidden back edges correctly dashed. P and Q auxiliary points marked. Dimension labels (16cm, 12cm, 15cm, 9cm) present. Right-angle marks at corners. Box proportions and oblique projection match the ground truth well. Issue (manual inspection): Ground truth draws an internal plane — a diagonal cross-section through the box showing the face on which P lies — which our generated SVG completely omits. Programmatic checks pass (structural edges/labels correct) but the key construction geometry for the exam question is missing.

2017 Q39 — Perpendicular Planes (Vertical Pole)

PARTIAL 11/11 PASS

AD vertical pole on horizontal ground BCD. AB=25m, AD=15m, BC=29m, CD=21m. Find angle between AB and plane ACD.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: Structural match — AD solid vertical pole, AB/AC dashed oblique lines, BD/DC/BC dotted ground lines. Right-angle mark at D (BD⊥DC). Dimension labels present. Issue: "29 m" label (for BC) is placed near the BD edge area, not along the BC base edge. Ground truth has "29 m" label absent (dimensions shown differently). Also: ground truth shows AB/AC as thick dashed lines, while my SVG makes them thinner.

2018 Q41 — Rectangular Block with Point X

PARTIAL 8/8 PASS

ABCDEFGH rectangular block. X on edge DE, DX=9, EX=4. Find cosθ between BX and plane ABGF.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: Box structure correct with all 12 edges. Vertex mapping (A-F, B-G, C-H, D-E as vertical pairs) matches ground truth labeling. X marked on edge DE. BX construction line drawn as dotted. Issue: X positioned at ~60% up DE instead of 9/13≈69%. SVG has a duplicate line (FG edge drawn twice). The ground truth shows height dimension but my SVG omits it. Only 12cm and 8cm labeled.

2019 Q40 — Point Above Ground

PASS 10/10 PASS

Tetrahedron PQRS, base QRS on ground. Q vertically below P. ∠PRQ=47°, ∠PSQ=53°, ∠RQS=120°. Find ∠RPS.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: Good match. P at apex, Q below on ground (dashed vertical PQ). PR/PS solid oblique edges, QR/QS dashed ground edges, RS solid base. Right-angle mark at Q (PQ⊥ground). Ground truth shows the same structure with similar proportions.

2020 Q38 — Right Triangular Prism

PARTIAL 7/7 PASS

ABCDEF right triangular prism. P on AE, AB=AC=12, AP=9, EP=5, BD=2k. Find area of △BDP.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: All 9 prism edges present. AC correctly dashed (hidden back edge). P marked on edge AE. Construction lines BP, DP, BD drawn as dotted. Issues (manual inspection): (1) BF edge is rendered as a tilted/diagonal line in the generated SVG instead of straight as shown in the ground truth — structural positioning error in the prism projection. (2) Ground truth prism has B protruding further forward/downward, creating a more pronounced 3D perspective; triangular face EDF is more tilted than in my SVG.

2022 Q40 — Cube (Cross-Sections)

FAIL 7/7 PASS

Cube ABCDEFGH. α = angle between △AFG and △AFH. β = angle between △AFH and △FGH.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: Completely deformed. The generated SVG fails to reproduce the cube — the overall shape, vertex connections, and projection are fundamentally wrong compared to the ground truth. Programmatic checks passed (vertex labels present, edge count correct) but the structural geometry is deformed beyond usability. This case highlights the core problem: our programmatic validator cannot detect whether the shape is correctly drawn, only whether required symbols are present.

2024 Q40 — Tetrahedron on Ground

PASS 10/10 PASS

Tetrahedron PQRS, base PQR on ground. Q vertically below S. ∠PQR=90°, ∠QPS=30°, ∠QRS=45°. Find cos∠PRS.

Ground Truth (Past Paper)

Generated SVG (v3)

Visual Inspection: Good match. S at apex, Q below (dashed vertical SQ). SR/SP solid oblique, QR/QP dashed ground, RP solid base. Right-angle mark at Q. Same structural pattern as 2019 Q40. Ground truth shows the same layout with S at top and Q slightly offset in the ground plane.

What the Validator Catches vs. What It Misses

Catches (programmatic):

Color violations (non-black strokes/fills)
Missing vertex labels
Missing dimension values
Missing dashed edges
Missing right-angle marks
Unwanted angle arcs or colored markers
Edge count (structural completeness)
Basic vertex orientation (top/bottom/left/right)

Misses (requires visual inspection):

Label placement accuracy (near correct edge vs. wrong edge)
Projection proportions (how "3D" the figure looks)
Missing construction lines (medians, perpendiculars within the figure)
Auxiliary point positioning accuracy (X at 9/13 vs 8/13 up an edge)
Edge thickness and dash pattern matching ground truth
Whether the figure "reads" correctly as a 3D shape at a glance

Verdict

3 of 9 cases pass visual inspection. 5 have partial issues. 1 is a structural failure:

2012 Q40: Missing internal construction lines (medians, perpendiculars shown in ground truth)
2016 Q39: Missing internal plane (diagonal cross-section shown in ground truth); programmatic checks pass but key construction geometry absent
2017 Q39: Dimension label "29 m" misplaced near wrong edge
2018 Q41: X point position slightly off; duplicate SVG line; missing height dimension
2020 Q38: BF line rendered as tilted/diagonal instead of straight; prism projection less dramatic than ground truth
2022 Q40: FAIL — completely deformed. Cube shape is structurally wrong despite all 7 programmatic checks passing.

Bigger problem: we cannot reliably detect if a diagram is correctly drawn. The 2022 Q40 case is the clearest demonstration: programmatic checks (label presence, edge count, orientation flags) all passed 7/7, yet the visual output is completely deformed. The same validator issue affects 2016 Q39, where missing internal geometry went undetected. The ability to draw a diagram is secondary — the more critical unsolved problem is whether we can verify correctness programmatically without human visual inspection.