# TCAI Benchmark — Answer Key & Scoring Guide

Scoring per the article: **accuracy / usefulness / clarity, 1–5 each, two reviewers.** This file supplies ground truth for objective prompts and hard constraints + rubric anchors for subjective ones. `[OBJECTIVE]` = auto-gradable against a fact/answer. `[RUBRIC]` = judge/human scores against anchors. `[CONSTRAINT]` = checkable pass/fail rules that gate the score.

---

## Writing (W) — all [RUBRIC], with [CONSTRAINT] gates

- **W1** — Constraint: < 120 words; keeps the substance (materials overdue, deadline = end of week, penalty risk). Anchor: warm and plain, contractions ok, no "instant"/legalese, still clear about the deadline. Deduct for losing the penalty warning or exceeding word count.
- **W2** — Constraint: 150–200 words, first person, no hashtags, mentions all 4 bullets. Anchor: confident not boastful; credits the team; the hiring line reads as invitation not brag.
- **W3** — Constraint: 2–3 sentences; voice match (short, wry, self-aware, uses a "Radical, I know"-style aside). Anchor: reads like the sample author wrote it.
- **W4** — Constraint: ~120 words (accept 100–140); keeps: portal is live, self-service account/invoice/order-tracking/support, benefit to customer. Anchor: no fluff, no "thrilled and excited" padding.
- **W5** — Constraint: < 130 words; contains (a) clear ownership, (b) no blaming the client, (c) one concrete next step/date. Anchor: sincere, not groveling, not over-explaining the internal mistake.
- **W6** — Constraint: ~60 words (accept 50–70); benefit-led opening; ≥1 sensory line; mentions 24h cold / 12h hot. Anchor: doesn't open with specs.
- **W7** — Constraint: 2–3 sentences, no jargon (no "microservices/pub-sub/horizontally scalable"). Anchor: a layperson understands what it does (handles lots of data reliably, scales up).
- **W8** — Constraint: exactly 3 lines, each < 8 words, each labeled curiosity/benefit/urgency. Anchor: labels actually fit the lines.
- **W9** — Constraint: < 90 words; explicitly acknowledges she gave *specific/detailed* feedback. Anchor: genuine, not sycophantic, no empty flattery.
- **W10** — Constraint: ~100 words, first person, includes all facts (12 yrs PM, 2 startups/1 acquired, leads 15, mentoring, Austin, trail running). Anchor: warm, credible, not a bullet dump.

---

## Research (R) — [OBJECTIVE] facts R1–R5; [RUBRIC + citation check] R6–R10

Citation check for ALL R: every URL must (a) resolve, (b) actually support the claim. Log fabricated/broken URLs — this is the hallucination tally the article reports by name.

- **R1** [OBJECTIVE] — Completed **1889** (opened March 31, 1889). Height **≈330 m / ~1,083 ft** including antennas (≈300 m / 984 ft without). Accept 324 m (older antenna figure). Wrong year or a height off by >10 m = accuracy miss.
- **R2** [OBJECTIVE] — **100 °C / 212 °F** at sea level. Exact only.
- **R3** [OBJECTIVE] — **Roger Penrose, Reinhard Genzel, Andrea Ghez** — for work on **black holes** (Penrose: black-hole formation from general relativity; Genzel & Ghez: supermassive compact object at the galactic center). All three names required for full accuracy.
- **R4** [OBJECTIVE, human-verify current figure] — **≈123–124 million**, **decreasing**. Ground-truth the exact current figure at test time (UN/Statistics Bureau of Japan). "Declining/shrinking" is the non-negotiable part; a ±2M figure is fine.
- **R5** [OBJECTIVE] — **160.934 km** (100 × 1.60934). Accept 160.9 km. Factor 1 mi = 1.60934 km required.
- **R6** [RUBRIC] — Should include ≥3 of: reduced cardiovascular risk; improved lipid/cholesterol profile; lower type-2 diabetes risk; reduced all-cause mortality; possible cognitive benefit. Each needs a supporting citation.
- **R7** [RUBRIC] — Right to erasure = individuals can request deletion of personal data (GDPR Art. 17). Limitations: freedom of expression, legal obligations, public-interest/health, archiving/research, establishment of legal claims. Full marks require naming ≥2 real limitations + a citation to an official source (e.g., gdpr.eu, official EU text).
- **R8** [RUBRIC] — Must distinguish the three intake levels: *moderate/daily* (≈3–5 cups/day) is generally safe for most adults and linked to neutral-to-beneficial outcomes (lower risk of type-2 diabetes, some CVD and mortality benefit); *heavy* (roughly >5–6 cups or high caffeine) shifts toward harms — anxiety, insomnia, acute blood-pressure rise, and reduced benefit; caveats for pregnancy. Full marks require addressing all three levels + authoritative citations (e.g., a major cohort review, EFSA/health authority). Penalize answers that ignore the dose gradient or make confident one-sided claims.
- **R9** [RUBRIC] — HTTPS = HTTP over TLS/SSL: encrypts traffic, authenticates the site, needed for trust/SEO/security; HTTP is plaintext. Needs a correct, source-backed distinction.
- **R10** [RUBRIC] — Causes should include ≥3 of: demand-pull (demand outstripping supply), cost-push (rising input/energy/wage costs), monetary expansion (excess money-supply growth), and inflation expectations. Central-bank tools should include ≥2 of: raising/lowering the policy interest rate, open-market operations, reserve requirements, quantitative tightening/easing, and forward guidance. Full marks require a balanced causes + tools answer with authoritative citations (e.g., a central bank — Fed/ECB — or an economics reference). Penalize one-dimensional answers ("printing money" only).

---

## Reasoning (RE) — [OBJECTIVE]

- **RE1** — **84 mph.** (240+180=420 mi; 3+2=5 h; 420/5.) Common wrong answer: averaging 80 & 90 = 85. Penalize 85.
- **RE2** — **$50.** (40 / 0.80.)
- **RE3** — **9 hours.** (3×12 = 36 worker-hours; 36/4.)
- **RE4** — **$1,157.63.** (1000 × 1.05³ = 1000 × 1.157625.)
- **RE5** — **No, it does not follow.** The roses need not be among the fast-fading flowers. Full marks require the correct "no" + valid reason.
- **RE6** — **Cara 1st, Beth 2nd, Ann 3rd.** (Beth immediately ahead of Ann + Ann not first ⇒ Beth 2/Ann 3; Cara not last ⇒ Cara 1.) Unique.
- **RE7** — **$0.05** (5 cents). Classic trap answer $0.10 = fail.
- **RE8** — **Total 8 hours.** Any order respecting A→B and (B,C)→D, e.g. **A, C, B, D** or **C, A, B, D.** (Single worker ⇒ total is the 8h sum; the skill is valid ordering.) Penalize orders that violate dependencies.
- **RE9** — Any schedule with **Z at 11:00** and **X before Y.** Example: X 9:00, Y 9:30, Z 11:00. Fail if Z ≠ 11:00 or X after Y.
- **RE10** — **Yes, you make the connection, with 25 minutes to spare.** Land 1:35 → +25 customs = 2:00 → +15 walk = 2:15 at gate; boarding closes at 2:40 (20 min before 3:00 departure); 2:15 is before 2:40. Full marks require the correct yes + the arrival-vs-boarding-cutoff comparison (not just "there's 85 minutes"). Wrong if it compares arrival to 3:00 departure instead of the 2:40 boarding cutoff.

---

## Everyday work (E)

- **E1** [OBJECTIVE] — `=SUMIF(A:A,"Paid",B:B)`. Accept ranged equivalents (e.g., `A2:A100`). SUMIFS acceptable.
- **E2** [RUBRIC] — 4 agenda items mapping to the 4 notes, time allocations summing to ~60 min, sensible order (open floor last).
- **E3** [OBJECTIVE] — Must list: **Mar 3 2026** — initial draft (Provider); **within 10 business days** [of the draft] — Client feedback; **Apr 15 2026** — final materials; **May 1 2026** — remaining balance payment. Missing/incorrect date = accuracy miss.
- **E4** [RUBRIC] — 2 sentences, no jargon; conveys: split the big system into smaller independent pieces so we can ship/fix faster and recover from outages sooner.
- **E5** [CONSTRAINT + RUBRIC] — Must satisfy: starts 9:00, ends ≤17:00, includes Louvre, includes a 1-hour lunch, walking legs ≤30 min total (as stated). Fail any constraint = capped accuracy. Rubric for realism/usefulness.
- **E6** [OBJECTIVE-ish] — Exactly 3 action items ≈ **Sam: send final headcount by Friday; Lee: loop in finance once headcount is in; Priya: review before it goes out.** Owners required. Budget draft is blocked on headcount — good answers reflect the sequence.
- **E7** [RUBRIC] — 2–3 sentences, polite, declines the meeting, proposes email/async. No over-apologizing.
- **E8** [RUBRIC] — Prioritized (not just reordered arbitrarily); the Thursday deck should rank as time-sensitive; each item has a rough estimate.
- **E9** [OBJECTIVE] — `=VLOOKUP(D2,A:B,2,FALSE)` or `=XLOOKUP(D2,A:A,B:B)`. Exact-match required (FALSE / default XLOOKUP).
- **E10** [OBJECTIVE-ish] — **Schedule: Amber (or Red) — one week behind; Budget: Green; Scope: Green.** Schedule must not be Green.

---

## Speed (S) — [OBJECTIVE] except S2/S6/S9

- **S1** — **36.**
- **S2** [RUBRIC] — Correct concept: the value of the next-best alternative given up when making a choice.
- **S3** — **Canberra.** (Trap: Sydney/Melbourne = fail.)
- **S4** — **22 °C.** (72−32=40; ×5/9 = 22.2.)
- **S5** — **Red, blue, yellow** (traditional). Accept RGB (red, green, blue) if framed as additive/light.
- **S6** [RUBRIC] — Any valid synonym: thorough, careful, precise, painstaking, fastidious.
- **S7** — **necessary** (spelled correctly).
- **S8** — **132.**
- **S9** [RUBRIC] — Roughly 5–7–5, on topic (coffee), coherent. Judge clarity/craft.
- **S10** — **Tuesday.**

---

## Scoring mechanics (for the harness/judge)

- Objective prompts: exact/near-exact match → accuracy 5; minor slip → 3; wrong → 1. Usefulness/clarity still rubric-scored.
- Subjective prompts: LLM-judge drafts 1–5 on each axis (blind to model identity, order randomized), human confirms; disagreements > 1 point flagged for human.
- **Constraint gates:** if a `[CONSTRAINT]` fails (word count, missing required element), cap accuracy at 2 regardless of quality — the model didn't do the task as asked.
- **Citation/hallucination log:** for every R prompt, record count of URLs given, count that resolve, count that support the claim. Fabricated citations are named in the article.
