Interpretation Simulator (Phase 0) ​

What it is ​

A zero-API harness that mass-generates templated natural-language inputs with known-answer routing, runs them through the deterministic front half of the interpretive pipeline (extract → resolve), and asserts the routed verb + resolved slots match what the template intended.

It locks down the extractor/resolver floor so misroute classes (like #987 create-envelope→log_time, #958 wrong-person, #973 phantom-user) fail CI the instant they regress — with zero Anthropic spend.

Scope boundary (Phase 0): tests the pure half only — extract (verb router + slot tokeniser) and resolve (slot resolver). It does not call Claude. Phase 1 (#991) covers the Claude-layer assertion.

Running it ​

bash
# Print confusion matrix and write simulator/last-run.json
node simulator/cli.js

# Or via the test suite (runs on every PR)
npm test

Reading the matrix ​

The confusion matrix rows are expected verbs, columns are actual verbs. A perfect run has all counts on the diagonal and zeros elsewhere.

expected \ actual     allocate_hours        log_time              no_verb_match
------------------------------------------------------------------------------
allocate_hours        26                    0                     1
log_time              0                     72                    0

The single off-diagonal count above is in no_verb_match for allocate_hours — this is the regression-987-create-envelope KNOWN_GAP.

Slot accuracy is reported beneath the matrix. A slot is correct when status === 'resolved' and entity.id (or entity.name) matches the expected seed entity. Phantom-user slots are correct when status !== 'resolved'.

Adding a template ​

Templates live in simulator/templates.js. Before declaring expectedSlots, run extract() on a filled instance to confirm where each filler lands:

js
// In a node REPL or scratch script:
const { extract } = require('./lib/entity-reference-extractor');
console.log(extract('log 2h on Development to CityCore App Development'));
// → confirm project_name.raw and task_name.raw are what you expect

Common gotcha: log {dur} to {project} as {task} produces project_name.raw = "{project} as {task}" which findProjectByName cannot resolve (its substring pass checks whether the project name includes the query, not the reverse). Use log {dur} on {task} to {project} instead — verified clean.

Then add the template:

js
{
  id: 'my-new-template',
  pattern: 'log {dur} on {task} to {project}',
  verb: 'log_time',
  expectedSlots: { project_name: 'project', task_name: 'task' },
  assert_slots: true,
}

KNOWN_GAPS convention ​

KNOWN_GAPS arrays in simulator/cli.js and tests/simulator.test.js contain misroute IDs that are excluded from the failing count. Each entry must reference the issue or PR that will fix it. Never silently skip — always document why.

js
const KNOWN_GAPS = [
  // "create an envelope" → allocate_hours requires verb alias from PR #987.
  // Remove when #987 merges to main.
  'regression-987-create-envelope',
];

When a fix lands on main, remove the ID from both files and verify CI is green.

Phase 0 scope boundary ​

Phase 0 asserts canonical_verb and resolved slot entities. It does not:

  • Call Claude or the Anthropic API (no @anthropic-ai/sdk import)
  • Boot the MCP server or connect to Postgres
  • Test multi-turn conversation or paraphrase quality

Those are covered in Phases 1–3 (#991–#993).