Specshift evaluates how well a documentation site serves an LLM trying to use the platform the docs describe. Not whether the docs are well-written. Whether an agent reading these docs ends up at a correct answer, a working call, a passing build.
§ 01
What we measure.
Future suite: oscal (compliance crosswalk for FedRAMP-targeting platforms). Built — wiring it into the live suite list above is pending.
§ 02
Scoring.
Every test produces a score in [0, 1]. NaN means the test couldn't run (timeout, infrastructure failure) and is excluded from the suite roll-up. Suite scores are weighted averages of their tests. The overall score is the weighted average of suite scores.
Reports are reproducible. Every scorecard pins both the methodology version and the engine version. Re-run the same suite on the same target with the same engine and methodology and you get a deterministic score — or the methodology documents the source of randomness.
§ 03
How to run.
Paste a docs URL into /eval and Specshift runs all four suites against it right now — no install required. The scorecard, with the overall score, the per-suite breakdown, and every test, renders inline in the response.
npx @wentzel/specshift run https://docs.example.com --methodology=v1Not yet published to npm — this is the CLI surface once it ships. /eval is the working way to run Specshift today.
§ 04