How good are OpenSpec specs?
We read 16,804 requirements in 2,444 spec files from 40 public OpenSpec projects, scored them against OpenSpec's own rules and the classic tests of a good requirement, and had two blind graders read a sample. Then we did the same for the 25 most-starred OpenSpec projects on GitHub, and for 25 teams that write change proposals but never archive them. The format is followed almost everywhere. The writing is where projects differ, and it gets worse as specs grow.
What the study shows
OpenSpec's own writing guide already sets the right rules: one SHALL per requirement, no implementation detail, scenarios for the error cases, a real Purpose for every spec. Its validator checks the grammar, not those rules. Across 87 scored projects and 1,725 open changes, that gap explains most of what we found.
- The format is solved; the writing is not. Nearly every project passes OpenSpec's grammar checks: in 37 of 40, essentially every requirement has a
SHALLand a scenario. Passing says nothing about whether the requirements read well. The blind reading and the automatic checks are unrelated (ρ = −0.02). - Teams break OpenSpec's own writing rules, and more so as specs grow. In a median project only 55% of requirements state a single obligation, and only 35% have a scenario for a failure. Larger projects score lower, mainly because requirements bundle obligations and name source files and functions. The size effect appears in two independent samples (ρ = −0.53 and −0.52).
- Requirements start focused and erode later. In teams that only write proposals, 85% of requirements state one obligation, against 56% in the random draw's main specs. The bundling arrives as specs are merged and edited.
- Specs lag behind the code. In projects with main specs, 39% of open changes have every task ticked but were never archived, so the specs don't include that work, and a third of all spec files still carry the placeholder Purpose that archiving writes. Many teams use OpenSpec as a task tracker with a spec attached.
- Reputation is not a quality signal. GitHub stars don't predict spec quality (ρ = −0.03 in the random draw, −0.20 among the 25 most-starred), and the projects people point to as examples score no better than a random one. OpenSpec's own repository is the best reference we found, 3rd of 40, though 23 of its own 30 open changes also have every task ticked and are not archived.
- Small mechanical slips cost a lot. Two well-known repositories translated OpenSpec's headings, so OpenSpec reads none of their requirements. In Part 1, 768 requirements in 8 projects sit under change headings inside main specs, where OpenSpec never sees them.
What a team can do
- Check the writing rules automatically, in CI, not by checklist: one obligation per requirement, no file paths or signatures, at least one failure scenario.
- Make
openspec archivepart of "done", so the specs move with the code. - Write the Purpose when a capability is created, and keep OpenSpec's headings in English whatever the prose language.
- Judge an example by its specs, not its stars.
Limits: these are correlations across public repositories, not experiments. The reading was done by language models against a rubric, with two passes agreeing within one point on more than 99.9% of grades. Repositories are snapshots of their default branch on October 15, 2026. ρ (rho) is a rank correlation: it runs from −1 to 1, and a value near 0 means no relationship. Full method below.
Forty projects, scored and ranked
Twenty-five projects drawn at random, the projects people hold up as examples, four large projects from a first look, and OpenSpec's own repository, all scored on the same yardstick.
Specs get worse as they grow
Every measure we use is a rate per requirement, so a big project gets no credit and no penalty for its size alone. Size still predicts the score better than anything else we looked at. Projects with a few dozen requirements fill the top of the list, and all five projects with more than a thousand requirements rank in the bottom half.
The decline is specific. Clarity and testability hold up at any size. What erodes is atomicity and staying out of the implementation: in large projects one requirement starts carrying three obligations, and statements start naming source files, internal classes and function signatures.
The ranking, by size
Because size matters so much, we rank projects only against others of similar size. A project is in the top tier when its whole 90% confidence interval sits in the upper half of its band, in the bottom tier when it sits in the lower half, and in the middle otherwise. Most projects land in the middle: the data separates the ends of each band, not the order within it, so names inside a tier are alphabetical.
Following the format is the easy part
OpenSpec's tooling enforces its grammar, and it shows. In 37 of 40 projects essentially every requirement says SHALL or MUST and has a scenario; scenarios almost always have a WHEN and a THEN. None of that tells you whether a requirement is any good. Across projects, the score from the blind reading has no relationship with the score from the automatic checks.
The reading penalized two patterns most often. Both pass every format check. The examples below are written for this article to show the patterns; they are not quoted from any project.
Common defects
A handful of problems show up across the sample. Each is cheap to find and cheap to fix.
### Requirement: blocks left under ## ADDED Requirements or a second ## Requirements heading in a main spec, where OpenSpec's own reader never sees them.The examples people point to
We searched docs, blogs, talks, forums and Chinese and Japanese write-ups for projects that someone holds up as good OpenSpec. We found very few. OpenSpec's maintainers name one example, OpenSpec's own repository, and it earns the reference: it ranks 3rd of 40 overall and tops the medium band. The handful of third-party projects that others point to do no better than a random draw.
Do popular projects write better specs?
No. The 25 most-starred OpenSpec projects, from 28,000 stars down to about 500, score about the same as the random draw, and within that group more stars do not mean better specs.
Part 1 is a random draw, so it is mostly small projects with few or no stars. Here we take the other end: the 25 most-starred repositories that keep at least three main spec files. Each is scored against Part 1's 40 projects as a fixed yardstick, so the scores are directly comparable. Three of the 25 were already in Part 1.
Popular repositories that use OpenSpec turn out to be rarer than the search hits suggest. Of the 100 most-starred repositories our searches matched (all above 400 stars), 49 only mention OpenSpec, in a README, an agent prompt or a config file, and have no openspec/ folder. Only 28 keep three or more main spec files.
Two famous repos OpenSpec can't read
Two of the ten most-starred candidates, studyzy/imewlconverter (10,438 stars) and hawk86104/three-vue-tres (2,447 stars), translated OpenSpec's structural headings into Chinese: ## 需求 for Requirements, ### 需求:, #### 场景:. The prose can be in any language, but the headings are syntax. OpenSpec's own validator rejects every spec in both projects, so to OpenSpec they contain no requirements at all. We could not score them and took the next two projects by stars instead.
The best of the popular projects are as good as anything in Part 1: aiogram/aiogram, Savory/Danet, justskiv/goschedviz, b310-digital/teammapper, dotnetfactory/fluid-calendar and SplashtopInc/winstall would all place in the top ten of the 40.
At the other end, alibaba/druid, the most-starred project we could score, has a lower reading score than 36 of the 40 projects in Part 1, and one in five of microsoft/rayfin's 46 spec files fails OpenSpec's own validator.
Teams that never archive
Some teams use OpenSpec only to write change proposals and never archive them, so their main spec folder stays empty. Their proposals are written a little more tightly than the average main spec, but nothing ever turns them into one current description of the system.
Parts 1 and 2 score only openspec/specs/, the agreed description of how a system behaves. A change proposal lives in openspec/changes/<id>/ as delta specs until openspec archive merges it into the main specs. In our pool, 130 repositories had changes but no main specs at all. We scored the proposals of 25 of them, drawn at random, on the same measures as Part 1's random draw. These scores describe proposals, not a current specification, and stay out of the earlier rankings.
The bundling arrives later. In proposals, a median of 85% of requirements state exactly one obligation; in Part 1's random draw it is 56%. These are different projects, so this is not a before-and-after measurement, but it fits Part 1: requirements start focused, and obligations pile up as specs grow and get edited.
Proposals almost only add. Across the 25 projects, active changes hold 1,581 ADDED requirements, 76 MODIFIED, 4 REMOVED and none RENAMED. Without archiving, nothing reconciles one proposal with the next; the project's real specification is the sum of every open change.
Most changes pass the validator. 216 of 232 active changes have no validator error; 9 of the 25 projects have at least one that fails, most often a change with no delta spec in it at all. Four of the 60 eligible projects write change specs that aren't deltas at all, using their own template or the main-spec layout, so OpenSpec has nothing to apply.
At the bottom sits monid-ai/monid, at 4,235 stars the most-starred proposal-only project we found: 35 open changes and 323 requirements, never archived, with low marks for atomicity and in the blind reading. Loft 2.1's importer reads main specs only: it declines a project that has none and names the active changes to archive first.
A task tracker with a spec attached
In most projects the change folder is where the work happens: a proposal, a task list, boxes ticked one by one. The step that turns finished work into specification, openspec archive, is the one teams skip.
Parts 1 to 3 suggested a pattern: specs created by archiving and never written, proposals that almost only add, changes with no spec content at all. Here we test it by counting, without grading, across every project we cloned: 65 projects with main specs and 112 proposal-only projects. For each open change we read its tasks.md checkboxes and its delta specs. A change whose every task is ticked but which still sits in openspec/changes/ is work that has been done and not folded back into the specification.
tasks.md with no delta spec behind it. Counts at the default branch on October 15, 2026; a finished change may still be waiting for review or a release.The maintainers are no exception: 23 of the 30 open changes in OpenSpec's own repository have every task ticked. Finished-but-open changes are not necessarily neglect. Some teams archive at release time, and some wait for review. But until a change is archived, the main specs describe the system as it was before that work, and every tool that reads them, OpenSpec's own show and validate included, sees the old behavior.
Five fixes for an OpenSpec repo
- One obligation per requirement. If a requirement has a second SHALL, it is probably a second requirement. Across the sample a median of 55% of requirements state exactly one.
- Keep the implementation out. File paths, internal class names and function signatures belong in
design.mdor the code. A requirement should still hold if the code is rewritten. - Write the Purpose.
TBD - created by archiving change …is still the Purpose of 866 of 2,444 spec files. It takes a sentence to replace. - Archive what you finish. A change whose tasks are all ticked has shipped; until
openspec archiveruns, the main specs still describe the system without it. And merge deltas properly:## ADDED Requirementsleft in a main spec hides those requirements from OpenSpec itself. - Cover the failure path. A median of 35% of requirements have a scenario for invalid input, an error or a refusal. The happy path is the part least likely to be wrong.
What Loft sees in the same specs
Loft is a specification format, a command line and an agent skill. Loft 2.1, released on October 15, 2026, converts the main specs of an OpenSpec project into a Loft specification: loft import openspec writes them into spec/ as a product contract, the structured specification of what a product should do and of the decisions and constraints that govern it. Each capability becomes a Feature, each requirement a Requirement, and each scenario an acceptance criterion.
We kept Loft's own checks out of the ranking above, so the scores don't favor our tool. For the record, we ran the released importer on all 40 projects. It converted every one without an error, with the same requirement count as our own extractor. A median of 94% of statements land in EARS forms (the structured "When …, the system shall …" shapes), because most OpenSpec requirements already follow them once SHALL is read as shall.
The two most frequent findings of a first loft check are the patterns the blind reading penalized. OpenSpec states obligations in scenario text, which loft check flags as warnings: 41,116 of them, in 38 of the 40 projects. And in every project, it notes statements that may hold more than one verifiable statement.
The loft-spec skill's adoption workflow has a coding agent rewrite acceptance criteria as sentences a test can check and fill in placeholder purposes, and the people responsible review the changes and ratify the result. Loft's open-source edition is free to try on your own project.
How we scored
- Sample
- 25 projects drawn at random (seed 20261015) from 2,868 repositories found through GitHub code and repository search, stratified by size, one per owner, excluding forks, archives, templates and tutorials. Plus 4 large projects from a first look, 4 projects showcased as examples (2 of them demos), 6 named in OpenSpec community lists, and OpenSpec itself. Delta-only projects (130 in the pool) can't be scored and were counted, not ranked.
- What is scored
- Only
openspec/specs/**/spec.md, read the way OpenSpec's archive reader reads it. Our extractor and Loft 2.1's importer, written independently, agree on the requirement count of every project. - Grammar (20%)
- Share of specs without an error from
openspec validate1.14.1; requirements with SHALL/MUST; requirements with a scenario; scenarios with WHEN and THEN. - Statements (20%)
- One obligation per statement; no vague terms (a fixed list after ISO/IEC/IEEE 29148); no file paths or function signatures.
- Scenarios (20%)
- Two or more scenarios per requirement; at least one failure-path scenario; THEN lines that name an observable outcome.
- Upkeep (10%)
- Real Purpose sections; no delta headers in main specs; unique requirement names; requirements visible to OpenSpec's reader.
- Reading (30%)
- 20 random requirements per project (797 in all), stripped of the project's name and shuffled across projects, graded 1–5 for clarity, testability, atomicity and staying solution-free against a rubric. Two independent passes by Claude agreed within one point on 99.9% of grades (weighted κ 0.80–0.94). A third pass by a different model put the same five projects at the top and four of the same five at the bottom.
- Scoring
- Each measure becomes a percentile across the 40 projects; dimensions average their measures; the score weights the dimensions as shown. Rank intervals come from 1,000 bootstrap resamples of each project's spec files and graded requirements.
- Part 2
- The 25 most-starred repositories in the same pool (plus web-search finds) with at least three main spec files, one per owner, same exclusions; OpenSpec's own repository, already in Part 1, is left out. Each is scored by adding it alone to Part 1's 40 projects, so Part 1's scores don't move; intervals come from 1,000 bootstrap resamples of the project against that fixed set. Its 439 requirements were graded blind in two passes with the same rubric (weighted κ 0.79–0.92), in separate batches from Part 1. Two repos whose OpenSpec headings are translated were counted, not scored, and replaced by the next two by stars. Because the graders saw only Part 2 items, their scale may have drifted slightly: mean solution-free grades were 3.79 against 3.56 in Part 1. A drift in that direction would favor the popular projects, and they still show no reliable advantage.
- Part 3
- 25 proposal-only repositories drawn at random (seed 20261015) from the 60 delta-only pool repositories with at least three delta spec files in active changes, one per owner. The
## ADDEDand## MODIFIEDrequirements in their active changes are scored with the same measures and the same blind reading, without Upkeep, which doesn't apply to a proposal, and compared with Part 1's random draw scored the same way; validity is checked per change withopenspec validate --changes. 483 requirements were graded blind in two passes (weighted κ 0.81–0.92). Each project is scored against Part 1's 40 with Upkeep weighted zero, and the bootstrap resamples whole changes. - Statistics
- ρ is Spearman's rank correlation, from −1 to 1; near 0 means no relationship, and a negative value means one measure falls as the other rises. κ measures how far two grading passes agree beyond chance, where 1 is perfect agreement. A Mann–Whitney z between −2 and 2 means a difference between two groups is within what chance alone produces. A 90% interval is the range a score or rank falls in nine times out of ten when the data is resampled.
- Limits
- Language-dependent checks cover English and Chinese only; French and German projects get "n/a" there. Blinding removes a project's own name but not other product names its text mentions. The graders are language models. Popular projects are rare in a random draw (median 0 stars), which is what OpenSpec use looks like on GitHub.
Disclosure: Loft builds a specification tool that imports OpenSpec projects. The method was written down before scoring, and every change made after the pilot is logged with its reason.
All 40 projects
Click a column to sort. Overall ranks across bands mostly reflect size; use the band tiers above for comparisons.