Test todo2code deterministically on a fixed, reviewable corpus of external
repositories, derive evidence-backed failure categories, and improve the
library one measured defect at a time.
The initial corpus is:
semcod/code2llm
semcod/domd
semcod/pactfix
semcod/code2logic
semcod/code2docs
semcod/redup
subactor/platform
Every repository run must use an isolated detached worktree at a recorded
commit. The benchmark must not modify an external repository or consume its
private and untracked files.
Acceptance criteria
AC-01: The baseline records repository commit, graph fingerprint, record
and relation counts, topic status, implementation/documentation coverage,
diagnostic counts, warnings and elapsed time for at least five external
repositories.
AC-02: Results use the same documented deterministic command and document
selection policy, with repository-specific exceptions recorded explicitly.
AC-03: At least one repeated semantic failure is demonstrated on external
evidence and represented by a focused gold or unit regression test before
its implementation changes.
AC-04: Each library change is evaluated independently against gold v2 and
the external corpus; improvements and regressions are both reported.
AC-05: The selected improvement raises its target metric on at least two
external repositories, or is rejected with a documented reason, without
reducing gold precision/recall or introducing forbidden-pair violations.
AC-06: npm run verify, relevant smoke tests and Docker validation pass;
the Java test may only be skipped locally when the required CI job remains
verified.
AC-07: Conclusions, raw command output, changed files, remaining risks and
follow-up candidates are preserved in this ticket.
Risks and mitigations
External worktrees may be dirty or contain secrets. Only detached tracked
commits are analyzed; private and untracked files are excluded.
Repository sizes and document sets differ. Absolute counts are never
compared without recording the input policy.
A broad synonym rule may raise recall by destroying precision. A hard
negative is required before changing semantic matching.
Provider-dependent runs would make the baseline unstable and potentially
costly. The primary corpus is offline; live LLM work is a separate result.
project/README.md is also generated by the current analysis workflow.
Ticket indexing must be preserved or explicitly reconciled before running
project.sh.
Parallel agents or builds can race on dist/. Validation must run from a
stable worktree without another build writing the same output directory.
Iteration 01 is accepted. It reduced false review_required findings on five
external repositories without changing any graph fingerprint or gold metric.
Iteration 02 fixed a tracked-evidence false positive in the generated-analysis
isolation gate while retaining the original untracked-input hard negative.
The next iteration should be a separate approved ticket: either broaden
cross-language semantic evidence beyond the hand-written PL→EN dictionary, or
sample and classify the remaining 1,853 actionable changelog findings before
changing linker policy.