Scoring formula

Scoring version 1 gives each applicable check its weight as the maximum contribution. A pass earns its full weight and adds that full weight to the possible total. A warn earns and adds half its weight. A fail earns zero and adds the full weight. Not-applicable checks are excluded from both totals.

Score = Math.round(100 × earned ÷ possible). When possible is zero, the score is 0 and the grade is D.

• Package manager lockfile
  setupWeight 510.4%
• Environment variable example
  setupWeight 36.3%
• Node.js version pinned
  setupWeight 36.3%
• Test script
  verificationWeight 510.4%
• Tests pass
  verificationWeight 510.4%
• Build script
  verificationWeight 48.3%
• Test files exist
  verificationWeight 48.3%
• Lint script
  verificationWeight 36.3%
• Typecheck script
  verificationWeight 36.3%
• Installation documented
  docsWeight 48.3%
• Contributor or agent guidance
  docsWeight 24.2%
• Continuous integration config
  hygieneWeight 36.3%
• Formatter configuration
  hygieneWeight 24.2%
• Useful .gitignore
  hygieneWeight 24.2%
• Total weight: 48 · 100%

Benchmark task design and verification

Each benchmark task defines objective pass, partial, and fail criteria and verifies the working result locally. At omot bench start, each task checks its own preconditions; task-specific baseline scripts may run and their exit codes are recorded. Some tasks require a clean working tree or a passing baseline test. During verification, a script failure already present at baseline is not counted, unless that script is the task's explicit target.

Every verification attempt is appended to the run, and the report records the number of attempts. Changing a task's prompt.md or verify.ts requires a version bump; aggregates count only reports for the current task version.

• dep-patch-bump

  Version 6

• add-unit-test

  Version 8

• fix-lint-rule

  Version 5

• a11y-img-alt

  Version 2

• ts-strict-file

  Version 2

• fix-failing-test

  Version 3

Limits of self-reported data

Verification runs on the submitter's machine. Agent name and version, model, human corrections, and minutes are reported claims, not independently observed measurements. Forged or misleading reports are possible. Ingest validates a strict schema and applies rate limits, and maintainers can remove report lines from the dataset.

Known biases

• People choose whether to submit, and successful runs may be more likely to be reported.
• Submitters choose which repositories and tasks to run.
• Agent versions change over time.
• Small samples can move rates substantially.
• Readiness bands are confounded by the repository's stack and type; they do not establish that readiness caused an outcome.
• Human minutes are self-estimated.
• The current task set is small and focused on JavaScript and TypeScript repositories.