Oh My Otter

Contribute

Make the toolkit better

Check and task proposals, reports, and bug fixes make the guidance more useful for every coding agent and repository.

Add a check

There are currently 14 built-in checks.

  1. Create packages/checks/src/checks/<id>.ts with a deterministic, read-only check.
  2. Add its named re-export to packages/checks/src/registry.ts.
  3. Add a row to the README built-in checks table.
  4. Add tests for the check and update the relevant expected counts, statuses, and scores.

Add a benchmark task

There are currently 6 built-in tasks.

  1. Create packages/benchmarks/src/tasks/<id>/ with task.json, prompt.md, and verify.ts.
  2. Add the task to packages/benchmarks/src/registry.ts.
  3. Add its benchmark documentation page and tests for pass, partial, and fail.
  4. Bump the task's version whenever its prompt or verifier changes.

Submit a benchmark report

  1. Run omot bench start <task>.
  2. Let a coding agent work on the task.
  3. Run omot bench verify.
  4. Run omot report. It prints the report JSON and asks before opening a prefilled issue; nothing is submitted without your confirmation.

Report a bug

Include what you expected, what happened, and the command or page involved. Report security vulnerabilities privately as described in SECURITY.md.

Open a bug report