Skip to content

Compare the evidence.

Same changes. Recorded configurations. Inspectable results. A feature checklist alone cannot tell you which tool catches the regressions that matter to your team.

Open comparison lab Product guides

Published measurements

Only reviewed observations with documented historical information boundaries enter these calculations.

Exploratory advisory-review pilot

Both memory variants identified 1 of 2 documented regression causes. Neither flagged the passing negative control. Memory did not improve detection in this single-change pilot.

1 historical change · 2 regression causes · 1 within-change negative control · competitors not measured.

  • One historical change, two regression causes and one within-change control; no independent benign PRs.
  • One API call per variant; no repeated-run variance or significance estimate.
  • Advisory-review detection only. The isolated harness validates labels; it is not detector proof.
  • First-party, Codex-assisted labeling and target matching; not independently audited.
  • Competitors have not been run. This is not evidence of a competitive advantage.

Inspect isolated label-validation results · Inspect target-match decisions

3 shared cases · 3 reviewed labels. Results apply to this dataset, not all production repositories.

Paired comparison across the shared reviewed cases.
ToolCasesPrecisionRecallFalse positive rateUsable outcomesInconclusive / errors
ShipCheckgpt-6.1-sol/advisory-review-pilot-v1 · memory disabled
Configuration

Production advisory review + deterministic context; fixed historical diff and base-test context; identical model/settings; no history miner or runtime detector. Memory enabled supplies 12 base-only discovered guards. One review call per variant.

3100.0%1/1 reports50.0%1/2 causes0.0%0/1 controls100.0%0 / 0
ShipCheckgpt-6.1-sol/advisory-review-pilot-v1 · memory enabled
Configuration

Production advisory review + deterministic context; fixed historical diff and base-test context; identical model/settings; no history miner or runtime detector. Memory enabled supplies 12 base-only discovered guards. One review call per variant.

3100.0%1/1 reports50.0%1/2 causes0.0%0/1 controls100.0%0 / 0

Actual review calls

One completed API call per memory variant. These are advisory reviews, not runtime-verification executions. Billed cost was not measured.

Measured call latency and returned API token usage; discovery cost and label-validation execution are separate.
MemoryReview timeInput tokensOutput tokensEvidence
disabled23.3s8,386952Raw report ↗
enabled25.5s9,9221,119Raw report ↗
Inspect cases and run evidence

Engineering validation

These checks validate the application and its release. They are separate from historical regression detection and provide no competitor-accuracy evidence.

Automated tests
364/364 passed across 53 test files.
Release checks
TypeScript, lint, test coverage thresholds, and production build passed.
Scoped code coverage
81.23% lines · 74.54% branches. Existing configured Vitest coverage allowlist; not the whole application.
Live schema checks
Rollback-only checks passed for revision capture, verifier version capture, owner access, and cross-account read/insert isolation.

Inspect the engineering validation record ↗

How to read the numbers

Exploratory advisory-review pilot: three behavior-level cases on ONE public Zod regression-causing change, representing two documented regression causes and one within-change negative control. The control is not an independently benign PR. Twelve behavior candidates were extracted from bounded base-only source and verified for exact source quotation; no head/fix/label/harness was passed to discovery. The production review engine ran once per memory variant with identical diff and pre-existing test context; the enabled variant additionally received all 12 discovered guards without hindsight filtering. Later fix assertions were used ONLY in a separate isolated label-validation job. One finding is counted once for its ExactOptional values/pattern root cause; the recursion-break cause was missed. Results are correlated within one change, single-run, first-party and Codex-assisted, not independently audited. Model-training contamination and selection bias remain possible. These advisory findings are NOT runtime-verified detector results, and usable review coverage is NOT contract-verification coverage. No competitor was run. No general accuracy or superiority claim is supported.

Precision
Confirmed target detections divided by all target violation reports. Requires reported violations.
Recall
Detected regressions divided by all regression cases, including inconclusive runs and execution errors.
False positive rate
False alarms divided by conclusive benign cases. Read alongside conclusive coverage to account for abstentions.
Conclusive coverage
The share of cases with a violation or clear result. An error or inconclusive result never counts as a clean pass. For advisory reviews, a usable outcome means the review completed; it does not mean the target behavior was runtime verified.

These are first-party, Codex-assisted reviewed observations. Tool versions, settings, selected repositories, sample size, model knowledge of public fixes, and labeling judgments can affect results. A public historical benchmark cannot fully eliminate model-training contamination.

Greptile already documents memory and custom rules and runtime verification with TREX. We evaluate outcomes rather than assuming those capabilities are absent.