ShipCheck documentation

Open Comparison lab from the dashboard sidebar. A study contains one target behavior per historical case, exact base/head commits, reviewed labels, tool configurations, evidence links, and recorded outcomes. Unknown costs or timings stay blank.

1. 01Define a protocol and include regressions plus negative controls.
2. 02Review labels using separate evidence. Keep later fixes and label tests outside detector inputs.
3. 03Record the same cases for each tool and configuration, including memory-enabled and memory-disabled ShipCheck reviews.
4. 04Review target matches and information boundaries, then inspect the default paired-case comparison.
5. 05Save and export the study. Publication uses a separate reviewed dataset; private saves never publish automatically.

The public measured-comparisons page links raw reports, source commits, separate label-validation evidence, call timing and token usage. First-party, Codex-assisted adjudication and public-model contamination remain limitations. Software test-suite pass counts are engineering checks, not detection-accuracy measurements.

Configure test evidence under a repository's protected behavior. Link an existing Vitest test path, an optional test-name pattern, and the exact stored claims its assertions exercise. A matching filename, retained string, or AI-generated plan does not prove behavior.

• A new scenario regression requires a passing base, failing head, matching test-file digest, report identity, and an explicit claim mapping.
• A failure on both revisions is pre-existing. Changed tests, empty runs, setup failures, missing reports and unsupported execution are inconclusive.
• Runtime support currently targets the versioned Vitest/npm workflow. Dependencies requiring install hooks, private package credentials or external services need a supported adapter.
• The workflow uses resource-limited containers, no application credentials, and disabled networking during test execution. Dependency installation still needs network access.
• The public ground-truth workflow validates historical labels separately; its later assertions are never supplied to the advisory detector.

ShipCheck is the decision layer for software change. It turns code, tests, history, safeguards, and team decisions into a living model of what the system is meant to do.

Traditional code review asks whether newly changed code looks correct. ShipCheck asks the larger question: does this change still belong in the system your team intended to build, and is there enough evidence to prove it?

• Detect meaningful behavior changes.
• Find security and authorization regressions.
• Detect weakened safeguards and validation.
• Identify test and regression-coverage gaps.
• Discover important existing repository behavior.
• Protect important behavior with Behavior Guards.
• Govern protected behavior with versioned Behavior Contracts.
• Execute eligible protected scenarios against both base and head revisions.
• Publish risk-aware GitHub merge checks with audited exceptions.
• Share repositories, contracts, policies, reports, and approvals in Team Workspaces.
• Map provider contracts to consumers across APIs, SDKs, events, and data boundaries.
• Require specific evidence before a behavior can be verified.
• Preserve uncertainty when the available proof is incomplete.
• Compare future pull requests against repository memory.
• Track how behavior changes over time.
• Mine Git history for important fixes worth protecting.
• Generate focused fixes and regression tests.
• Track findings across rescans.
• Run scans directly from GitHub with the /shipcheck pull-request command.
• Produce a risk-aware Ready, Review, or Do not ship verdict.

ShipCheck connects through its GitHub App. The integration allows ShipCheck to discover repositories, inspect pull requests, read repository context, respond to repository events, accept the /shipcheck pull-request command, and publish scan information back into the GitHub workflow.

1. 01Create or sign in to your ShipCheck account.
2. 02Choose the option to connect GitHub.
3. 03Install the ShipCheck GitHub App on your GitHub account or organization.
4. 04Select the repositories ShipCheck should be allowed to access.
5. 05Return to ShipCheck and synchronize your repositories.
6. 06Activate the repositories you want ShipCheck to analyze.

After the GitHub App is installed, ShipCheck synchronizes repositories from that installation into the Repositories page.

Repository synchronization and ShipCheck activation are separate concepts. A repository may be visible because GitHub granted ShipCheck access while remaining inactive for scanning until you enable it.

1. 01Open Repositories from the dashboard.
2. 02Synchronize GitHub if a newly authorized repository is not yet shown.
3. 03Select a repository.
4. 04Enable ShipCheck for that repository.
5. 05Configure repository-specific scan settings if necessary.

After connecting and activating a repository, you can start a pull-request scan from either the ShipCheck dashboard or directly from the GitHub pull request.

1. 01Connect and activate the repository.
2. 02Open one of the repository's pull requests inside ShipCheck.
3. 03Start the ShipCheck scan.
4. 04ShipCheck queues the analysis and displays scan progress.
5. 05Open the completed report to review its risk-aware verdict, contract decisions, evidence, uncertainty, behavior changes, findings, and available actions.

/shipcheck

ShipCheck can be triggered directly from a GitHub pull request. You do not need to leave the pull request and open the ShipCheck dashboard just to request another scan.

/shipcheck

1. 01Open an active pull request in a repository connected to ShipCheck.
2. 02Add a new pull-request comment containing /shipcheck.
3. 03GitHub sends the comment event to the ShipCheck GitHub App.
4. 04ShipCheck verifies that the repository is connected and that the commenter has sufficient repository permission.
5. 05ShipCheck queues a scan for the open pull request.
6. 06A ShipCheck GitHub comment is created and updated with the scan result.

The command is useful after pushing new commits, when requesting an on-demand behavioral review, or whenever you want to rerun ShipCheck while remaining inside the GitHub workflow.

ShipCheck combines deterministic checks with repository-aware reasoning. It does not analyze a pull-request diff as an isolated block of text.

• Changed files and pull-request diffs.
• Relevant surrounding implementation.
• Authentication and authorization boundaries.
• Validation logic.
• Callers and related helpers.
• Existing tests.
• Error handling.
• State transitions.
• Data semantics.
• Repository-specific safeguards.
• Protected Behavior Guards.
• Behavior Contract versions and evidence requirements.
• Differential base-versus-head runtime results.
• Mapped downstream repositories and consumer relationships.
• Production Sentinel incidents and approved incident regression scenarios.
• Relevant Git history.

This additional repository context helps ShipCheck distinguish meaningful behavior changes from ordinary implementation changes.

ShipCheck processes scans asynchronously and reports progress while analysis is running. While processing, the scan progress page reports the state of the analysis instead of requiring the browser request to remain open for the entire scan.

When processing completes, ShipCheck redirects you toward the completed report where the full analysis can be inspected.

A completed scan combines conventional findings with the verification result, criticality, enforcement, lifecycle, and approval state of every affected Behavior Contract. The numeric score remains useful context, but the verdict is no longer derived from score alone.

The report shows the policy version, contract decisions, evidence, citations, contradictions, and remaining uncertainty behind the verdict. The verdict is still a review signal rather than a substitute for engineering judgment.

Affected-area categories and behavior-change categories are intentionally separate. The first says what area of the software is involved; the second says what kind of behavior changed.

Behavior Discovery identifies meaningful rules that already exist inside the repository without requiring developers to manually document everything first.

• Authorization guarantees.
• Validation rules.
• Business rules.
• Error-handling expectations.
• Data semantics.
• Existing safeguards.
• User-facing behavioral expectations.

Discovered behaviors appear as candidates. A developer can protect a useful candidate or dismiss one that should not become persistent repository memory.

A Behavior Guard is a persistent expectation about what the repository is supposed to keep doing.

Only administrators may delete a tenant.

Expired sessions must always be rejected.

Course codes are normalized before duplicate detection.

Once protected, future pull requests can be evaluated against the behavior and its supporting evidence rather than relying on each reviewer to rediscover the rule from scratch.

Protected behavior
        ↓
Pull request changes code
        ↓
ShipCheck verifies the behavior
        ↓
Preserved / Changed / Violated

A Behavior Guard identifies what the repository must preserve. Behavior Contracts V2 adds the governance needed to decide how strongly that expectation should affect verification and the final ship decision.

• Lifecycle: proposed, active, changing, deprecated, or retired.
• Criticality: advisory, important, critical, or regulated.
• Enforcement: observe, warn, require review, or block merge.
• Owner and approval state.
• Structured preconditions, invariants, scenarios, and observables.
• Required runtime, test, static, historical, or manual evidence.
• Evidence freshness and supersession metadata.

Behavior Contract v3

Criticality     Critical
Enforcement     Block merge
Approval        Approved
Invariant       Unauthorized cancellation never changes state.
Required proof  Runtime + test

The Behavior Impact Graph selects contracts connected to a pull request. Higher-criticality contracts receive priority, while deprecated and retired contracts are not scheduled for active verification.

• The planner converts the contract into falsifiable invariants and scenarios.
• Read-only tools can retrieve files, look up references, discover tests, and inspect historical evidence.
• A strict call budget bounds the investigation.
• Runtime, code-presence, test-presence, and investigation results are normalized into evidence.
• Coverage, citations, source diversity, contradictions, and required evidence are evaluated by policy.

A source-code diff describes which lines changed. Behavioral Diff describes what those changes mean for the software.

BEFORE

Only project owners can update a project.

AFTER

Any authenticated user can update the project.

RESULT

Authorization behavior changed.

Not every behavior change is a bug. Product requirements change, old restrictions are intentionally removed, and APIs evolve.

ShipCheck distinguishes likely unintentional changes, unclear changes, and likely intentional changes. Developers can explicitly mark supported behavior changes as intentional.

Behavior History gives important software expectations their own timeline, separate from the raw Git history of files.

Introduced
    ↓
Preserved
    ↓
Changed intentionally
    ↓
Violated
    ↓
Restored

Git remembers how source code changed. Behavior History is designed to remember how expected software behavior changed.

ShipCheck can inspect Git history to determine where a protected behavior appears to have first become true.

• Source commit.
• Historical date.
• Relevant implementation file.
• Related implementation and test evidence.
• Cross-file origin evidence.
• Confidence score.
• Explanation of why the commit established the behavior.

Origin tracing is not limited to the exact file where a Behavior Guard was discovered. ShipCheck can inspect related helpers, authorization logic, tests, and implementation history.

Some of the strongest evidence for what a repository must preserve is hidden inside bugs the team already fixed.

fix duplicate charges

prevent unauthorized access

reject expired sessions

stop duplicate submissions

ShipCheck can mine repository history for these types of fixes and convert supported fixes into candidate behaviors.

1. 01Open a repository in ShipCheck.
2. 02Find the Historical fixes section.
3. 03Choose Mine git history.
4. 04Review the candidate behavior, evidence, source commit, and confidence.
5. 05Protect the candidate if it represents behavior the repository should continue preserving.
6. 06Dismiss candidates that should not become repository memory.

For eligible runtime Behavior Guards, ShipCheck runs the same scenario against the pull request's base commit and head commit. Comparing both executions gives the verdict direct evidence about what the change introduced.

After a pull-request scan completes, ShipCheck evaluates the repository's merge policy and publishes a GitHub check. Ready verdicts pass, uncertainty can remain visible as a warning, and review-required or do-not-ship verdicts can prevent merge.

• Choose whether Review before shipping blocks the check.
• Require write, maintain, or admin GitHub permission to approve an exception.
• Limit how long an exception remains active.
• Choose whether regulated Behavior Contracts may ever be waived.
• Retain policy, verdict, actor, reason, expiry, and revocation as audit evidence.

A Team Workspace gives an engineering group a shared view of repository behavior while connected repositories remain owned by the users who added them.

• Invite teammates through a seven-day link tied to a verified email address.
• Share repositories without transferring GitHub ownership or installation access.
• Assign owners to Behavior Contracts and apply organization policy templates.
• Require another administrator to review pending governed contract changes.
• Inspect shared scans and the latest workspace audit events.

The Organization tab in a Team Workspace is the control plane for every shared repository. Owners and administrators save one versioned policy that ShipCheck synchronizes to active Behavior Contracts and repository merge-governance settings.

1. 01Open a Team Workspace and choose Organization.
2. 02Choose workspace-wide or restricted repository access.
3. 03Set the Behavior Contract defaults and approval mode.
4. 04Set the organization merge-governance and exception policy.
5. 05List protected deployment environments and choose their deployment gate.
6. 06Apply the policy and review its synchronized contract changes in Contracts and its audit records in Audit.

Inside a Team Workspace, map a provider Behavior Contract to another shared repository that consumes it. The relationship can represent an API consumer, SDK dependency, event consumer, or data contract, with an optional consumer Behavior Contract and integration note.

Every provider scan stores the downstream result. At-risk, violated, and inconclusive relationships become findings and contribute to the risk-aware ship verdict. Workspace reports show the consumer, relationship, message, and evidence behind the decision.

A Production Sentinel incident is evidence that protected behavior drifted after deployment. ShipCheck can turn that evidence into a reviewable proposal containing the incident cause, its resolution, a permanent invariant, and a regression scenario tied to the Sentinel's source Behavior Guard.

1. 01Open Production Sentinels from the affected repository.
2. 02Find the open, acknowledged, or resolved incident and choose Capture learning.
3. 03Review the generated cause, resolution, invariant, and regression scenario.
4. 04Approve the proposal to add its protection to the source Behavior Contract, or dismiss it when the incident should not change repository policy.
5. 05Use the recorded contract version to trace the new protection back to its production incident.

ShipCheck fingerprints findings so the same issue can be tracked between revisions instead of appearing as an unrelated new finding after every scan.

ShipCheck findings can include the affected file, source lines, code snippets, behavioral before/after information, impact, and supporting repository evidence.

Review the evidence before acting on a finding. A finding should be treated as a reasoned engineering claim that can be inspected, not as an unexplained AI score.

Eligible findings can include a suggested repair. Some repairs can be applied directly when ShipCheck has enough concrete context to construct a bounded edit plan.

• Repairs remain tied to a specific finding.
• Machine-applicable repair plans are bounded.
• A repair can include multiple related edits when the account plan supports multi-file fixes.
• ShipCheck limits generated repair plans to a small number of files and edits instead of allowing unrestricted repository rewrites.

A meaningful regression is more valuable when it produces durable protection. Eligible findings can be used to generate focused regression tests.

1. 01Open a finding that supports regression-test generation.
2. 02Ask ShipCheck to generate the regression test.
3. 03Review the proposed test and the environment information ShipCheck used.
4. 04Apply the generated test when it accurately captures the intended behavior.

This helps move the repository from detecting a bug once toward preventing the same class of regression in the future.

When ShipCheck correctly detects a behavior change but that change is intentional, it can be marked accordingly rather than being treated as an accidental regression.

This distinction is particularly important for behavioral memory because expected behavior must be allowed to evolve as the product evolves.

Completed scan findings include feedback controls. Feedback provides a direct way to communicate whether an analysis was useful or inaccurate.

Finding feedback is separate from marking a behavior change intentional. An intentional change can still be correctly detected by ShipCheck.

Each connected repository has its own ShipCheck page. It brings together repository status, pull requests, protected behaviors, Behavior Contract governance, scan activity, historical fixes, and repository-specific tools.

From a protected behavior, developers can configure lifecycle, criticality, enforcement, owner, approval, structured claims, evidence freshness, and required evidence. Every saved contract change requires a reason and produces a new immutable version.

ShipCheck includes a repository branch-comparison workflow. This is useful when you want to analyze the behavioral difference between two branches directly.

1. 01Open the repository.
2. 02Choose Compare branches.
3. 03Select the base branch.
4. 04Select the branch containing the changes.
5. 05Start the comparison.
6. 06Review the resulting ShipCheck analysis.

Repository settings allow ShipCheck to adapt to the conventions of individual codebases instead of applying exactly the same scanning configuration everywhere.

Ignored paths support simple safe glob matching. They are useful for generated files, vendored code, snapshots, documentation, or other repository-specific content that should not produce findings.

docs/**
generated/**
vendor/**
src/**/*.snapshot.ts
coverage

• * matches any characters except a path separator.
• ** matches characters across directories.
• ? matches one non-separator character.
• A plain directory name matches that directory and files beneath it.

A deterministic check that is useful for most repositories may not make sense everywhere. Add its rule ID to Ignored rules to suppress that rule for the selected repository.

The ShipCheck GitHub App receives supported GitHub webhook events so repository and pull-request activity can flow into ShipCheck without requiring every action to begin from the dashboard.

This allows ShipCheck scanning to fit the normal pull-request workflow rather than requiring teams to maintain a separate code-review process.

ShipCheck can publish scan information associated with a pull request so developers can move between GitHub and the full ShipCheck report.

When ShipCheck applies an eligible machine-generated fix or regression test, its GitHub integration can create the required repository changes as focused commits rather than asking the user to manually copy generated code.

ShipCheck's product entitlements are enforced by the application. During the current beta, the Free plan receives expanded beta usage.

Billing controls are available from the account settings area. ShipCheck supports starting checkout, changing a paid plan, scheduling cancellation, and resuming an eligible subscription.

1. 01Open Settings.
2. 02Open Billing.
3. 03Review your current plan and usage.
4. 04Choose the relevant plan or subscription action.
5. 05Complete any Stripe-hosted checkout step when required.

Account settings are distinct from repository settings. Repository settings control scanning behavior; account settings control the ShipCheck account itself.

The account settings area also includes account-deletion controls for users who want to remove their ShipCheck account.

ShipCheck includes an in-product feedback control on the public experience. During beta, feedback is especially useful for reporting confusing findings, missing behavior, workflow issues, or product suggestions.

For information about ShipCheck's own security posture and handling of repository-related access, see the dedicated Security page.