Skip to content

P-022 step 7a: add dual-engine shadow mode and zero-diff reproduction artifacts #260

Description

@PhysShell

Parent: #250
Recommended tier: strong/local-agent collaboration
Blocked by: #259
Preferably after: #256

Goal

Run the Python and Rust cores against the exact same OwnIR bytes, normalize their outputs, and fail on any unexplained difference while preserving a standalone reproduction artifact.

This is the safety gate before Rust CLI/default-engine work.

Engine modes

Provide internal/dev surfaces equivalent to:

python
rust
compare

The final flag names may differ, but semantics must not:

  • python: current reference behavior;
  • rust: Rust bridge/core only;
  • compare: extract OwnIR once, feed identical bytes to both engines, compare normalized outputs.

Python remains the public default in this issue.

Same-input invariant

Compare mode must:

  1. Produce or load OwnIR exactly once.
  2. Hash the raw OwnIR bytes.
  3. Feed those exact bytes to Python and Rust.
  4. Record the hash in the diff artifact.

Running the extractor twice is forbidden because frontend nondeterminism would contaminate the comparison.

Normalized comparison

Compare at least:

path
line
code
severity
message
subject
resource_kind
ordered Evidence

When #256 is complete, also compare:

canonical .ownreport.json
canonical SARIF

Classify differences as:

  • Python-only;
  • Rust-only;
  • changed diagnostic;
  • ordering-only;
  • explicitly ignored volatile metadata;
  • unexplained.

Reproduction artifact

On mismatch, emit a self-contained artifact containing:

{
  "input": "...",
  "ownir_sha256": "...",
  "python_engine": "...",
  "rust_engine": "...",
  "python_only": [],
  "rust_only": [],
  "changed": [],
  "ordering_only": [],
  "ignored_metadata": [],
  "unexplained": []
}

Include the OwnIR input or a stable artifact reference sufficient to replay without the original source repository.

Test matrix

Run compare mode over:

The five-repository sweep may use a local/corpus-capable agent, but its commands, commits and output artifacts must be recorded.

Reliability controls

  • Rust panic/crash is a hard failure, never a Python-fallback success.
  • Python crash is also visible and classified.
  • Timeout/cancellation behavior is explicit.
  • Ordering instability is treated as a bug.
  • Re-running the same OwnIR produces byte-identical normalized outputs.
  • Compare mode never mutates user source or caches in a way that changes the second engine's input.

CI strategy

Fast PR CI:

  • committed fixtures;
  • C# samples;
  • generated compact controls.

Scheduled/manual broader gate:

  • five pinned OSS repos;
  • large-solution controls;
  • reproduction artifacts uploaded on failure.

Guardrails

  • No Rust-default cutover.
  • No removal of Python.
  • No new diagnostics.
  • No separate extraction per engine.
  • No normalization that deletes semantic fields.
  • No automatic acceptance of a diff because one engine produced fewer findings.

Acceptance

  • Compare mode feeds identical OwnIR bytes to both engines.
  • Fast CI reports zero unexplained differences.
  • Five-repository sweep reports zero unexplained differences.
  • Mismatches produce replayable artifacts.
  • Rust crashes cannot be hidden by fallback.
  • Python remains the default engine.
  • Commands and normalization policy are documented.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions