Skip to content

Add duplicate-immutable detector — the project's "gold" (Plan.md §2 cat. 11)#103

Merged
PhysShell merged 2 commits into
mainfrom
claude/brave-gauss-elvquy
Jun 25, 2026
Merged

Add duplicate-immutable detector — the project's "gold" (Plan.md §2 cat. 11)#103
PhysShell merged 2 commits into
mainfrom
claude/brave-gauss-elvquy

Conversation

@PhysShell

@PhysShell PhysShell commented Jun 25, 2026

Copy link
Copy Markdown
Owner

A heap full of identical immutable values — the same "Country" / unit / currency
string held by thousands of separate instances — is wasted memory that interning, a
flyweight, or a reference-by-id would collapse. Plan.md calls category 11 the
project's "gold". This adds that detector to the runtime layer; its findings flow
through the same normalize → score → report pipeline as everything else.

Testable core — pure Python, Linux-CI-gated

  • ingest.py refactored around a shared _runtime_sarif() core, with a new
    duplicate_detector_to_sarif() adapter: type/value-based findings stay
    file-level (reusing the Add merged-SARIF and HTML renderers — complete Phase 1 audit outputs #101 region handling), SARIF level warning (a P2
    memory/perf finding, not a correctness error), and a below-threshold finding
    (report: false) is dropped. --selftest extended to prove the dup path maps to
    category 11 and carries wastedBytes (ingest 15/15).
  • run_static.run() folds <out>/duplicate-detector.sarif into aggregation as
    a runtime tier (alongside leak-harness.sarif), so a cat-11 finding reaches the
    report through the orchestrator (run_static 15/15).
  • RUNTIME-DUP-IMMUTABLE was already mapped to category 11 (P2) in Start runtime layer: leak-harness ingest bridge + C# harness skeleton (Plan.md §4) #102 — no
    taxonomy change needed.

Windows / build-required skeleton — NOT CI-gated

  • audit/runtime/DuplicateDetector/ (net472 C#, ClrMD): walks a full dump,
    groups live strings by value, computes per-group wasted bytes
    ((count-1) * bytesPerInstance), and emits JSON findings above
    --min-wasted-bytes. Reuses the hardened procdump pattern from Start runtime layer: leak-harness ingest bridge + C# harness skeleton (Plan.md §4) #102 (drain pipes,
    bounded wait, exit-code check) and the ClrVersions guard. Strings first — the
    highest-value case; arbitrary immutable types (field-by-field equality) are a noted
    later refinement.

Tests

normalize 19/19 · score 11/11 · report 34/34 · run_static 15/15 · ingest 15/15 · ruff clean

End-to-end: a duplicate-detector JSON → SARIF → report renders the finding as
[P2 · duplicate-immutable] (category 11). C# verified by reading (it's
Windows/build-required and not compiled in CI).

🤖 Generated with Claude Code

https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE


Generated by Claude Code

Summary by CodeRabbit

  • New Features
    • Added a new runtime duplicate-immutable string detector (ClrMD-based) that analyzes full process dumps and reports estimated wasted memory.
    • Expanded runtime-to-SARIF conversion to generate SARIF for both leak-harness and duplicate-detector findings.
    • Static analysis now ingests duplicate-detector runtime SARIF when present.
  • Bug Fixes
    • Improved SARIF generation to correctly handle file-level vs line-level findings and to only report findings that meet the configured wasted-bytes threshold (unless overridden).

A heap full of identical immutable values (the same "Country"/unit/currency string
held by thousands of separate instances) is wasted memory that interning / a
flyweight / reference-by-id would collapse. This adds that detector to the runtime
layer, flowing through the same normalize -> score -> report pipeline.

Testable core (pure Python, Linux-CI-gated):
- ingest.py refactored around a shared _runtime_sarif() core, with a new
  duplicate_detector_to_sarif() adapter: type/value-based findings stay file-level,
  SARIF level warning (a P2 memory/perf finding), and a below-threshold finding
  (report:false) is dropped. --selftest extended to prove the dup path maps to
  category 11 and carries wastedBytes; ingest 15/15.
- run_static.run() folds <out>/duplicate-detector.sarif into aggregation as a
  runtime tier (alongside leak-harness.sarif), so a cat-11 finding reaches the
  report through the orchestrator; run_static 15/15.
- RUNTIME-DUP-IMMUTABLE was already mapped to category 11 (P2) in #102, so no
  taxonomy change is needed.

Windows / build-required skeleton (NOT CI-gated):
- audit/runtime/DuplicateDetector/ (net472 C#, ClrMD): walks a full dump, groups
  live strings by value, computes per-group wasted bytes ((count-1)*bytesPerInstance),
  and emits JSON findings above --min-wasted-bytes. Reuses the hardened procdump
  pattern (drain pipes, bounded wait, exit-code check) and the ClrVersions guard.
  Strings first (the highest-value case); arbitrary immutable types deferred.

All audit selftests green (normalize/score/report/run_static/ingest), ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE
@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: f831e256-ff8e-46bb-9cc0-75249d7a94bb

📥 Commits

Reviewing files that changed from the base of the PR and between 1c3ec01 and 5ff16dc.

📒 Files selected for processing (3)
  • audit/runtime/DuplicateDetector/Program.cs
  • audit/runtime/ingest.py
  • audit/static/run_static.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • audit/runtime/DuplicateDetector/Program.cs

📝 Walkthrough

Walkthrough

Adds a new DuplicateDetector runtime tool that analyzes duplicate strings from dumps, emits JSON findings, converts runtime outputs to SARIF, and teaches the static pipeline to load duplicate-detector results.

Changes

Duplicate detector runtime pipeline

Layer / File(s) Summary
Tool project, CLI, and docs
audit/runtime/DuplicateDetector/DuplicateDetector.csproj, audit/runtime/DuplicateDetector/Program.cs, audit/runtime/README.md
Defines the net472 x64 DuplicateDetector executable, parses its command-line options, and updates runtime documentation for the new tool and layout.
Dump acquisition and duplicate analysis
audit/runtime/DuplicateDetector/Program.cs
Selects or creates a dump, loads it with ClrMD, groups string values, computes wasted bytes, and writes duplicate findings as JSON.
SARIF conversion and runtime routing
audit/runtime/ingest.py
Converts leak-harness and duplicate-detector runtime JSON into SARIF, selects per-tool CLI behavior, and updates self-tests for the new duplicate-detector path.
Static runtime tier loading
audit/static/run_static.py
Loads duplicate-detector SARIF alongside leak-harness SARIF in the static runtime tier and extends the static self-test fixtures.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • PhysShell/Own.NET#100: Extends audit/static/run_static.py to ingest runtime SARIF, which this PR adds for the duplicate detector.
  • PhysShell/Own.NET#102: Updates audit/runtime/ingest.py for leak-harness runtime conversion, which this PR generalizes to the duplicate detector path.

Poem

I hopped through heaps with curious nose,
and found the strings in tidy rows.
I turned their waste to SARIF gold,
and tucked the dump where tales are told.
🐇✨ The burrow hums in tidy code.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: adding the duplicate-immutable detector for Plan.md category 11.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/brave-gauss-elvquy

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c3ec01601

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread audit/runtime/ingest.py Outdated
Comment thread audit/runtime/DuplicateDetector/Program.cs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
audit/static/run_static.py (1)

144-154: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a selftest for the new duplicate-detector pickup.

These lines introduce a second runtime SARIF path, but _selftest() still exercises only leak-harness.sarif. A typo in this filename/tool mapping would miss CI until a Windows run drops the new artifact.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@audit/static/run_static.py` around lines 144 - 154, Add a selftest for the
new duplicate-detector runtime pickup in _selftest() so both leak-harness.sarif
and duplicate-detector.sarif are covered. Update the selftest setup to
create/verify the duplicate-detector artifact and assert that run_static.py’s
runtime import logic maps the duplicate-detector tool name to the correct SARIF
filename, alongside the existing leak-harness check.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@audit/runtime/DuplicateDetector/Program.cs`:
- Around line 69-75: The grouping in DuplicateDetector.Program is using
obj.AsString(opts.MaxStringLength), which can collapse distinct long strings
into the same bucket. Update the logic around the val/groups.TryGetValue path to
group by the full string value instead of the truncated display value, and apply
truncation only when formatting the output fields like value/message.

In `@audit/runtime/ingest.py`:
- Around line 135-153: Make the CLI inputs in ingest.py mutually exclusive so
only one of --leak-harness or --duplicate-detector can be provided at a time.
Update the argparse setup near ap.add_argument / args parsing to place these
options in a mutually exclusive group (or otherwise validate and reject both
being set), and keep the existing ap.error path for invalid combinations so the
tool fails fast instead of silently choosing leak_harness over
duplicate_detector.

---

Nitpick comments:
In `@audit/static/run_static.py`:
- Around line 144-154: Add a selftest for the new duplicate-detector runtime
pickup in _selftest() so both leak-harness.sarif and duplicate-detector.sarif
are covered. Update the selftest setup to create/verify the duplicate-detector
artifact and assert that run_static.py’s runtime import logic maps the
duplicate-detector tool name to the correct SARIF filename, alongside the
existing leak-harness check.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 79c54600-0368-46e4-9cf1-33c5e1dd7e4b

📥 Commits

Reviewing files that changed from the base of the PR and between 9509635 and 1c3ec01.

📒 Files selected for processing (5)
  • audit/runtime/DuplicateDetector/DuplicateDetector.csproj
  • audit/runtime/DuplicateDetector/Program.cs
  • audit/runtime/README.md
  • audit/runtime/ingest.py
  • audit/static/run_static.py

Comment thread audit/runtime/DuplicateDetector/Program.cs Outdated
Comment thread audit/runtime/ingest.py
Address Codex + CodeRabbit findings on the duplicate-immutable detector:

- DuplicateDetector/Program.cs: group strings by their FULL value, not the
  --max-string-length-truncated read. Distinct long strings sharing a prefix
  (JSON/XML blobs) no longer collapse into one bogus group with inflated
  count/wastedBytes. Overlong strings key on length + a SHA-256 of the full
  content so the dictionary never retains large blobs; a short sample is kept
  only for the value/message display.

- ingest.py: the C# emits no source location for heap-wide duplicate groups, so
  every finding previously fell back to (type, line 0) and the scorer collapsed
  Country/Currency/... into a single cluster, corrupting totals/heatmap.
  Synthesize a UNIQUE per-value synthetic uri (heap://<type>/<n>-<slug>); all
  groups still roll up under one heap://<type> module. Harden the selftest with
  a realistic location-less fixture proving two distinct values stay two
  clusters.

- ingest.py: make --leak-harness / --duplicate-detector mutually exclusive so a
  stray flag fails fast instead of silently converting the wrong file.

- run_static.py: extend _selftest() to also drop a duplicate-detector.sarif and
  assert the runtime pickup folds it in, so a filename/tool-name typo fails on
  Linux CI, not only on a Windows run.

Selftests: ingest 18/18, run_static 16/16, normalize/score/report unchanged;
ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE
@PhysShell PhysShell merged commit 2b9098a into main Jun 25, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants