feat(dashboard): consume the FP-judge triage overlay - #33
Conversation
Closes the deferred gap from #31: build_dashboard.py rendered the raw audit and ignored triage_class, so apply_verdicts' judged_fp findings were never retired. - collect() auto-prefers <DATA_DIR>/findings-triaged.json (else raw findings.json), reads each finding's triage_class/verdict, and emits a triage dimension + reason. - UI: KPIs lead with the real/uncertain/unjudged/judged-FP split; a triage chip filter (visibility toggles) and a Triage table column with the verdict reason on hover; judged_fp is retired from every aggregate by default (counted in a KPI, revealable via its chip) per verdict-contract.md §3. - Backward compatible: with no overlay, renders exactly as before. - CI smokes both paths: raw render, then apply_verdicts merge + triaged render, via a committed synthetic overlay viz/fixtures/fp-verdicts.json (its generated_from pins the fixture digest; the derived findings-triaged.json is gitignored). - STATUS.md: mark dashboard-consumption done. Verified locally: raw path shows no triage UI; triaged path -> triaged=true, counts {real:2,uncertain:2,unjudged:9,judged_fp:1}, Triage column + chips present, run_id provenance carried; apply_verdicts --selftest green; stale overlay rejected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFSpD9yXgXePrFFX62XZ53
|
Warning Review limit reached
Next review available in: 10 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughAdds a CI-driven FP-judge triage overlay: a synthetic verdicts fixture, an ChangesFP-judge triage overlay
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant CI as CI Workflow
participant AV as apply_verdicts.py
participant BD as build_dashboard.py
participant Data as findings-triaged.json
participant UI as Dashboard UI
CI->>BD: build_dashboard.py viz/fixtures (raw findings)
CI->>AV: apply_verdicts.py findings.json fp-verdicts.json
AV->>Data: write findings-triaged.json
CI->>BD: build_dashboard.py viz/fixtures (rerender)
BD->>Data: detect and load triaged findings
BD->>BD: compute triage_counts, verdict_summary
BD->>UI: embed triage fields, dims, KPIs
UI->>UI: renderTriageChips(), filteredRows() hides judged_fp
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d66cb0be5f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # present: it carries per-finding `triage_class`/`verdict`, so the dashboard can retire | ||
| # confident false positives. Fall back to the raw findings.json otherwise. | ||
| triaged_path = os.path.join(STS, "findings-triaged.json") | ||
| src_path = triaged_path if os.path.isfile(triaged_path) else os.path.join(STS, "findings.json") |
There was a problem hiding this comment.
When artifacts/findings-triaged.json is left over from an earlier judge run and Run-Audit.ps1 refreshes only artifacts/findings.json, this unconditional auto-preference bypasses the staleness guard in apply_verdicts and renders the old triage as current truth. In that state judged_fp rows from the previous audit are still retired from the dashboard even though verdict_summary.generated_from no longer matches the current raw findings, which is exactly the stale-overlay case the contract says must be refused.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Real hole — fixed in dd54336. The consumer now re-applies the FP-judge staleness rule: findings-triaged.json is preferred only when its verdict_summary.generated_from equals sha256(findings.json) (same check apply_verdicts.verify_fresh enforces). A leftover/stale overlay (e.g. Run-Audit.ps1 refreshed findings.json but not the overlay) is ignored — it falls back to raw findings.json and prints a warning — so an old triage is never rendered as current truth.
Verified: fresh overlay → preferred; mutate findings.json after judging → overlay ignored + warning.
Generated by Claude Code
| raw = open(src_path, encoding="utf-8").read() | ||
| # content digest = the audit run's identity for the trend series (below). | ||
| digest = hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16] |
There was a problem hiding this comment.
When a triaged file exists, raw now contains findings-triaged.json, so the history digest includes judge metadata, verdict reasons, output ordering, and threshold changes rather than just the audit findings. Running the documented raw render followed by apply_verdicts and a triaged render for the same audit will add a second trend point, and rerunning the judge with different overlay content will look like a new audit run even if findings.json did not change.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed — fixed in dd54336. The trend digest is now always sha256 of the raw findings.json bytes, independent of which file is rendered. So the documented raw-then-triaged render of one audit collapses to a single trend point (identical digest), and re-running the judge with different overlay content no longer looks like a new audit run. The digest is the audit's identity, not the overlay's.
Verified: raw render + triaged render of the same audit → one history point (no double-count).
Generated by Claude Code
… on raw findings Addresses Codex review on #33: - P1 (staleness): auto-preferring findings-triaged.json bypassed the FP-judge staleness guard — a leftover overlay (Run-Audit refreshed findings.json but not the overlay) would render an old triage as current truth. Now the consumer re-checks verdict_summary.generated_from == sha256(findings.json) and ignores a stale overlay (falls back to raw findings + a warning), same rule as apply_verdicts. - P2 (trend identity): the history digest was taken from the rendered file, so a triaged render (judge metadata/reasons/threshold) diverged from the raw render of the same audit and double-counted trend points. Digest is now always sha256 of the raw findings.json bytes — the audit run's identity, independent of the overlay. Verified: fresh overlay -> preferred; raw+triaged render of one audit -> single trend point; stale overlay -> ignored with warning; CI triaged render still shows the Triage view. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DFSpD9yXgXePrFFX62XZ53
|
@coderabbitai review Generated by Claude Code |
|
✅ Action performedReview finished.
|
2 participants
What & why
Closes the gap deferred from #31 (the original Codex P2):
build_dashboard.pyrendered the raw audit and ignoredtriage_class, soapply_verdicts'judged_fpfindings were never retired — the FP-judge overlay had no effect on the dashboard. Per the roadmap (docs/fp-judge/STATUS.md, Phase 2 step 2:apply_verdicts --out findings-triaged.json→build_dashboard.py), the dashboard now actually consumes the triage.The real
o7 judgerun itself is stand-only (007's private harness); this PR is the domain-side consumer that makes its output render.Changes
viz/build_dashboard.pycollect()auto-prefers<DATA_DIR>/findings-triaged.jsonwhen present (else rawfindings.json), reads each finding'striage_class/verdict, and adds a triage dimension + interned verdictreasonto the per-finding rows.reasonon hover.judged_fpis retired from every aggregate by default — counted in a KPI, revealable via its chip — perverdict-contract.md§3 ("counted, not hidden").CI / fixtures
viz/fixtures/fp-verdicts.json— a committed synthetic overlay (itsgenerated_frompins the fixture digest) exercising every triage class..github/workflows/ci.yml— the smoke now renders the raw path, then runsapply_verdicts(also smoking its staleness guard + merge) and re-renders the triaged path..gitignore— the derivedviz/fixtures/findings-triaged.json(overlay is committed; this is generated).Docs
docs/fp-judge/STATUS.md— mark dashboard consumption done.Design decisions (confirmed with maintainer)
findings-triaged.json, fall back tofindings.json— no change to the roadmap/CI invocation shape.How verified
Locally, reproducing the CI job:
build_dashboard.py viz/fixtures, no overlay) → no triage UI; identical to prior behavior.apply_verdicts --out …then re-render) →triaged=true, counts{real:2, uncertain:2, unjudged:9, judged_fp:1}, Triage column + chips present,verdict_summary.run_idcarried through.apply_verdicts --selftestgreen; a drifted fixture correctly rejected by the staleness guard (exit 2).🤖 Generated with Claude Code
https://claude.ai/code/session_01DFSpD9yXgXePrFFX62XZ53
Generated by Claude Code
Summary by CodeRabbit
New Features
Bug Fixes