feat: interactive STS dashboard + local-AI fixer tier (verify→revise) - #6
Conversation
Build a self-contained, offline interactive HTML dashboard from the validated sts_audit/ artifacts. viz/build_dashboard.py reads findings.json (72,569 raw findings) + the health-report.md module pain table and renders, via Plotly: where-it-hurts treemap, by-category/tool/top-rules bars, by-source coloured by whether it ships a code fix (the 85% story), and OWN fixer shape coverage. Inlines a vendored viz/plotly.min.js when present (single 4.5MB file, no internet) else falls back to the CDN. Verified headless (Chromium): 6 charts, zero console errors. Generated html/png + vendored lib are git-ignored (reproducible from the README one-liner). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
Live theme switcher in the dashboard header — re-skins both the CSS (glassmorphism cards, gradient title, glow, per-theme background) and the Plotly palette (colorway + treemap colorscale) on click via Plotly.react. Midnight (blue), Terminal (neon-green mono + scanlines), Aurora (purple/teal/pink gradients). Verified headless: 6 charts render in all three, zero console errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
Add AiFixApplier: a pluggable Applier that asks a LOCAL LLM to rewrite a window around findings mechanical fixers can't touch (T3 detect-only, T4-refused). The model is NOT trusted — riding the same wrapper, its proposal is verified by the no-new-findings re-audit, shown as a reviewable diff, gated to REVIEW (never auto-commit), and rolled back on regression. The LLM proposes; the audit + human judge (so it doesn't break the 'never AI-judged' discipline). - Local-only by design (STS is proprietary): OpenAI-compatible client, default Ollama (qwen2.5-coder); works with llama.cpp/LM Studio/vLLM too. MockLlmClient drives the same path in CI with no server. - cli: --applier ai --llm-url --model, with the REVIEW gate forced for AI. - tests (mock): prompt/parse, verified+gated apply, no-change skip, regression reject+revert. 5/5 ai + 25/25 own + 7/7 wrapper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
When a reaudit is supplied, AiFixApplier runs a per-finding verify->revise loop: propose -> apply candidate -> re-audit -> if the finding isn't cleared (or new ones appear) feed the failure back and the model revises, up to max_rounds. Every round still passes through the audit, so the loop only helps a weak local model converge — it never relaxes the trust model. Planning stays side-effect-free (candidates are written for the audit then the tree is restored; apply writes the memoized result). single-shot remains the default when no reaudit is given. - MockLlmClient now takes a list -> [bad, good] drives the loop in CI. - cli: --applier ai --max-rounds, wired with reaudit+before. - tests: loop converges on round 2, gives up after max_rounds (side-effect-free). 7/7 ai + 25/25 own + 7/7 wrapper, normal and -O. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR adds a local LLM AI fixing path for audit findings, wires it into the fix CLI with review-only gating and tests, and adds a standalone Plotly dashboard generator for STS audit outputs with docs and ignore rules. ChangesAI fix workflow
STS dashboard build
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c12a3174f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return None | ||
| with open(path, "w", encoding="utf-8") as fh: # let re-audit see the candidate | ||
| fh.write("".join(cand)) | ||
| after = self.reaudit(workdir) |
There was a problem hiding this comment.
When the verify→revise loop is used and the re-audit fails (for example ScriptReaudit exits nonzero or its output is missing), this call happens during dry_run after the candidate has already been written to the real worktree. Because the restore at the end of planning is not in a finally block and run_fix only rolls back after apply, the command can abort with unverified LLM edits left in the checkout, breaking the safety contract that planning is side-effect-free. Wrap the candidate write/re-audit in cleanup that restores the original/current contents before propagating the error.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@codex Fixed in 28a28a7. Wrapped the per-file planning body in try/finally so the original content is always restored before any error propagates — including when reaudit raises mid-loop (ScriptReaudit nonzero / missing output). Planning stays side-effect-free, so dry_run can never leave an unverified LLM edit in the worktree. Test: test_ai_plan_restores_tree_when_reaudit_raises (reaudit throws → dry_run re-raises, tree byte-for-byte unchanged).
Generated by Claude Code
There was a problem hiding this comment.
To use Codex here, create an environment for this repo.
The verify->revise loop writes a candidate to the real worktree before calling reaudit(workdir). If reaudit raises (e.g. ScriptReaudit exits nonzero / no output), the exception left _plan before the final restore, leaking an unverified LLM edit during dry_run. Wrapped the per-file planning body in try/finally so the original is always restored before the error propagates — planning stays side-effect-free. +test (reaudit raises -> tree unchanged). 8/8 ai, incl -O. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@fix/fixarm/ai_fix.py`:
- Around line 143-157: The retry feedback in ai_fix.py is being carried across
findings through self._feedback, so one finding’s rejection message can leak
into the next finding’s first prompt. Make the feedback state local to each
finding by resetting or scoping it inside the per-finding repair flow in the
revise/propose path (using the relevant methods around _propose and the retry
loop), so each finding starts with no prior rejection context unless it was
generated for that same finding.
- Around line 148-155: The _revise() flow leaves the candidate written to disk
if self.reaudit(workdir) raises, which breaks the side-effect-free behavior of
_plan()/dry_run(). Update the _revise loop in ai_fix.py so the temporary
candidate write is always restored in a finally block around the re-audit call,
ensuring the original contents are put back whether re-audit succeeds, returns,
or throws; use the existing _revise, self.reaudit, diff_findings, and run_fix
flow as the anchor points.
- Around line 68-80: Restrict LocalLlmClient.base_url in the __init__ and
complete flow so only loopback/local endpoints are accepted; currently --llm-url
can point to a remote host and leak source code. Add validation in
LocalLlmClient to reject non-HTTP(S) schemes and any hostname other than
localhost, 127.0.0.1, or ::1 before constructing the urllib.request.Request in
complete.
In `@fix/fixarm/cli.py`:
- Around line 72-81: Thread args.line_tol through the AI verification path so
AiFixApplier uses the same tolerance as run_fix. Update the AI revise/verify
loop to pass the line tolerance into the introduced and still-present checks
instead of relying on exact-line matching and the hard-coded <= 2 threshold, so
proposals that only shift diagnostics are judged consistently.
In `@viz/build_dashboard.py`:
- Around line 78-79: The dashboard summary in build_dashboard is still using
hardcoded snapshot values for clusters, totals, fixable percentage, and related
UI text instead of deriving them from the loaded sts_audit artifacts. Update the
summary/card/footer/caption logic in build_dashboard and the helper sections
that feed those fields so they compute from the current findings/by_source data
rather than frozen literals, keeping the UI consistent with the generated
charts.
- Around line 258-259: The payload inserted into the HTML template from
build_dashboard.py is not safe for a script context because json.dumps(data) can
leave </script> intact and break out of the block. Update the HTML generation
path around HTML.replace and the %DATA% substitution to encode/escape the JSON
for embedding inside the <script> tag before it is inserted, so values in
module, rule, or category cannot terminate the script early.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7f266450-2f81-4f77-a4c2-f0a4f8eea028
📒 Files selected for processing (8)
.gitignoredocs/fix-arm.mdfix/README.mdfix/fixarm/ai_fix.pyfix/fixarm/cli.pyfix/tests/test_ai_fix.pyviz/README.mdviz/build_dashboard.py
…ived KPIs) ai_fix.py: - LocalLlmClient now rejects non-loopback endpoints (localhost/127.0.0.1/::1 only) so STS code can't be pointed at a remote LLM by mistake. - verify->revise loop: feedback is local to each finding (no leak across prompts); inner try/finally restores the base file whether re-audit returns or raises; thread line_tol through so the loop judges findings the same way the wrapper does. cli.py: pass --line-tol into AiFixApplier. viz/build_dashboard.py: - KPI tiles derive from the current sts_audit inputs (D.total, D.clusters.high, D.own_shapes_fixed) instead of frozen literals; cluster headline parsed from health-report.md. - escape </ in the embedded JSON payload so a "</script>" in data can't break out of the script context. Tests: 8/8 ai + 25/25 own + 7/7 wrapper. Dashboard re-verified headless (6 charts, no console errors). Addresses Codex and CodeRabbit review on PR #6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
fix/fixarm/ai_fix.py (1)
186-194: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winProcess per-file findings from bottom to top to keep line windows stable.
Line 194 updates
curafter each accepted rewrite, but Lines 187-188 keep computing later windows from original audit line numbers. If an earlier rewrite inserts/removes lines, the next finding can be prompted/spliced against the wrong region. Iterate in descending line order so edits lower in the file cannot shift pending windows above them.Proposed fix
- for f in fs: + for f in sorted(fs, key=lambda f: f.line, reverse=True):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@fix/fixarm/ai_fix.py` around lines 186 - 194, Process the findings in descending line order inside the loop in ai_fix.py so each later rewrite does not shift the line windows for earlier pending items; update the iteration over fs in the revision flow around _revise, cur, and occupied to process bottom-to-top and keep each prompt/splice aligned with the intended region.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@viz/build_dashboard.py`:
- Line 273: The footer text is using data['clusters']['total'] from the parsed
findings total, so the “raw findings clustered to N clusters” message can show
the wrong count. Update the footer generation in build_dashboard.py where the
dashboard template replacement happens to use the parsed candidate/high cluster
count instead of the findings count, and make sure the symbol handling the
cluster summary data feeds the footer with the correct cluster total.
---
Outside diff comments:
In `@fix/fixarm/ai_fix.py`:
- Around line 186-194: Process the findings in descending line order inside the
loop in ai_fix.py so each later rewrite does not shift the line windows for
earlier pending items; update the iteration over fs in the revision flow around
_revise, cur, and occupied to process bottom-to-top and keep each prompt/splice
aligned with the intended region.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b56b37fb-5af1-428d-8cc9-ff71ecc81247
📒 Files selected for processing (3)
fix/fixarm/ai_fix.pyfix/fixarm/cli.pyviz/build_dashboard.py
🚧 Files skipped from review as they are similar to previous changes (1)
- fix/fixarm/cli.py
…ding windows An accepted rewrite that changes a file's line count shifted the windows of findings still pending above it, because each window is computed from the finding's original audit line. Iterate findings in descending line order: edits lower in the file leave upper line numbers intact, keeping every window aligned. New test test_ai_plans_findings_bottom_to_top locks the proposal order (lower finding proposed first). 9/9 ai_fix tests green. Addresses CodeRabbit review on PR #6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLHz4cevQyXHKZnjzCqhoa
|
@coderabbitai re: the outside-diff finding on Note I did not apply the other finding from this run (footer cluster count, line 273) — replied inline explaining why it's a false positive: the headline's "15803 findings" is already the clustered count (3602 + 12201), so Generated by Claude Code |
|
Tip For best results, initiate chat on the files or code changes.
Also agreed on the dashboard note: if the (ᵔᴥᵔ)ﾉ ✏️ Learnings added
|
2 participants
Dashboard + AI fixer tier
Two threads built on the branch since #5.
1. Interactive STS dashboard (
viz/)viz/build_dashboard.pybuilds a self-contained, offline HTML dashboard from the validatedsts_audit/artifacts (72,569 findings + the module pain table): a where-it-hurts treemap, by-category/tool/top-rules bars, by-source coloured by whether it ships a code fix (the 85% story), and OWN-fixer shape coverage. Plotly is inlined (no CDN needed). Three live-switchable themes — Midnight / Terminal / Aurora — re-skin both the CSS and the Plotly palette on click. Verified headless (Chromium): 6 charts render in all themes, zero console errors. Generated HTML/PNG + vendored Plotly are git-ignored (reproducible from the README).2. AI fixer tier (
fix/fixarm/ai_fix.py)A pluggable
Applierthat asks a local LLM to propose patches for the residual mechanical fixers can't touch (T3 detect-only, T4-refused suggest-only). The model is not trusted — riding the same wrapper, its proposal is verified by the no-new-findings re-audit, shown as a reviewable diff, gated to REVIEW, and rolled back on regression. The LLM proposes; the audit + human judge (so it doesn't break "never AI-judged").qwen2.5-coder); works with llama.cpp / LM Studio / vLLM.MockLlmClientdrives the same path in CI.reaudit, each proposal is re-audited per round; on failure the reason is fed back and the model revises up to--max-rounds. Every round still passes through the audit — the loop only helps a weak model converge, never relaxes the trust model. A framework (LangChain/LangGraph) buys nothing here; the thin client is zero-lock-in.--applier ai --llm-url … --model … --max-rounds N(REVIEW gate forced).Test plan
🤖 Generated with Claude Code
Generated by Claude Code
Summary by CodeRabbit
--applier ai) using local LLM proposals with review-only gating and a bounded verify→revise loop (--max-rounds,--llm-url,--model).