Add audit/ — Own.NET Audit static-layer orchestrator + aggregation pipeline#100
Merged
Conversation
…d Phase 0–1) First deliverable of the Own.NET Audit plan: the build-free static tier plus the full normalize -> score -> report aggregation, all testable on Linux CI with no target build and no external tools. Aggregation core (audit/aggregate/), each with embedded-fixture --selftest in the oracle_compare style: - normalize.py: reads every tool's SARIF through the SAME parse_sarif as oracle_compare (reused, not duplicated), maps (tool, ruleId) -> Plan.md §2 category via static/taxonomy/categories.yml. Splits the umbrella OWN001 code by its [resource: ...] tag so subscription/timer leaks land in cat. 2/3 instead of collapsing into IDisposable (cat. 1); labels OWN014 as region-escape. DevExpress findings are baseline-suppressed — counted in coverage, never hidden. Unmapped rules surface as pending taxonomy, not dropped. - score.py: generalizes oracle_compare.compare() into cross-tool agreement (>=2 tools at basename+line-window -> high confidence), category-driven P0-P3 severity, and a per-module pain heatmap answering "where does it hurt most". - report.py: markdown health report + machine JSON, with a load-bearing coverage section (NO-TOOL categories, suppressed count, unmapped rules). Static layer (audit/static/): - run_static.py orchestrator (build-free runners -> aggregate -> report), with a full end-to-end --selftest on embedded SARIF fixtures. - tools/owncheck.py (own-check.sh --format sarif, build-free; graceful when dotnet is absent), codeql.sh (build-mode=none, security-and-quality suite), and the build-required Windows runners roslyn_pack.ps1 / infersharp.sh (real skeletons, NO-TOOL exit when the tool is absent). - inject/ analyzer props+targets under MSBuild's recognized names, gated on /p:OwnAudit=true; taxonomy/categories.yml; config/profiles/desktop-wpf.yml. Decoupling: imports nothing from the ownlang core; the only in-repo seam is scripts/oracle_compare.parse_sarif (vendored on lift-out), own-check via CLI only. PyYAML is scoped to audit/ (requirements.txt) so the core suite stays zero-dep; a new CI job runs the four selftests (48 checks). The target itself is audited on a local Windows machine, never in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds an ChangesOwn.NET Audit Pipeline
Sequence Diagram(s)sequenceDiagram
participant Developer as Developer / CI
participant RunStatic as run_static.py
participant OwnCheck as owncheck.py
participant CodeQL as codeql.sh
participant FS as SARIF outputs
participant Normalize as normalize.py
participant Score as score.py
participant Report as report.py
Developer->>RunStatic: --target and profile
RunStatic->>OwnCheck: build-free SARIF
RunStatic->>CodeQL: build-free SARIF
RunStatic->>FS: collect roslyn/ and infersharp.sarif
RunStatic->>Normalize: normalize SARIF inputs
Normalize-->>RunStatic: findings + coverage
RunStatic->>Score: score findings
Score-->>RunStatic: clusters + heatmap + totals
RunStatic->>Report: render Markdown / JSON
Report-->>RunStatic: report.md and report.json
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b3a3029895
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
audit/static/tools/owncheck.py (1)
55-60: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAdd a timeout to the subprocess call.
own-check.shruns a Roslyn fact extractor over the whole target tree; without atimeout, a hang stalls the runner (and CI) indefinitely.subprocess.TimeoutExpiredshould be caught and surfaced as an honest unavailable/partial result, consistent with the continue-on-error discipline in the module docstring.♻️ Proposed change
- proc = subprocess.run(cmd, capture_output=True, text=True, check=False) - sarif_text = proc.stdout.strip() + try: + proc = subprocess.run(cmd, capture_output=True, text=True, check=False, + timeout=900) + except subprocess.TimeoutExpired: + status["reason"] = "own-check timed out" + return status + sarif_text = proc.stdout.strip()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@audit/static/tools/owncheck.py` around lines 55 - 60, The subprocess invocation in owncheck’s main flow lacks a timeout, so a hung own-check.sh can block the runner indefinitely. Update the subprocess.run call in the own-check logic to use a bounded timeout, and handle subprocess.TimeoutExpired by returning an unavailable/partial status with a clear reason, consistent with the module’s continue-on-error behavior. Keep the fix localized around the subprocess.run and the existing status["reason"] / return status path so the command still reports a graceful failure instead of hanging.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@audit/aggregate/normalize.py`:
- Around line 339-342: The normalize selftest summary in the self-check block is
using the wrong total count, causing the printed pass/fail ratio to be off by
one. Update the hardcoded check count used by the summary near the selftest loop
so it matches the actual number of distinct checks validated in this section,
and keep the failure reporting in the same normalize selftest path consistent
with that corrected total.
In `@audit/aggregate/report.py`:
- Around line 79-83: The high-confidence findings section in report generation
is truncating entries with high[:max_list] without signaling that more items
exist, which conflicts with the visible total in the header. Update the logic in
aggregate/report.py around the high-confidence list assembly in the report
builder so it mirrors the candidates overflow handling: keep the same slicing by
max_list, but append a “+N more” note when len(high) exceeds max_list. Use the
existing report formatting flow and the high/candidates list patterns in the
report output to locate and align the fix.
In `@audit/static/run_static.py`:
- Around line 195-236: The selftest in _selftest() hard-codes the total number
of checks as 8 even though there are 9 validation branches appending to fails,
so update the total to be derived from the actual number of checks (or otherwise
keep it in sync automatically) so the final summary printed by run_static
selftest matches the real count.
In `@audit/static/tools/infersharp.sh`:
- Around line 36-44: The fallback in infersharp.sh is invalid because
infersharpaction is a GitHub Action, not a CLI compatible with the infersharp
--sarif --results-dir invocation. Update the command check and execution flow
around the infersharp/infersharpaction handling so this script either requires
infersharp explicitly or branches into a separate action-based path instead of
calling infersharpaction as a shell command.
---
Nitpick comments:
In `@audit/static/tools/owncheck.py`:
- Around line 55-60: The subprocess invocation in owncheck’s main flow lacks a
timeout, so a hung own-check.sh can block the runner indefinitely. Update the
subprocess.run call in the own-check logic to use a bounded timeout, and handle
subprocess.TimeoutExpired by returning an unavailable/partial status with a
clear reason, consistent with the module’s continue-on-error behavior. Keep the
fix localized around the subprocess.run and the existing status["reason"] /
return status path so the command still reports a graceful failure instead of
hanging.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 8ac3407d-bf2f-4b84-a24e-0213a2ed78e5
📒 Files selected for processing (15)
.github/workflows/ci.ymlaudit/README.mdaudit/aggregate/normalize.pyaudit/aggregate/report.pyaudit/aggregate/score.pyaudit/config/profiles/desktop-wpf.ymlaudit/requirements.txtaudit/static/inject/OwnAudit.Directory.Build.propsaudit/static/inject/OwnAudit.Directory.Build.targetsaudit/static/run_static.pyaudit/static/taxonomy/categories.ymlaudit/static/tools/codeql.shaudit/static/tools/infersharp.shaudit/static/tools/owncheck.pyaudit/static/tools/roslyn_pack.ps1
Codex (4):
- OWN050 notes kept out of scored findings: own-check emits OWN050 ("cannot
verify — unresolved") as a SARIF note, a non-verdict. The taxonomy now lists it
under coverage_notes; normalize routes such rules to the coverage ledger
(analysis-skipped count) instead of scoring them, so they no longer surface as
phantom uncategorized P3 candidates.
- Roslyn ErrorLog now honors the runner's -Out: the injected props anchor the
per-project SARIF at $(OwnAuditOutDir) (default unchanged), roslyn_pack.ps1
passes /p:OwnAuditOutDir=<abs -Out>, and the targets MakeDir mirrors it.
- run_static picks up the Roslyn per-project SARIFs: it globs <out>/roslyn/*.sarif
(the props write $(MSBuildProjectName).sarif under roslyn/) instead of a single
fixed roslyn_pack.sarif, so analyzer-pack findings reach totals/coverage.
- infersharp.sh requires the infersharp CLI and drops the invalid infersharpaction
fallback (that is a GitHub Action, not a CLI with this interface); aligns the
invocation to Infer#'s documented infer-out/report.sarif output.
CodeRabbit (4 + 1 nitpick):
- report.py: high-confidence list now appends a "+N more" note past max_list,
matching the candidates overflow — no silent truncation against the header count.
- owncheck.py: subprocess.run gains a 900s timeout; TimeoutExpired returns a
graceful unavailable status (continue-on-error).
- selftest totals are now derived from a check() helper in all four modules
(normalize/score/report/run_static), so the printed pass ratio can no longer
drift from the real check count.
- infersharp.sh fallback removed (same as Codex item above).
Adds selftest coverage for the OWN050 routing and the roslyn/ pickup. All four
selftests green (19/11/15/11), ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
audit/static/tools/infersharp.sh (1)
25-31: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winGuard
--bin/--outagainst missing values.If either flag is passed without a value,
$2is read underset -u, which aborts with a shell error instead of your intended usage exit path.Suggested fix
while [[ $# -gt 0 ]]; do case "$1" in - --bin) bin="$2"; shift 2 ;; - --out) out="$2"; shift 2 ;; + --bin) + [[ $# -ge 2 ]] || { echo "infersharp.sh: --bin requires a value" >&2; exit 2; } + bin="$2"; shift 2 + ;; + --out) + [[ $# -ge 2 ]] || { echo "infersharp.sh: --out requires a value" >&2; exit 2; } + out="$2"; shift 2 + ;; -h|--help) sed -n '2,18p' "$0"; exit 0 ;; *) echo "infersharp.sh: unknown arg $1" >&2; exit 2 ;; esac done🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@audit/static/tools/infersharp.sh` around lines 25 - 31, The argument parser in infersharp.sh does not validate that --bin and --out are followed by values, so reading $2 can fail under set -u before the script reaches the intended usage/error path. Update the while/case handling in the main argument loop to check that a value exists before assigning bin or out, and route missing-value cases through the same usage/error exit path used by the script; use the existing infersharp.sh option parsing block as the place to apply the fix.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@audit/static/tools/infersharp.sh`:
- Around line 25-31: The argument parser in infersharp.sh does not validate that
--bin and --out are followed by values, so reading $2 can fail under set -u
before the script reaches the intended usage/error path. Update the while/case
handling in the main argument loop to check that a value exists before assigning
bin or out, and route missing-value cases through the same usage/error exit path
used by the script; use the existing infersharp.sh option parsing block as the
place to apply the fix.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 15e9cde6-d16c-47a1-9552-34d4c6299910
📒 Files selected for processing (10)
audit/aggregate/normalize.pyaudit/aggregate/report.pyaudit/aggregate/score.pyaudit/static/inject/OwnAudit.Directory.Build.propsaudit/static/inject/OwnAudit.Directory.Build.targetsaudit/static/run_static.pyaudit/static/taxonomy/categories.ymlaudit/static/tools/infersharp.shaudit/static/tools/owncheck.pyaudit/static/tools/roslyn_pack.ps1
🚧 Files skipped from review as they are similar to previous changes (6)
- audit/static/inject/OwnAudit.Directory.Build.targets
- audit/static/taxonomy/categories.yml
- audit/static/inject/OwnAudit.Directory.Build.props
- audit/static/tools/roslyn_pack.ps1
- audit/aggregate/report.py
- audit/aggregate/score.py
CodeRabbit (PR #100): infersharp.sh read $2 for --bin/--out without checking a value was supplied, so under `set -u` a flag with no argument aborted with a raw shell error instead of the intended usage exit. Added the `[[ $# -ge 2 ]]` guard that own-check.sh already uses, and applied the same to codeql.sh (--target/ --out/--db) which had the identical pattern. Each missing value now exits 2 with a clear message. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE
Owner
Author
|
@coderabbitai Good catch — fixed in Generated by Claude Code |
|
Tip For best results, initiate chat on the files or code changes.
|
|
Caution Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted. Error details |
PhysShell
pushed a commit
that referenced
this pull request
Jun 25, 2026
…) contract CodeRabbit (PR #101): - _coverage_section() now renders the analysis_skipped bucket (own-check OWN050 coverage notes) alongside total/kept/suppressed. It was counted in coverage() and carried in the SARIF run properties since #100, but the human markdown/HTML report never showed it — so notes silently vanished from the honesty ledger. Added a note=True fixture to the report selftest and assertions that both the markdown and HTML coverage sections surface it, keeping SARIF/HTML aligned. - Refreshed the run_static.aggregate() docstring to state it writes all four artifacts (report.md/json/sarif/html) and returns the four report_* paths. report selftest 34/34, run_static 14/14, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE
This was referenced Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First deliverable of the Own.NET Audit plan (
Plan.md, merged in #99): thebuild-free static tier plus the full normalize → score → report
aggregation. Everything here runs and is tested on Linux CI with no target build
and no external tools — the target itself is audited on a local Windows machine,
never in CI.
What's here
Aggregation core (
audit/aggregate/), each module with an embedded-fixture--selftestin theoracle_comparestyle (48 checks total):normalize.py— reads every tool's SARIF through the sameparse_sarifas
oracle_compare(reused, not duplicated), maps(tool, ruleId)→ a Plan.md§2 category via
static/taxonomy/categories.yml. The umbrellaOWN001codeis split by its
[resource: …]tag so subscription/timer leaks land incategories 2/3 instead of collapsing into IDisposable (cat. 1);
OWN014islabelled region-escape. DevExpress findings are baseline-suppressed — counted in
coverage, never hidden. Unmapped rules surface as pending taxonomy, not dropped.
score.py— generalizesoracle_compare.compare()into cross-toolagreement (≥2 tools at basename + line-window → high confidence), category-driven
P0–P3 severity, and a per-module pain heatmap ("where does it hurt most").
report.py— markdown health report + machine JSON, with a load-bearingcoverage/honesty section (NO-TOOL categories, suppressed count, unmapped rules).
Static layer (
audit/static/):run_static.pyorchestrator (build-free runners → aggregate → report), witha full end-to-end
--selfteston embedded SARIF fixtures.tools/—owncheck.py(own-check.sh --format sarif, build-free; gracefulwhen dotnet is absent),
codeql.sh(build-mode=none, security-and-qualitysuite), and the build-required Windows runners
roslyn_pack.ps1/infersharp.sh(real skeletons, NO-TOOL exit when the tool is absent).
inject/analyzer props+targets under MSBuild's recognized names, gated on/p:OwnAudit=true;taxonomy/categories.yml;config/profiles/desktop-wpf.yml.Decoupling (Plan.md §7)
Imports nothing from the
ownlangcore. The only in-repo seam isscripts/oracle_compare.parse_sarif(a pure SARIF reader, vendored on lift-out);own-check is consumed only via its CLI. PyYAML is scoped to
audit/(
requirements.txt) so the core test suite stays zero-dependency.CI
A new
audit-selftestsjob installs the audit-scoped PyYAML and runs the fourselftests (normalize + score + report + orchestrator). Tree-wide
ruffalreadycovers
audit/.Deferred (later phases)
HTML + merged-SARIF renderers; the runtime layer (FlaUI + ClrMD leak-harness,
duplicate-immutable detector); the AI-reviewer layer; feeding confirmed findings
back into the OwnLang corpus.
🤖 Generated with Claude Code
https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE
Generated by Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests