Skip to content

Commit f09fdb2

Browse files
authored
docs: update runner doctor catalog - D1 gVisor resolved, add B11
- Update D1: gVisor resolved in #6093; Kata Containers still open - Add B11: silent repair failure causes exit 1, fixed in #6072 - Add B11 error-string lookup entry in all three catalog files - Update Known unresolved items: narrow D1 to Kata only - Update symptom mapping and known-gaps step in both doctor files - Update CI test assertions for D1 and B11 Closes #6103
1 parent 7839686 commit f09fdb2

4 files changed

Lines changed: 27 additions & 6 deletions

File tree

.github/agents/self-hosted-runner-doctor.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ Prefer the narrowest match. Examples:
8181
- `FATAL: http_port: IPv6 is not available` → B3
8282
- `No CA certificates were loaded from the system` in chroot on RHEL/Fedora/Amazon Linux → B9 (missing /etc/pki/ mount)
8383
- `[WARN] Rootless artifact permission repair failed` with each attempt taking ~30 s (registry timeout, not instant) → B10 (compound tag@digest ref causes Docker to attempt GHCR manifest verification even under `--pull never`)
84+
- `[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success → B11 (repair container stderr not captured; chroot-home removal failure escalated to fatal exit)
8485
- `none of the git remotes correspond to the GH_HOST environment variable` → C4
8586
- `400 bad request: Authorization header is badly formatted` → C3
8687
- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
@@ -89,7 +90,7 @@ Prefer the narrowest match. Examples:
8990

9091
### 4. Check for known gaps and notable fixes
9192

92-
If the best match is one of the known open gaps (gVisor/Kata runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
93+
If the best match is one of the known open gaps (Kata Containers runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
9394

9495
A13 / github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 — ARC/DinD split-fs base-userland staging is **fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. The `sysroot-stage` init container copies the signed `build-tools` image filesystem into a `sysroot` volume mounted at `/host:ro` before the agent starts.
9596

@@ -103,6 +104,8 @@ B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible i
103104

104105
B10 / github/gh-aw-firewall#6025`fixArtifactPermissionsForRootless()` compound `tag@digest` ref timeout is **fixed** in AWF version including github/gh-aw-firewall#6025. `resolvePermFixerImageRef()` now returns tag-only refs, eliminating registry I/O during `--pull never` repair.
105106

107+
B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and chroot-home removal failure escalated to exit 1 is **fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**. Stderr is now included in the `[WARN]` message; chroot-home removal failure is downgraded to `debug`.
108+
106109
C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.
107110

108111
C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872.
@@ -184,6 +187,7 @@ Establish these facts before matching a failure mode:
184187
| B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 |
185188
| B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 |
186189
| B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025 |
190+
| B11 | Agent task succeeds but AWF process exits with code 1; logs show `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` and `[WARN] Failed to remove chroot home directory after permission repair: Error: EACCES`; no actionable detail visible — repair container stderr was silently discarded | `fixArtifactPermissionsForRootless()` ran a `docker run` for chown/chmod but did not capture or log stderr, so the root cause was opaque. `removeWorkDirectories()` then escalated the subsequent `EACCES` from `fs.rmSync` on remapped-UID files to a fatal exit, overriding the successful agent exit code. | **Fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured and included in the `[WARN]` message. Chroot-home removal failure is downgraded from `warn` to `debug` so it no longer propagates as a non-zero exit code — the post-step cleanup via `sudo` handles it. Upgrade to AWF including github/gh-aw-firewall#6072. | `[WARN] Rootless artifact permission repair failed` with no additional detail (no stderr), followed by `Command completed with exit code: 1` even though the agent log shows success | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 |
187191

188192
## Category C — GHES / GHEC / `ghe.com`
189193

@@ -202,7 +206,7 @@ Establish these facts before matching a failure mode:
202206

203207
| ID | Signal | Root cause | Status | Probe | Citations |
204208
|---|---|---|---|---|---|
205-
| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model | Known unresolved research area | `docker info | grep -i runtime`; inspect NAT rules in `awf-iptables-init` | #3264 |
209+
| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model in their default configuration | **gVisor (`runsc`) resolved** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor`; AWF detects `runsc` and applies the appropriate seccomp/capability profile. **Kata Containers** remain an unresolved research area. | `docker info | grep -i runtime`; use `--container-runtime gvisor` with AWF including github/gh-aw-firewall#6093 | github/gh-aw-firewall#3264, github/gh-aw-firewall#6093 |
206210
| D2 | cli-proxy fails on IPv6-enabled runners | The tunnel was bound only to `127.0.0.1` | Fixed by dual-stack binding | Check `ss -tlnp` inside `awf-cli-proxy` | #4626 |
207211
| D3 | `--enable-dind` still exists after DinD removal | Legacy flag cleanup is incomplete | Known unresolved cleanup item | `awf --help | grep enable-dind` | #1727 |
208212
| D4 | Enterprise LLM gateway needs an injected auth header | API proxy lacks a user extension point for that hop | Known unresolved proposal | No general probe; capture the required header flow in the report | #4849 |
@@ -218,6 +222,7 @@ Establish these facts before matching a failure mode:
218222
| `unknown shorthand flag: 'd' in -d` from `docker compose up -d` on ARC/DinD | A14 |
219223
| `Rootless artifact permission repair failed for .../sandbox/firewall/logs` on ARC/DinD | A15 |
220224
| `[WARN] Rootless artifact permission repair failed` with each attempt taking ~30 s (timeout, not an instant error), followed by cascading `EACCES: permission denied` on chroot-home removal | B10 |
225+
| `[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success | B11 |
221226
| `FATAL: http_port: IPv6 is not available` or `Bungled ... [::]:3128` | B3 |
222227
| `node: command not found` on self-hosted or DinD | A8 or B4 |
223228
| `none of the git remotes correspond to the GH_HOST environment variable` | C4 |
@@ -240,7 +245,7 @@ Establish these facts before matching a failure mode:
240245

241246
Flag these explicitly instead of implying there is a complete fix:
242247

243-
- D1 / #3264gVisor and Kata compatibility research
248+
- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093)
244249
- D3 / #1727 — lingering `--enable-dind` cleanup
245250
- D4 / #4849 — enterprise header injection extension point
246251
- C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes

.github/workflows/self-hosted-runner-doctor.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ Prefer the narrowest match. Examples:
109109
- `FATAL: http_port: IPv6 is not available` → B3
110110
- `No CA certificates were loaded from the system` in chroot on RHEL/Fedora/Amazon Linux → B9 (missing /etc/pki/ mount)
111111
- `[WARN] Rootless artifact permission repair failed` with each attempt taking ~30 s (registry timeout, not instant) → B10 (compound tag@digest ref causes Docker to attempt GHCR manifest verification even under `--pull never`)
112+
- `[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success → B11 (repair container stderr not captured; chroot-home removal failure escalated to fatal exit)
112113
- `none of the git remotes correspond to the GH_HOST environment variable` → C4
113114
- `400 bad request: Authorization header is badly formatted` → C3
114115
- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
@@ -117,7 +118,7 @@ Prefer the narrowest match. Examples:
117118

118119
### 4. Check for known gaps and notable fixes
119120

120-
If the best match is one of the known open gaps (gVisor/Kata runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
121+
If the best match is one of the known open gaps (Kata Containers runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
121122

122123
A13 / github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 — ARC/DinD split-fs base-userland staging is **fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. The `sysroot-stage` init container copies the signed `build-tools` image filesystem into a `sysroot` volume mounted at `/host:ro` before the agent starts.
123124

@@ -131,6 +132,8 @@ B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible i
131132

132133
B10 / github/gh-aw-firewall#6025`fixArtifactPermissionsForRootless()` compound `tag@digest` ref timeout is **fixed** in AWF version including github/gh-aw-firewall#6025. `resolvePermFixerImageRef()` now returns tag-only refs, eliminating registry I/O during `--pull never` repair.
133134

135+
B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and chroot-home removal failure escalated to exit 1 is **fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**. Stderr is now included in the `[WARN]` message; chroot-home removal failure is downgraded to `debug`.
136+
134137
C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.
135138

136139
C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872.

0 commit comments

Comments
 (0)