You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- Update D1: gVisor resolved in #6093; Kata Containers still open
- Add B11: silent repair failure causes exit 1, fixed in #6072
- Add B11 error-string lookup entry in all three catalog files
- Update Known unresolved items: narrow D1 to Kata only
- Update symptom mapping and known-gaps step in both doctor files
- Update CI test assertions for D1 and B11
Closes#6103
Copy file name to clipboardExpand all lines: .github/agents/self-hosted-runner-doctor.md
+8-3Lines changed: 8 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -81,6 +81,7 @@ Prefer the narrowest match. Examples:
81
81
-`FATAL: http_port: IPv6 is not available` → B3
82
82
-`No CA certificates were loaded from the system` in chroot on RHEL/Fedora/Amazon Linux → B9 (missing /etc/pki/ mount)
83
83
-`[WARN] Rootless artifact permission repair failed` with each attempt taking ~30 s (registry timeout, not instant) → B10 (compound tag@digest ref causes Docker to attempt GHCR manifest verification even under `--pull never`)
84
+
-`[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success → B11 (repair container stderr not captured; chroot-home removal failure escalated to fatal exit)
84
85
-`none of the git remotes correspond to the GH_HOST environment variable` → C4
85
86
-`400 bad request: Authorization header is badly formatted` → C3
86
87
-`400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
@@ -89,7 +90,7 @@ Prefer the narrowest match. Examples:
89
90
90
91
### 4. Check for known gaps and notable fixes
91
92
92
-
If the best match is one of the known open gaps (gVisor/Kata runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
93
+
If the best match is one of the known open gaps (Kata Containers runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
93
94
94
95
A13 / github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 — ARC/DinD split-fs base-userland staging is **fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. The `sysroot-stage` init container copies the signed `build-tools` image filesystem into a `sysroot` volume mounted at `/host:ro` before the agent starts.
95
96
@@ -103,6 +104,8 @@ B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible i
103
104
104
105
B10 / github/gh-aw-firewall#6025 — `fixArtifactPermissionsForRootless()` compound `tag@digest` ref timeout is **fixed** in AWF version including github/gh-aw-firewall#6025. `resolvePermFixerImageRef()` now returns tag-only refs, eliminating registry I/O during `--pull never` repair.
105
106
107
+
B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and chroot-home removal failure escalated to exit 1 is **fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**. Stderr is now included in the `[WARN]` message; chroot-home removal failure is downgraded to `debug`.
108
+
106
109
C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.
107
110
108
111
C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872.
@@ -184,6 +187,7 @@ Establish these facts before matching a failure mode:
184
187
| B8 |`EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner**| A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. |**Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. |`ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` |github/gh-aw-firewall#5983|
185
188
| B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 |
186
189
| B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025 |
190
+
| B11 | Agent task succeeds but AWF process exits with code 1; logs show `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` and `[WARN] Failed to remove chroot home directory after permission repair: Error: EACCES`; no actionable detail visible — repair container stderr was silently discarded | `fixArtifactPermissionsForRootless()` ran a `docker run` for chown/chmod but did not capture or log stderr, so the root cause was opaque. `removeWorkDirectories()` then escalated the subsequent `EACCES` from `fs.rmSync` on remapped-UID files to a fatal exit, overriding the successful agent exit code. | **Fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured and included in the `[WARN]` message. Chroot-home removal failure is downgraded from `warn` to `debug` so it no longer propagates as a non-zero exit code — the post-step cleanup via `sudo` handles it. Upgrade to AWF including github/gh-aw-firewall#6072. | `[WARN] Rootless artifact permission repair failed` with no additional detail (no stderr), followed by `Command completed with exit code: 1` even though the agent log shows success | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 |
187
191
188
192
## Category C — GHES / GHEC / `ghe.com`
189
193
@@ -202,7 +206,7 @@ Establish these facts before matching a failure mode:
202
206
203
207
| ID | Signal | Root cause | Status | Probe | Citations |
204
208
|---|---|---|---|---|---|
205
-
| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model | Known unresolved research area | `docker info | grep -i runtime`; inspect NAT rules in `awf-iptables-init`|#3264|
209
+
| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model in their default configuration |**gVisor (`runsc`) resolved** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor`; AWF detects `runsc` and applies the appropriate seccomp/capability profile. **Kata Containers** remain an unresolved research area.| `docker info | grep -i runtime`; use `--container-runtime gvisor` with AWF including github/gh-aw-firewall#6093|github/gh-aw-firewall#3264, github/gh-aw-firewall#6093|
206
210
| D2 | cli-proxy fails on IPv6-enabled runners | The tunnel was bound only to `127.0.0.1`| Fixed by dual-stack binding | Check `ss -tlnp` inside `awf-cli-proxy`|#4626|
207
211
| D3 |`--enable-dind` still exists after DinD removal | Legacy flag cleanup is incomplete | Known unresolved cleanup item | `awf --help | grep enable-dind` |#1727|
208
212
| D4 | Enterprise LLM gateway needs an injected auth header | API proxy lacks a user extension point for that hop | Known unresolved proposal | No general probe; capture the required header flow in the report |#4849|
@@ -218,6 +222,7 @@ Establish these facts before matching a failure mode:
218
222
|`unknown shorthand flag: 'd' in -d` from `docker compose up -d` on ARC/DinD | A14 |
219
223
|`Rootless artifact permission repair failed for .../sandbox/firewall/logs` on ARC/DinD | A15 |
220
224
|`[WARN] Rootless artifact permission repair failed` with each attempt taking ~30 s (timeout, not an instant error), followed by cascading `EACCES: permission denied` on chroot-home removal | B10 |
225
+
|`[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success | B11 |
221
226
|`FATAL: http_port: IPv6 is not available` or `Bungled ... [::]:3128`| B3 |
222
227
|`node: command not found` on self-hosted or DinD | A8 or B4 |
223
228
|`none of the git remotes correspond to the GH_HOST environment variable`| C4 |
@@ -240,7 +245,7 @@ Establish these facts before matching a failure mode:
240
245
241
246
Flag these explicitly instead of implying there is a complete fix:
242
247
243
-
- D1 / #3264 — gVisor and Kata compatibility research
248
+
- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093)
244
249
- D3 / #1727 — lingering `--enable-dind` cleanup
245
250
- D4 / #4849 — enterprise header injection extension point
246
251
- C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes
Copy file name to clipboardExpand all lines: .github/workflows/self-hosted-runner-doctor.md
+4-1Lines changed: 4 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -109,6 +109,7 @@ Prefer the narrowest match. Examples:
109
109
-`FATAL: http_port: IPv6 is not available` → B3
110
110
-`No CA certificates were loaded from the system` in chroot on RHEL/Fedora/Amazon Linux → B9 (missing /etc/pki/ mount)
111
111
-`[WARN] Rootless artifact permission repair failed` with each attempt taking ~30 s (registry timeout, not instant) → B10 (compound tag@digest ref causes Docker to attempt GHCR manifest verification even under `--pull never`)
112
+
-`[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success → B11 (repair container stderr not captured; chroot-home removal failure escalated to fatal exit)
112
113
-`none of the git remotes correspond to the GH_HOST environment variable` → C4
113
114
-`400 bad request: Authorization header is badly formatted` → C3
114
115
-`400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
@@ -117,7 +118,7 @@ Prefer the narrowest match. Examples:
117
118
118
119
### 4. Check for known gaps and notable fixes
119
120
120
-
If the best match is one of the known open gaps (gVisor/Kata runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
121
+
If the best match is one of the known open gaps (Kata Containers runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
121
122
122
123
A13 / github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 — ARC/DinD split-fs base-userland staging is **fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. The `sysroot-stage` init container copies the signed `build-tools` image filesystem into a `sysroot` volume mounted at `/host:ro` before the agent starts.
123
124
@@ -131,6 +132,8 @@ B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible i
131
132
132
133
B10 / github/gh-aw-firewall#6025 — `fixArtifactPermissionsForRootless()` compound `tag@digest` ref timeout is **fixed** in AWF version including github/gh-aw-firewall#6025. `resolvePermFixerImageRef()` now returns tag-only refs, eliminating registry I/O during `--pull never` repair.
133
134
135
+
B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and chroot-home removal failure escalated to exit 1 is **fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**. Stderr is now included in the `[WARN]` message; chroot-home removal failure is downgraded to `debug`.
136
+
134
137
C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.
135
138
136
139
C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872.
0 commit comments