Skip to content

Commit 1f435b6

Browse files
lpcoxCopilot
lpcox
and
Copilot
authored
feat: add Docker sandbox smoke tests for Claude, Codex, and build workloads (#6145)
* feat: add Docker sandbox smoke tests for Claude, Codex, and build workloads Add three new smoke test workflows for the Docker sandbox (default AWF mode): - smoke-docker-sbx-claude.md: Docker sbx + Claude engine (haiku-4-5) - smoke-docker-sbx-codex.md: Docker sbx + Codex engine (gpt-5.4) - smoke-docker-sbx-build-test.md: Docker sbx + Node.js/Go build & test workloads These mirror the gVisor smoke tests to provide a comparison baseline and ensure Docker sandbox continues to work across engines and workloads. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * compiled sbx smoke tests * compiled sbx smoke tests --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 674a445 commit 1f435b6

7 files changed

Lines changed: 5187 additions & 124 deletions

‎.github/workflows/smoke-docker-sbx-build-test.lock.yml‎

Lines changed: 1497 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.github/workflows/smoke-docker-sbx-build-test.md‎

Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
---
2+
description: Smoke test Docker sandbox with multi-ecosystem build and test workloads
3+
on:
4+
workflow_dispatch:
5+
label_command:
6+
name: test-docker-sbx-build
7+
events: [pull_request]
8+
remove_label: false
9+
reaction: "eyes"
10+
permissions:
11+
contents: read
12+
pull-requests: read
13+
issues: read
14+
actions: read
15+
copilot-requests: write
16+
name: Smoke Docker Sbx Build Test
17+
engine:
18+
id: copilot
19+
version: 1.0.34
20+
runtimes:
21+
node:
22+
version: "20"
23+
go:
24+
version: "1.22"
25+
network:
26+
allowed:
27+
- defaults
28+
- github
29+
- node
30+
- go
31+
tools:
32+
bash:
33+
- "*"
34+
github:
35+
toolsets: [pull_requests]
36+
safe-outputs:
37+
threat-detection:
38+
enabled: false
39+
add-comment:
40+
hide-older-comments: true
41+
add-labels:
42+
allowed: [smoke-docker-sbx-build]
43+
messages:
44+
footer: "> 🐳🏗️ *Docker sbx build test by [{workflow_name}]({run_url})*"
45+
run-started: "🐳🏗️ [{workflow_name}]({run_url}) is testing Docker sandbox with build workloads..."
46+
run-success: "🐳🏗️ [{workflow_name}]({run_url}) completed. Docker sbx build test passed. ✅"
47+
run-failure: "🐳🏗️ [{workflow_name}]({run_url}) reports {status}. Docker sbx build compatibility issue detected."
48+
timeout-minutes: 30
49+
sandbox:
50+
mcp:
51+
version: v0.3.32
52+
strict: false
53+
jobs:
54+
verify_build:
55+
needs: agent
56+
if: always() && needs.agent.result != 'skipped' && needs.agent.result != 'cancelled'
57+
runs-on: ubuntu-latest
58+
permissions:
59+
contents: read
60+
steps:
61+
- name: Checkout repository
62+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
63+
with:
64+
persist-credentials: false
65+
- name: Download agent artifact
66+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
67+
with:
68+
name: agent
69+
path: /tmp/gh-aw-agent
70+
- name: Token-usage sanity check
71+
run: node scripts/ci/check-token-usage.js --artifact-root /tmp/gh-aw-agent --engine copilot
72+
steps:
73+
- name: Setup Go
74+
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
75+
with:
76+
go-version: '1.22'
77+
- name: Capture environment info
78+
id: env-info
79+
run: |
80+
echo "::group::Runtime versions"
81+
echo "Node: $(node --version 2>&1)"
82+
echo "npm: $(npm --version 2>&1)"
83+
echo "Go: $(go version 2>&1)"
84+
echo "::endgroup::"
85+
86+
echo "::group::GitHub.com connectivity"
87+
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 https://github.com)
88+
echo "github.com returned HTTP $HTTP_CODE"
89+
echo "SMOKE_HTTP_CODE=$HTTP_CODE" >> "$GITHUB_OUTPUT"
90+
echo "::endgroup::"
91+
- name: Build AWF project (Node.js)
92+
id: build-node
93+
run: |
94+
echo "::group::npm ci"
95+
npm ci 2>&1 | tail -5
96+
NPM_CI_EXIT=$?
97+
echo "::endgroup::"
98+
99+
echo "::group::npm run build"
100+
npm run build 2>&1 | tail -5
101+
NPM_BUILD_EXIT=$?
102+
echo "::endgroup::"
103+
104+
if [ $NPM_CI_EXIT -eq 0 ] && [ $NPM_BUILD_EXIT -eq 0 ]; then
105+
echo "NODE_BUILD_STATUS=PASS" >> "$GITHUB_OUTPUT"
106+
echo "✅ Node.js build succeeded"
107+
else
108+
echo "NODE_BUILD_STATUS=FAIL" >> "$GITHUB_OUTPUT"
109+
echo "❌ Node.js build failed (ci=$NPM_CI_EXIT, build=$NPM_BUILD_EXIT)"
110+
fi
111+
- name: Run AWF unit tests (Node.js)
112+
id: test-node
113+
run: |
114+
echo "::group::npm test"
115+
npx jest --ci --forceExit --maxWorkers=2 --testPathPattern='squid-config|docker-manager|logger' 2>&1 | tail -20
116+
TEST_EXIT=$?
117+
echo "::endgroup::"
118+
119+
if [ $TEST_EXIT -eq 0 ]; then
120+
echo "NODE_TEST_STATUS=PASS" >> "$GITHUB_OUTPUT"
121+
echo "✅ Node.js tests passed"
122+
else
123+
echo "NODE_TEST_STATUS=FAIL" >> "$GITHUB_OUTPUT"
124+
echo "❌ Node.js tests failed (exit=$TEST_EXIT)"
125+
fi
126+
- name: Clone and build Go test project
127+
id: build-go
128+
run: |
129+
echo "::group::Clone Go test repo"
130+
git clone --depth 1 https://github.com/Mossaka/gh-aw-firewall-test-go.git /tmp/test-go 2>&1 | tail -3
131+
CLONE_EXIT=$?
132+
echo "::endgroup::"
133+
134+
if [ $CLONE_EXIT -ne 0 ]; then
135+
echo "GO_BUILD_STATUS=CLONE_FAILED" >> "$GITHUB_OUTPUT"
136+
echo "GO_TEST_STATUS=SKIPPED" >> "$GITHUB_OUTPUT"
137+
echo "❌ Go clone failed"
138+
else
139+
echo "::group::Go build and test - color"
140+
cd /tmp/test-go/color
141+
go mod download 2>&1 | tail -3
142+
go build ./... 2>&1 | tail -5
143+
BUILD_EXIT=$?
144+
go test ./... 2>&1 | tail -10
145+
TEST_EXIT=$?
146+
echo "::endgroup::"
147+
148+
echo "::group::Go build and test - uuid"
149+
cd /tmp/test-go/uuid
150+
go mod download 2>&1 | tail -3
151+
go build ./... 2>&1 | tail -5
152+
BUILD2_EXIT=$?
153+
go test ./... 2>&1 | tail -10
154+
TEST2_EXIT=$?
155+
echo "::endgroup::"
156+
157+
if [ $BUILD_EXIT -eq 0 ] && [ $BUILD2_EXIT -eq 0 ]; then
158+
echo "GO_BUILD_STATUS=PASS" >> "$GITHUB_OUTPUT"
159+
else
160+
echo "GO_BUILD_STATUS=FAIL" >> "$GITHUB_OUTPUT"
161+
fi
162+
163+
if [ $TEST_EXIT -eq 0 ] && [ $TEST2_EXIT -eq 0 ]; then
164+
echo "GO_TEST_STATUS=PASS" >> "$GITHUB_OUTPUT"
165+
else
166+
echo "GO_TEST_STATUS=FAIL" >> "$GITHUB_OUTPUT"
167+
fi
168+
fi
169+
- name: Network isolation check
170+
id: net-check
171+
run: |
172+
echo "::group::Blocked domain test"
173+
BLOCKED_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://example.com 2>/dev/null || echo "000")
174+
echo "example.com returned: $BLOCKED_CODE"
175+
if [ "$BLOCKED_CODE" = "000" ] || [ "$BLOCKED_CODE" = "403" ]; then
176+
echo "NET_ISOLATION=PASS" >> "$GITHUB_OUTPUT"
177+
echo "✅ Network isolation working (blocked domain returned $BLOCKED_CODE)"
178+
else
179+
echo "NET_ISOLATION=FAIL" >> "$GITHUB_OUTPUT"
180+
echo "❌ Network isolation failed (blocked domain returned $BLOCKED_CODE)"
181+
fi
182+
echo "::endgroup::"
183+
- name: Write results summary
184+
run: |
185+
mkdir -p /tmp/gh-aw/agent
186+
cat > /tmp/gh-aw/agent/build-test-results.json << RESULTS_EOF
187+
{
188+
"http_code": "${{ steps.env-info.outputs.SMOKE_HTTP_CODE }}",
189+
"node_build": "${{ steps.build-node.outputs.NODE_BUILD_STATUS }}",
190+
"node_test": "${{ steps.test-node.outputs.NODE_TEST_STATUS }}",
191+
"go_build": "${{ steps.build-go.outputs.GO_BUILD_STATUS }}",
192+
"go_test": "${{ steps.build-go.outputs.GO_TEST_STATUS }}",
193+
"net_isolation": "${{ steps.net-check.outputs.NET_ISOLATION }}"
194+
}
195+
RESULTS_EOF
196+
cat /tmp/gh-aw/agent/build-test-results.json
197+
post-steps:
198+
- name: Validate safe outputs were invoked
199+
run: |
200+
OUTPUTS_FILE="${GH_AW_SAFE_OUTPUTS:-${RUNNER_TEMP}/gh-aw/safeoutputs/outputs.jsonl}"
201+
if [ ! -s "$OUTPUTS_FILE" ]; then
202+
echo "::error::No safe outputs were invoked. Smoke tests require the agent to call safe output tools."
203+
echo "Checked path: $OUTPUTS_FILE"
204+
exit 1
205+
fi
206+
echo "Safe output entries found: $(wc -l < "$OUTPUTS_FILE")"
207+
if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
208+
if ! grep -q '"add_comment"' "$OUTPUTS_FILE"; then
209+
echo "::error::Agent did not call add_comment on a pull_request trigger."
210+
exit 1
211+
fi
212+
echo "add_comment verified for PR trigger"
213+
fi
214+
echo "Safe output validation passed"
215+
---
216+
217+
# Smoke Test: Docker Sandbox + Build/Test Workloads
218+
219+
**CRITICAL REQUIREMENT: You MUST call `add_comment` on pull_request triggers. This is the primary success criterion. Do this FIRST before any other analysis.**
220+
221+
**Keep all outputs extremely short and concise. Use single-line responses where possible. No verbose explanations.**
222+
223+
## Context
224+
225+
This workflow validates that the Docker sandbox (default AWF mode without gVisor) can handle real-world build and test workloads. All heavy computation has been executed in deterministic pre-agent steps. Your job is to read results and produce the summary.
226+
227+
## Step 1: Read Results
228+
229+
Read the pre-computed results:
230+
231+
```bash
232+
cat /tmp/gh-aw/agent/build-test-results.json
233+
```
234+
235+
The JSON contains:
236+
- `http_code`: GitHub.com HTTP response code
237+
- `node_build`: Node.js build status (PASS/FAIL)
238+
- `node_test`: Node.js test status (PASS/FAIL)
239+
- `go_build`: Go build status (PASS/FAIL/CLONE_FAILED)
240+
- `go_test`: Go test status (PASS/FAIL/SKIPPED)
241+
- `net_isolation`: Network isolation check (PASS/FAIL)
242+
243+
## Step 2: Output (MANDATORY)
244+
245+
**If triggered by a pull request** (check: `${{ github.event_name }}` equals "pull_request"), you MUST call `add_comment` to post a **brief** comment on the current pull request with:
246+
247+
### 🐳🏗️ Docker Sbx Build Test Results
248+
249+
| Test | Status |
250+
|------|--------|
251+
| GitHub.com connectivity | ✅/❌ |
252+
| Node.js build (`npm ci && npm run build`) | ✅/❌ |
253+
| Node.js tests (Jest subset) | ✅/❌ |
254+
| Go build (color, uuid) | ✅/❌ |
255+
| Go tests (color, uuid) | ✅/❌ |
256+
| Network isolation | ✅/❌ |
257+
258+
**Overall: PASS/FAIL**
259+
260+
If all tests pass on a pull request trigger:
261+
- Use the `add_labels` safe-output tool to add the label `smoke-docker-sbx-build` to the pull request
262+
263+
**If triggered by workflow_dispatch** (no PR context), call `noop` with a concise PASS/FAIL summary instead. Do NOT attempt to add pull request comments or labels when there is no pull request.

0 commit comments

Comments
 (0)