feat(orchestrator): count vanished workloads - #313
Conversation
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
dc8fe83 to
a4afac0
Compare
morgan-wowk
changed the title
a4afac0 to
c2bec2c
Compare
| ) | ||
|
|
||
| execution_missing_workloads = orchestrator_meter.create_counter( | ||
| name="execution.missing_workloads", |
There was a problem hiding this comment.
Alternatively poll.missing_workloads
Name suggestions welcome.
There was a problem hiding this comment.
Up to you, but maybe:
execution -> container_execution (to disambiguate from execution_node)
missing workloads is OK. Alternatives:
missing_launched_containers
disappeared_launched_containers
launched_container_not_found_errors
| @@ -1,4 +1,5 @@ | |||
| import copy | |||
| import http | |||
There was a problem hiding this comment.
Red flag. Orchestrator is not dependent on/tied to HTTP.
| return status is not None and 500 <= status < 600 | ||
|
|
||
|
|
||
| def _is_missing_workload_failure(exception: BaseException) -> bool: |
There was a problem hiding this comment.
This is too Kubernetes launcher specific and does not belong to Orchestrator.
Options:
- Move this tracking to kubernetes_laucnhers
- Create a LaunchedContainerNotFoundError exception that derives from ContainerLauncherError and then handle it.
There was a problem hiding this comment.
Implemented the latter. Thank you!
c3a74c9 to
6d972ae
Compare
c2bec2c to
eaf6fec
Compare
morgan-wowk
changed the title
When refreshing a running container fails because the workload no longer exists, that is a definitive failure -- the workload is gone rather than merely unobservable -- and it is worth watching on its own. The Kubernetes launcher now maps a 404 to a typed `LaunchedContainerNotFoundError`, and the orchestrator counts it on a dedicated counter, `execution.missing_workloads`, labelled with the status the container was in when it vanished (RUNNING vs PENDING usually point at different culprits: workloads killed mid-run vs workloads that never started). Deciding that a 404 means "gone" is the launcher's job, not the orchestrator's: the orchestrator acts on the typed error without inspecting HTTP status codes, staying free of platform specifics. This is observability only. A not-found error is non-retriable, so the execution terminalizes as SYSTEM_ERROR exactly as before -- no change to verdicts or pipeline semantics. Co-authored-by: Morgan Wowk <morgan.wowk@shopify.com>
eaf6fec to
90206b6
Compare
6d972ae to
8e67db9
Compare
| ) | ||
|
|
||
| execution_missing_workloads = orchestrator_meter.create_counter( | ||
| name="execution.missing_workloads", |
There was a problem hiding this comment.
Up to you, but maybe:
execution -> container_execution (to disambiguate from execution_node)
missing workloads is OK. Alternatives:
missing_launched_containers
disappeared_launched_containers
launched_container_not_found_errors
2 participants
What
Adds one counter,
execution.missing_workloads, incremented when refreshing a running container fails because the workload no longer exists — it is gone, not merely unobservable.The counter is labelled with the status the container was in when it vanished (
RUNNINGvsPENDING), since workloads killed mid-run and workloads that never started usually have different root causes.Consistent with the launcher/orchestrator boundary, the Kubernetes launcher maps a 404 to a typed
LaunchedContainerNotFoundError; the orchestrator counts that typed error without inspecting HTTP status codes.Why
In one recent 17h production window, vanished workloads accounted for 162 of 255 orchestrator
SYSTEM_ERRORs. This makes that slice measurable.Scope
Observability only. A not-found error is non-retriable, so the execution still terminalizes as
SYSTEM_ERRORexactly as before — no change to verdicts or pipeline semantics.