fix(egress): restart dead daemons and cap inbound scan body size (#455)
test / integration-docker (pull_request) Successful in 20s
lint / lint (push) Successful in 1m1s
test / unit (pull_request) Failing after 1m46s
test / integration-firecracker (pull_request) Successful in 3m22s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 10m4s
test / integration-docker (pull_request) Successful in 20s
lint / lint (push) Successful in 1m1s
test / unit (pull_request) Failing after 1m46s
test / integration-firecracker (pull_request) Successful in 3m22s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 10m4s
Two complementary fixes for the egress gateway OOM: 1. gateway_init: auto-restart any daemon that dies unexpectedly. The supervisor already had restart_daemon()/request_restart() logic; this wires it into tick() so an OOM-killed mitmdump is respawned without operator intervention. Implements the "eventual" failure policy described in the original module docstring. 2. egress_addon: cap response body bytes passed to the DLP inbound scan at EGRESS_INBOUND_SCAN_LIMIT_BYTES (default 1 MiB). mitmproxy buffers the full response before the hook fires; capping at scan time limits the additional amplification from decoded text and regex strings. Bodies over the limit emit an egress_scan_truncated log event. Set to 0 to disable the cap. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -5,18 +5,12 @@ the configured daemons (egress, git-gate, supervise),
|
||||
forwards SIGTERM/SIGINT to each child, and propagates per-daemon
|
||||
stdout+stderr to the container log with a `[name] ` prefix.
|
||||
|
||||
Failure policy (interim): when a child dies unexpectedly, the
|
||||
supervisor logs the death and leaves the surviving children
|
||||
running. The gateway stays up; whatever the dead daemon served
|
||||
will start failing, surfacing in the agent's own error path.
|
||||
The supervisor itself exits only when (a) the operator sends
|
||||
SIGTERM/SIGINT, or (b) every child has died.
|
||||
|
||||
Failure policy (eventual): on unexpected death, the supervisor
|
||||
restarts the daemon and emits a notification to the supervise
|
||||
daemon so the operator sees the event. That lands in a later
|
||||
PR; the interim policy is "don't take the gateway down for one
|
||||
sick daemon."
|
||||
Failure policy: when a child dies unexpectedly, the supervisor
|
||||
restarts it automatically and logs the restart. The gateway stays
|
||||
up; a temporary loss of one daemon (e.g. egress OOM-killed) is
|
||||
recovered without manual container recreation. The supervisor
|
||||
itself exits only when (a) the operator sends SIGTERM/SIGINT, or
|
||||
(b) every child has died.
|
||||
|
||||
Daemon subset is env-driven via `BOT_BOTTLE_GATEWAY_DAEMONS=egress`
|
||||
for callers that don't use git-gate or supervise. Default: all
|
||||
@@ -238,11 +232,8 @@ class _Supervisor:
|
||||
continue
|
||||
self._logged_dead.add(spec.name)
|
||||
if self.shutdown_at is None:
|
||||
_log(
|
||||
f"{spec.name} exited with code {rc}; leaving "
|
||||
f"surviving daemons running (operator-visible "
|
||||
f"via agent-side failure)"
|
||||
)
|
||||
_log(f"{spec.name} exited with code {rc}; scheduling restart")
|
||||
self._restart_requested.add(spec.name)
|
||||
else:
|
||||
_log(f"{spec.name} exited with code {rc}")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user