hermes-agent/tests/tools/test_code_kernel.py

610 lines
26 KiB
Python
Raw Normal View History

feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
#!/usr/bin/env python3
"""Tests for execute_code's session kernel.
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
Session kernels are always on (the ``code_execution.kernel_mode`` key is
retired): each (task, mode, interpreter, cwd, tool-set) owner keeps one
Python child alive so state survives across calls. These tests pin the
contract:
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
- state persists across cells and reset=true discards it
- a raised exception keeps the kernel (and its state) alive
- a timeout kills the kernel; the next call gets a fresh one
- fd-level output from user-spawned subprocesses reaches the result
- sys.exit() inside a cell ends the kernel deliberately
Mode is sourced from ``code_execution.mode`` in config.yaml only;
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
tests patch ``_load_config`` directly, mirroring test_code_execution_modes.
"""
import json
import os
import subprocess
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
import sys
import tempfile
import textwrap
import time
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
import unittest
from contextlib import contextmanager
from pathlib import Path
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
from unittest.mock import patch
import pytest
os.environ["TERMINAL_ENV"] = "local"
@pytest.fixture(autouse=True)
def _force_local_terminal(monkeypatch):
"""Mirror test_code_execution.py — guarantee local backend."""
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
monkeypatch.setenv("TERMINAL_ENV", "local")
from tools.code_execution_tool import execute_code
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
from tools.code_kernel import _KERNELS, shutdown_all_kernels
@contextmanager
def _kernel_config(**overrides):
"""Pin code_execution config; strict mode keeps the test hermetic.
``mode`` (strict/project) is the only config knob — session kernels are
always on; the retired ``kernel_mode`` key is ignored by the tool."""
config = {"mode": "strict", "timeout": 30}
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
config.update(overrides)
with patch("tools.code_execution_tool._load_config", return_value=config):
yield
@pytest.fixture(autouse=True)
def _fresh_kernel_registry():
shutdown_all_kernels()
yield
shutdown_all_kernels()
def _run(code, **kwargs):
return json.loads(execute_code(code, task_id="kernel-test", **kwargs))
class TestSessionStatePersistence(unittest.TestCase):
def test_state_persists_across_cells(self):
with _kernel_config():
first = _run("x = 41")
self.assertEqual(first["status"], "success", first)
self.assertEqual(first["kernel"]["reused"], False)
second = _run("print(x + 1)")
self.assertEqual(second["status"], "success", second)
self.assertIn("42", second["output"])
self.assertEqual(second["kernel"]["reused"], True)
self.assertEqual(second["kernel"]["execution_count"], 2)
def test_reset_discards_state(self):
with _kernel_config():
_run("x = 41")
second = _run("print(x + 1)", reset=True)
self.assertEqual(second["status"], "error", second)
self.assertIn("NameError", second.get("error", ""))
self.assertEqual(second["kernel"]["state_reset"], True)
def test_exception_keeps_the_kernel_alive(self):
with _kernel_config():
_run("a = 7")
boom = _run("1 / 0")
self.assertEqual(boom["status"], "error")
self.assertIn("ZeroDivisionError", boom["error"])
after = _run("print(a)")
self.assertEqual(after["status"], "success", after)
self.assertIn("7", after["output"])
self.assertEqual(after["kernel"]["reused"], True)
def test_imports_persist(self):
with _kernel_config():
_run("import json as _j")
second = _run("print(_j.dumps({'k': 1}))")
self.assertIn('{"k": 1}', second["output"])
class TestKernelLifecycle(unittest.TestCase):
def test_kernel_exits_when_its_backend_parent_dies(self):
"""A kernel must not outlive the host that spawned it, even when the
host dies without cleanup (SIGKILL/OOM/crash). Windows: inherited
fix(execute_code): a kernel's cell subprocesses die with the Hermes process that ran them (orphans_after_10s 3 -> 0) Symptom: the POSIX death pipe (365e2835d4) only told the KERNEL its host had died. Every subprocess a cell starts (a dev server, a `sleep`, a training job) shares the kernel's process group and never sees that pipe, so a SIGKILLed / OOM-killed host left them running on init indefinitely. Measured on Linux against origin/main 7770e397f3 (ck_probe: host spawns a kernel, the cell Popen()s `sleep 300`, host gets kill -9, check at +10 s): kernel 0/3 alive, cell child 3/3 alive. Change: _spawn starts the kernel through process_identity.spawn_owned(purpose="code-kernel"): it leads its own group (as before, via start_new_session), that group goes to the per-process owner-death supervisor, which killpg()s it the moment the host dies by any cause, and the kernel is recorded in the spawn ledger. teardown() releases the group once it is empty (waiting up to 1 s for SIGTERMed cell children to exit); a cell child that survives the teardown's SIGTERM keeps the group registered until it exits. The kernel-only death pipe (runner watchdog thread, pipe plumbing, HERMES_KERNEL_PARENT_DEATH_FD) is deleted: the supervisor kills the kernel too. Windows keeps its SYNCHRONIZE-handle watchdog unchanged; atexit graceful shutdown, the idle reaper and the stale-staging-dir sweep stay (they cover a live host and disk, not owner death). Result: same probe, 3/3 runs: kernel 0 alive, cell child 0 alive. tools/code_kernel.py +25/-57. New tests/tools/test_code_kernel_owned_group.py (kill -9 owner vs live control owner, graceful release, surviving-child coverage): 3/3 green x10 runs, all 3 red on the pre-change spawn, each red on its own mutation (no registration / no release / unconditional release). Family files on main 75 passed; on this branch 78 passed (75 + 3 new).
2026-10-12 14:58:12 +05:30
SYNCHRONIZE handle; POSIX: the kernel's group is registered with the
owner-death supervisor. Both are proven the same way — kill the host
mid-cell, the kernel is gone within seconds."""
import psutil
repo_root = str(Path(__file__).resolve().parents[2])
host_src = textwrap.dedent(f"""
import json, os, sys, time
os.environ["HERMES_HOME"] = sys.argv[1]
sys.path.insert(0, {repo_root!r})
from tools.code_kernel import SessionKernel, _spawn
k = SessionKernel(("parent-death",))
_spawn(k, task_id="parent-death", child_python=sys.executable,
child_cwd="", sandbox_tools=frozenset(), max_tool_calls=1)
cell = json.dumps({{"id": "x", "code": "import os, time\\n"
"assert 'HERMES_KERNEL_PARENT_PROCESS_HANDLE' not in os.environ\\n"
"assert 'HERMES_KERNEL_PARENT_DEATH_FD' not in os.environ\\n"
"time.sleep(300)"}}) + "\\n"
k.proc.stdin.write(cell.encode()); k.proc.stdin.flush()
print(k.proc.pid, flush=True)
time.sleep(600)
""")
with tempfile.TemporaryDirectory() as home:
host = subprocess.Popen(
[sys.executable, "-c", host_src, home],
stdout=subprocess.PIPE, text=True,
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0),
)
try:
kernel = psutil.Process(int(host.stdout.readline()))
time.sleep(0.5)
self.assertTrue(kernel.is_running(), "kernel never came up")
host.kill()
host.wait(timeout=10)
try:
kernel.wait(timeout=10)
except psutil.TimeoutExpired:
kernel.kill()
self.fail("session kernel survived its backend parent")
finally:
if host.poll() is None:
host.kill()
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
def test_timeout_kills_the_kernel_and_reports_state_loss(self):
with _kernel_config(timeout=1):
slow = _run("import time\ntime.sleep(30)")
self.assertEqual(slow["status"], "timeout", slow)
self.assertIn("state was lost", slow["error"])
self.assertEqual(len(_KERNELS), 0)
with _kernel_config():
fresh = _run("print('alive')")
self.assertEqual(fresh["status"], "success", fresh)
self.assertEqual(fresh["kernel"]["reused"], False)
self.assertIn("alive", fresh["output"])
def test_sys_exit_ends_the_kernel(self):
with _kernel_config():
done = _run("import sys\nsys.exit(0)")
self.assertEqual(done["kernel"].get("ended"), True, done)
self.assertEqual(len(_KERNELS), 0)
fresh = _run("print('respawned')")
self.assertEqual(fresh["kernel"]["reused"], False)
self.assertIn("respawned", fresh["output"])
def test_subprocess_fd_output_reaches_the_result(self):
code = (
"import subprocess, sys\n"
"subprocess.run([sys.executable, '-c', \"print('raw-passthrough')\"])\n"
)
with _kernel_config():
result = _run(code)
self.assertEqual(result["status"], "success", result)
self.assertIn("raw-passthrough", result["output"])
test: restore tools regression guards dropped by #120071 - test_code_kernel::TestModelFacingReset: reset stays in the schema under a stale kernel_mode and a model-shaped reset call really discards kernel state (#96787). - test_computer_use::TestStartupTimeoutPhaseDetail: cua-driver ready timeout names the wedged phase + doctor hint, and the session stays un-started (#57025/#69372). - test_file_operations::TestEscapeNativeToolArg (windows_only, was FAKE_OS): rg search and the node --check linter get native C:/ paths, not /c/ (#84303). - test_file_tools::TestSSHConfigWriteGate (2): a ~/.ssh/config write goes through the real approval gate, BLOCKED with nobody present / in -q mode, nothing written (#93201). - test_lazy_deps: a newer compatible plugin SDK is satisfied, never re-pinned down (#98407, #86992); windows_only real Matrix probe reports unsupported. - test_mcp_empty_error_message: empty/whitespace exception messages still yield a diagnostic (#19417). - test_mcp_stdio_children_dead: the fast-fail probe never calls the watcher on the plain path, no un-awaited coroutine (#96044). - test_subprocess_utf8_encoding: real _op_whoami survives invalid UTF-8 child output (#53428/#55339). - test_tts_output_timestamp: same-second TTS calls get distinct output files (#43911). - test_windows_agent_loop_papercuts::TestAutocompleteDevicePaths: a relpath ValueError on a device path is skipped, not raised (#42016). - test_windows_native_support::TestCronSchedulerBashResolution (2): cron .sh uses bash from PATH; missing bash is an actionable error, not WinError 2.
2026-09-23 04:39:48 -07:00
class TestModelFacingReset(unittest.TestCase):
def test_reset_is_reachable_from_a_model_call_despite_stale_kernel_mode(self):
"""Session kernels are always on (#96787), so ``reset`` is the model's only
way out of poisoned state. A stale ``kernel_mode: per-call`` key must not
drop it from the schema, and a model-shaped call routed through the
registered handler must actually discard the kernel's state."""
from tools.code_execution_tool import _execute_code_handler, build_execute_code_schema
with _kernel_config(kernel_mode="per-call"):
schema = build_execute_code_schema(mode="strict")
self.assertEqual(schema["parameters"]["properties"]["reset"]["type"], "boolean")
_execute_code_handler({"code": "x = 41"}, task_id="kernel-test")
kept = json.loads(_execute_code_handler({"code": "print(x + 1)"}, task_id="kernel-test"))
self.assertIn("42", kept["output"], kept)
reset = json.loads(_execute_code_handler(
{"code": "print(x + 1)", "reset": True}, task_id="kernel-test"))
self.assertEqual(reset["status"], "error", reset)
self.assertIn("NameError", reset.get("error", ""))
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-v"]))
class TestKernelOwnershipAndLifecycle(unittest.TestCase):
"""The kernel belongs to the conversation, and its lifetime is bounded.
run_agent mints a fresh task id per top-level turn, so a task-keyed
kernel would neither survive the next user turn nor ever be disposed
with anything. The owner is the approval session key; disposal rides
the same session boundary that clears approval/yolo state, idle
kernels are reaped, and the process-wide live count is capped (the
lifecycle shape carried forward from hermes-agent#88637).
"""
def _run_as(self, session_key, code, task_id, **kwargs):
from tools.approval_context import reset_current_session_key, set_current_session_key
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
token = set_current_session_key(session_key)
try:
return json.loads(execute_code(code, task_id=task_id, **kwargs))
finally:
reset_current_session_key(token)
def test_state_survives_across_turns_of_one_conversation(self):
# Two top-level turns: same session, different per-turn task ids.
with _kernel_config():
first = self._run_as("conv-a", "x = 41", task_id="turn-1")
self.assertEqual(first["status"], "success", first)
second = self._run_as("conv-a", "print(x + 1)", task_id="turn-2")
self.assertEqual(second["status"], "success", second)
self.assertIn("42", second["output"])
self.assertEqual(second["kernel"]["reused"], True)
def test_sessions_are_isolated_from_each_other(self):
# Same task id, different sessions: no state may cross.
with _kernel_config():
self._run_as("conv-a", "x = 41", task_id="turn-1")
other = self._run_as("conv-b", "print(x + 1)", task_id="turn-1")
self.assertEqual(other["status"], "error", other)
self.assertIn("NameError", other.get("error", ""))
def test_delegated_children_get_their_own_kernels(self):
"""A delegated child runs in a COPY of the parent's context and
inherits the parent's approval session key — the naive owner
resolution attached the child to the parent's kernel and leaked
in-memory state across the delegation boundary (both directions,
verified live). The owner must be qualified for child contexts."""
from agent.delegation_context import delegated_child_context
with _kernel_config():
self._run_as("conv-a", "parent_secret = 'p'", task_id="turn-1")
with delegated_child_context("child-1"):
leak = self._run_as(
"conv-a",
"print(globals().get('parent_secret', 'ISOLATED'))",
task_id="child-task",
)
self._run_as("conv-a", "child_secret = 'c'", task_id="child-task")
back = self._run_as(
"conv-a",
"print(globals().get('child_secret', 'ISOLATED'))",
task_id="turn-2",
)
self.assertIn("ISOLATED", leak.get("output", ""), leak)
self.assertIn("ISOLATED", back.get("output", ""), back)
def test_two_delegated_children_are_isolated_from_each_other(self):
"""Sibling children in one batch must not share a kernel either —
each child context carries its own delegation session id."""
from agent.delegation_context import delegated_child_context
with _kernel_config():
with delegated_child_context("child-A"):
self._run_as("conv-a", "sibling_secret = 'A'", task_id="t")
with delegated_child_context("child-B"):
peek = self._run_as(
"conv-a",
"print(globals().get('sibling_secret', 'ISOLATED'))",
task_id="t",
)
self.assertIn("ISOLATED", peek.get("output", ""), peek)
def test_live_children_keep_their_kernels_past_the_lru_cap(self):
"""A fan-out wider than max_session_kernels used to evict LIVE children's kernels (each
child's execute_code spawned a kernel, the cap reaped the oldest sibling's), so a child's
second call hit NameError on state its first call had set — 48 NameErrors across 28 lanes,
while the schema promised persistence. A live child's kernel is pinned for the child's life."""
import contextvars
from agent.delegation_context import delegated_child_context
with _kernel_config(max_session_kernels=2):
contexts = []
for index in range(5):
def _set(index=index):
with delegated_child_context(f"child-{index}"):
self._run_as("conv", f"v = {index}", task_id=f"child-{index}")
ctx = contextvars.copy_context()
ctx.run(_set)
contexts.append(ctx)
outcomes = {}
for index, ctx in enumerate(contexts):
def _read(index=index):
with delegated_child_context(f"child-{index}"):
outcomes[index] = self._run_as("conv", "print(v)", task_id=f"child-{index}")
ctx.run(_read)
for index, outcome in outcomes.items():
self.assertEqual(outcome["status"], "success", outcome)
self.assertTrue(outcome["kernel"]["reused"], outcome)
self.assertIn(str(index), outcome["output"])
def test_finished_children_release_their_kernels(self):
"""The pin is not a leak: when the child is torn down (the delegate_task cleanup path calls
``shutdown_kernels_for_delegated_child``) its kernels die and stop counting."""
from agent.delegation_context import delegated_child_context
from tools.code_kernel import shutdown_kernels_for_delegated_child
with _kernel_config():
with delegated_child_context("child-done"):
self._run_as("conv", "v = 1", task_id="child-done")
with delegated_child_context("child-live"):
self._run_as("conv", "v = 2", task_id="child-live")
doomed = [k for k in _KERNELS.values() if k.owner.endswith("::child::child-done")]
self.assertEqual(len(doomed), 1)
shutdown_kernels_for_delegated_child("child-done")
self.assertEqual([k for k in _KERNELS.values() if k.owner.endswith("::child::child-done")], [])
doomed[0].proc.wait(timeout=10)
self.assertFalse(doomed[0].alive())
# The sibling's kernel is untouched.
with delegated_child_context("child-live"):
still = self._run_as("conv", "print(v)", task_id="child-live")
self.assertIn("2", still["output"])
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
def test_session_clear_disposes_the_owners_kernels(self):
from tools.approval import clear_session
with _kernel_config():
self._run_as("conv-a", "x = 41", task_id="turn-1")
self.assertEqual(len(_KERNELS), 1)
kernel = next(iter(_KERNELS.values()))
self.assertTrue(kernel.alive())
clear_session("conv-a")
self.assertEqual(len(_KERNELS), 0)
kernel.proc.wait(timeout=10)
self.assertFalse(kernel.alive())
# The next turn in a cleared session starts fresh.
after = self._run_as("conv-a", "print('x' in dir())", task_id="turn-2")
self.assertEqual(after["status"], "success", after)
self.assertIn("False", after["output"])
def test_live_kernels_are_capped_lru_across_owners(self):
with _kernel_config(max_session_kernels=2):
kernels = []
for index in range(4):
self._run_as(f"conv-{index}", "x = 1", task_id=f"turn-{index}")
kernels.append(list(_KERNELS.values()))
self.assertLessEqual(len(_KERNELS), 2)
live_owners = {key[0] for key in _KERNELS}
# The two most recently used owners survive.
self.assertEqual(live_owners, {"conv-2", "conv-3"})
# Evicted kernels are actually dead, not orphaned.
evicted = [
kernel
for snapshot in kernels
for kernel in snapshot
if kernel.key not in _KERNELS
]
for kernel in evicted:
kernel.proc.wait(timeout=10)
self.assertFalse(kernel.alive())
def test_idle_kernels_are_reaped(self):
import time as time_module
with _kernel_config(kernel_idle_timeout=1):
self._run_as("conv-a", "x = 41", task_id="turn-1")
stale = next(iter(_KERNELS.values()))
time_module.sleep(1.2)
# Any owner's next call sweeps expired kernels process-wide.
self._run_as("conv-b", "y = 1", task_id="turn-2")
self.assertNotIn(stale.key, _KERNELS)
stale.proc.wait(timeout=10)
self.assertFalse(stale.alive())
def test_parallel_cells_share_one_kernel_process(self):
"""Parallel cells for one owner race the first spawn. Each racer
used to see proc=None as 'dead', replace the registry entry, and
orphan the winner's process — 110 live kernels under a 4-capped
process (Sep 2026). Every kernel process must stay registry-owned.
Capture actual children so an unregistered spawn cannot hide behind
the registry count. The owned process must exit on teardown."""
import threading
spawned = []
real_popen = subprocess.Popen
def _capturing_popen(args, **kwargs):
proc = real_popen(args, **kwargs)
spawned.append((proc, list(args)))
return proc
results = []
with patch("tools.code_kernel.subprocess.Popen", side_effect=_capturing_popen):
with _kernel_config():
def _cell():
results.append(self._run_as("conv-a", "import time; time.sleep(0.3)", task_id="t"))
threads = [threading.Thread(target=_cell) for _ in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
self.assertEqual([r["status"] for r in results], ["success"] * 6)
self.assertEqual(len(_KERNELS), 1)
runners = [proc for proc, args in spawned
if len(args) == 2 and Path(args[1]).name == "hermes_kernel_runner.py"]
self.assertEqual(len(runners), 1, "parallel cells spawned an unowned kernel")
kernel = next(iter(_KERNELS.values()))
self.assertIs(kernel.proc, runners[0])
self.assertIsNone(kernel.proc.poll())
shutdown_all_kernels()
for proc in runners:
proc.wait(timeout=10)
self.assertIsNotNone(proc.returncode)
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
class TestInScriptToolErrors(unittest.TestCase):
def test_ignored_helper_error_is_reported_for_that_cell_only(self):
"""A script that drops a helper's {"error": ...} return must not read as a clean success."""
def _handle(tool_name, tool_args, task_id=None):
if tool_name == "write_file":
return json.dumps({"error": "Refusing to overwrite a.py: never read"})
return json.dumps({"ok": True})
with _kernel_config(), patch("model_tools.handle_function_call", new=_handle):
failed = _run("import hermes_tools\nhermes_tools.write_file('a.py', 'x')\nprint('done')\n")
clean = _run("import hermes_tools\nhermes_tools.web_search(query='q')\n")
self.assertEqual(failed["status"], "success", failed)
self.assertEqual(failed["tool_errors"], [{"tool": "write_file", "error": "Refusing to overwrite a.py: never read"}])
self.assertNotIn("tool_errors", clean)
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
class TestPerCellRpcAuthority(unittest.TestCase):
"""Interpreter state persists across cells; RPC authority must not."""
def _recorder(self, seen):
def _handle(tool_name, tool_args, task_id=None):
from tools.thread_context import _callback_api
(get_approval, _set_a), *_rest = _callback_api()
feat(tools): session-persistent kernels for execute_code (kernel_mode: session) (#94647) * feat(tools): session-persistent kernels for execute_code (kernel_mode: session) execute_code spawns a fresh Python process per call, so every multi-step data task re-loads its inputs: a CSV parsed in call one is gone by call two, and scripts route state through temp files to survive. Hermes already rewards programmatic tool calling (execute_code-only turns refund the iteration budget), which makes the missing half — state that survives between calls — the bottleneck. Add opt-in `code_execution.kernel_mode: session`: one persistent kernel per (task, mode, interpreter, cwd, tool-set). Variables, imports, and loaded data persist across calls; `reset=true` discards state on demand. The default `per-call` keeps today's behavior byte-for-byte. Safety posture is unchanged by design: the child env comes from the same builder as the per-call path (extracted, not duplicated, so the secret scrubbing / PYTHONPATH hygiene cannot drift), the RPC server is the same `_rpc_server_loop` with the same token and a per-cell tool budget, and output passes the same ANSI strip + secret redaction. A timed-out or interrupted cell kills the whole kernel tree and the next call respawns — a wedged kernel can never hang the agent. The kernel env is frozen at spawn; the schema and config comment say so. Wire protocol: NDJSON requests on the kernel's stdin; responses framed on stdout behind a per-kernel random sentinel, with unframed bytes (fd-level output from user-spawned subprocesses) attributed to the serialized current cell. The generated RPC client reconnects once when HERMES_RPC_PERSISTENT=1, because a kernel legitimately outlives the RPC server's 300s idle window between cells. Tested on macOS 15 (Apple Silicon), Python 3.11: 13 new tests in tests/tools/test_code_kernel.py (persistence, reset, error-keeps-kernel, timeout-kills-kernel, sys.exit ends kernel, subprocess fd passthrough, schema surface, mode fallback) plus the existing test_code_execution.py / test_code_execution_modes.py suites (81 passed). * fix(tools): session kernels get a stable owner, bounded lifetime, and per-cell RPC authority Addresses the blocking review on the session-kernel design: two authority/lifecycle boundaries were wrong. 1. Ownership and bounded lifetime. The kernel key's first component is now the conversation's approval session key (_resolve_owner), not the per-turn task id run_agent mints per top-level invocation — so state genuinely survives across user turns of one conversation, and delegated subagent sessions isolate naturally under their own keys (the task id remains only the last-resort owner for embeds/tests with no session context). Lifetime is bounded on four edges: kernels are disposed at the same session boundary that clears the owner's approval/yolo state (tools.approval.clear_session -> shutdown_kernels_for_owner), reaped after code_execution.kernel_idle_timeout seconds idle (default 1800, swept on every entry), capped process-wide at code_execution.max_session_kernels live children (default 4, LRU evicted), and still torn down by reset/death/atexit as before. The ownership + disposal + idle-reap + cap shape deliberately carries forward the lifecycle invariants of the earlier session-persistent implementation in #88637 by @z80dev. 2. Per-cell RPC authority. The serving thread no longer freezes the spawning cell's context/callbacks for the kernel's life. Each cell installs a CellAuthority — captured on the calling thread exactly as propagate_context_to_thread would for a per-call RPC thread — before its request is written, and retires it on every settle path; _rpc_server_loop gains a dispatch hook the kernel uses to route each tool call through the CURRENT cell's context, callbacks, and task id. A call arriving with no active cell is refused. Interpreter state persists; RPC authority does not. Composition with the per-script static guard (see the config note): a persistent namespace lets cell N+1 invoke objects cell N created, which a single-cell static scan cannot see — the runtime RPC boundary (allow-list by name, per-cell budget, per-cell authority) is the operative cross-cell enforcement in this mode, and the adversarial alias test pins exactly that. Tests (9 new): state survives across turns of one conversation; sessions isolate; clear_session disposes the owner's kernels (and the next turn starts fresh); the live-kernel cap LRU-evicts with evicted children proven dead; idle kernels are reaped; a later cell's RPC runs under that cell's approval callback; a cross-cell alias dispatches under the CURRENT cell's authority; a settled cell's authority refuses dispatch; each cell installs a fresh authority. 22/22 kernel tests, 81 code-execution tests, ruff clean. The 7 test-order failures in the tools/-k-approval selection reproduce identically on the clean branch base (pre-existing pollution, not this change). * fix(code-kernel): delegated children get their own kernels — child contexts inherit the parent approval key, so qualify the owner with the delegation session id (live-verified leak, both directions) --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-08-28 04:26:31 +09:00
seen.append(
{
"tool": tool_name,
"task_id": task_id,
"approval_cb": get_approval(),
}
)
return json.dumps({"ok": True})
return _handle
def test_a_later_cells_rpc_runs_under_that_cells_authority(self):
from tools.terminal_tool import set_approval_callback
seen = []
cell = "import hermes_tools\nhermes_tools.web_search(query='q')\n"
with _kernel_config(), patch(
"model_tools.handle_function_call", new=self._recorder(seen)
):
def cb_one():
return "one"
def cb_two():
return "two"
set_approval_callback(cb_one)
try:
first = _run(cell)
set_approval_callback(cb_two)
second = _run(cell)
finally:
set_approval_callback(None)
self.assertEqual(first["status"], "success", first)
self.assertEqual(second["status"], "success", second)
self.assertEqual(len(seen), 2)
self.assertIs(seen[0]["approval_cb"], cb_one)
self.assertIs(seen[1]["approval_cb"], cb_two)
self.assertEqual(seen[0]["task_id"], "kernel-test")
def test_cross_cell_alias_dispatches_under_the_current_cell(self):
# Adversarial cross-cell dataflow: a callable captured in cell 1 and
# invoked by an opaque global name in cell 2 still crosses the RPC
# boundary — under cell 2's authority, allow-list, and budget — the
# operative enforcement a per-script static scan cannot provide once
# state persists (composition contract with the execute-code guard).
from tools.terminal_tool import set_approval_callback
seen = []
with _kernel_config(), patch(
"model_tools.handle_function_call", new=self._recorder(seen)
):
def cb_one():
return "one"
def cb_two():
return "two"
set_approval_callback(cb_one)
try:
first = _run("import hermes_tools\nalias = hermes_tools.web_search\n")
set_approval_callback(cb_two)
second = _run("alias(query='q')\n")
finally:
set_approval_callback(None)
self.assertEqual(first["status"], "success", first)
self.assertEqual(second["status"], "success", second)
self.assertEqual(len(seen), 1)
self.assertIs(seen[0]["approval_cb"], cb_two)
def test_a_settled_cells_authority_refuses_dispatch(self):
from tools.code_kernel import CellAuthority
authority = CellAuthority("turn-1")
authority.retire()
result = authority.dispatch("web_search", {"query": "q"})
self.assertIn("No active execute_code cell", result)
def test_each_cell_installs_a_fresh_authority(self):
with _kernel_config():
_run("x = 1")
kernel = next(iter(_KERNELS.values()))
first_authority = kernel.cell_authority
self.assertFalse(first_authority.active)
_run("y = 2")
self.assertIsNot(kernel.cell_authority, first_authority)
self.assertFalse(kernel.cell_authority.active)
class TestBackgroundIdleReaper(unittest.TestCase):
"""#117169: the idle sweep must not depend on the next kernel acquire — a host
that stays alive but wedged (e.g. pids exhaustion fail-closing every tool call)
never acquires again, so a background reaper reapplies the acquire-path criteria
on its own schedule, and staging dirs that outlived a dead host are swept by age."""
def _run_as(self, session_key, code, task_id, **kwargs):
from tools.approval_context import reset_current_session_key, set_current_session_key
token = set_current_session_key(session_key)
try:
return json.loads(execute_code(code, task_id=task_id, **kwargs))
finally:
reset_current_session_key(token)
def test_reap_once_sweeps_idle_kernels_without_a_new_acquire(self):
import time as time_module
from tools.code_kernel import _reap_once
with _kernel_config(kernel_idle_timeout=1):
self._run_as("conv-a", "x = 41", task_id="turn-1")
stale = next(iter(_KERNELS.values()))
time_module.sleep(1.2)
# No conv-b acquire here: the reaper pass alone must retire the kernel.
_reap_once()
self.assertNotIn(stale.key, _KERNELS)
stale.proc.wait(timeout=10)
self.assertFalse(stale.alive())
def test_reap_once_spares_attached_and_fresh_kernels(self):
from tools.code_kernel import _reap_once
with _kernel_config(kernel_idle_timeout=1):
fresh = self._run_as("conv-fresh", "x = 1", task_id="turn-1")
self.assertEqual(fresh["status"], "success", fresh)
kernel = next(iter(_KERNELS.values()))
kernel.attached += 1 # a cell is mid-flight: reaping must skip it
try:
_reap_once()
self.assertIn(kernel.key, _KERNELS)
self.assertTrue(kernel.alive())
finally:
kernel.attached -= 1
class TestStaleStagingDirSweep(unittest.TestCase):
def test_week_old_kernel_dirs_go_and_fresh_ones_stay(self):
import time as time_module
from tools.code_kernel import _sweep_stale_staging_dirs
with tempfile.TemporaryDirectory() as tmp:
with patch("tools.code_kernel.tempfile.gettempdir", return_value=tmp):
old = Path(tmp, "hermes_kernel_old")
young = Path(tmp, "hermes_kernel_young")
bystander = Path(tmp, "unrelated_dir")
for path in (old, young, bystander):
path.mkdir()
week_and_a_bit = time_module.time() - 8 * 86400
os.utime(old, (week_and_a_bit, week_and_a_bit))
removed = _sweep_stale_staging_dirs()
# Asserted inside the TemporaryDirectory: cleanup would flatten everything.
self.assertEqual(removed, 1)
self.assertFalse(old.exists())
self.assertTrue(young.exists())
self.assertTrue(bystander.exists())