## Summary On native Windows, connecting an agent that runs as a process chain failed every time with `[WinError 32] The process cannot access the file because it is being used by another process: '...\raven_subagent_ping_xxxx'`. The Claude Code preset is one such chain: `npx.cmd`, `cmd.exe`, the node adapter, then `claude.exe`. `ping_agent` handed the agent a fresh temporary directory to launch in and removed it when its `with` block ended, which was before the `finally` that closes the ping's pool and so ends the agent's process tree. Windows refuses to remove a directory that a live process has as its working directory, even an empty one, and the broad `except Exception` turned that removal error into the ping's verdict: an agent that had answered was reported as not answering a test message. A ping that timed out was misreported the same way, and each press left its directory in the temp folder. `verify_agent`, the handshake behind Test and behind the capability record a Connect writes once its ping passes, removed its session directory in the same order. The Claude Agent SDK starts `claude.exe` in the session's directory, so that check came back `unknown` with `verify failed: [WinError 32] ...`. Ending the tree was already right: the Job Object from #951 ends every process in it, including descendants started later. What was wrong was when the directory was removed, and what a failed removal was taken to mean. - `raven/utils/scratch.py` adds `ScratchDir`, a temporary directory that is removed only when its caller says the agent has ended. The removal is tried up to 5 times, 0.2 s apart, while a tree that was just ended finishes exiting. A directory that still cannot be removed is logged with its path and left in the temp folder, never raised. It removes through `TemporaryDirectory.cleanup`, so read-only entries are still cleared on Windows as before. - `ping_agent` removes its directory after `pool.close_all()`, and `verify_agent` after `client.close()`. The CLI test (`run_test`) and the Kimi Code print-mode ask use `ScratchDir` as an async context manager: their process has ended inside the block, and a refused removal no longer replaces the CLI's reply or raises out of the ask. - The `Windows ACP stdio` CI job runs a new real-process test of both checks against an agent of the reported shape. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Native Windows 11 (build 26300, Python 3.12) unless marked Linux. Reproduction with a stand-in agent of the reported shape: a `.cmd` wrapper starting a Python ACP agent that, on `session/new`, starts a worker in the session's directory and waits for it before answering. - `main`: the ping returns `ok=False` with `[WinError 32] ... raven_subagent_ping_...`; the handshake returns `unknown` with `verify failed: [WinError 32] ... raven_acp_verify_...`; both directories stay in `%TEMP%`. - This branch: the ping returns `ok=True` ("it ran and replied"), the handshake `ready`, and nothing stays in `%TEMP%`. Over 20 rounds of both checks, all 40 directories were removed on the first try after the close. ```text uv run --frozen --no-sync pytest tests/integration/test_subagent_connect_windows_real_process.py -m integration -q -n 0 ``` 1 passed. On `main` it fails with `[WinError 32] ... raven_subagent_ping_...`, the error text in the issue. New unit tests: 5 in `tests/test_utils_scratch.py`, 3 in `tests/test_subagent_acp.py`, 4 in `tests/test_subagent_probe.py` (the CLI one is POSIX-only, like the neighbours whose shell stand-in it shares) and 1 in `tests/test_subagent_kimi_code.py`. The ping and handshake ordering tests run the real pool, client and stub agent, with Windows' refusal stood in on any host by `tests/_windows_dirs.py`: a removal is refused while an agent the test launched is not yet closed. With the three changed modules put back to `main`, the six call-site tests fail, each with the symptom it names (the CLI one on Linux). Each new construct was then removed on its own, and a test goes red for every one: removal before the pool close, removal before the client close, either `None` guard, raising after the last try, a single try, no pause between tries, and `shutil.rmtree` in place of `TemporaryDirectory.cleanup`. That last one only shows on Windows, where a read-only file blocks the removal. Slices of the suite that reach the changed modules, this branch against `main` at a71b488e: - Linux, in Docker (`python3.12-bookworm-slim`, `--idle-ceiling-strict`), 20 test files plus the new one: 9 failed, 1718 passed, 88 errors on this branch; 8 failed, 1706 passed, 88 errors on `main`. The errors and eight of the failures are the same ids on both, in remote-machine and file-permission tests: the container has no `ps` and runs as root. The ninth, `test_a_ping_answers_from_the_machine_and_leaves_one_directory`, timed out once starting the machine's daemon (`did not answer within 20s`) while the host was under load; run alone it passed 3 of 3 times on both sides. - Windows, `tests/test_subagent_acp.py tests/test_subagent_probe.py tests/test_subagent_kimi_code.py tests/test_subagent_host_env.py tests/test_acp_windows.py tests/test_rpc_subagents.py tests/test_subagent_github_copilot.py`: 57 failed, 635 passed, 4 skipped on this branch; 61 failed, 619 passed, 3 skipped on `main`. No test fails only on this branch. Four tests that fail on `main` under Windows pass with this change, `test_the_roster_and_the_connect_button_read_one_answer` among them. For anyone reproducing on Windows: the suite isolates the home by setting `HOME`, which `Path.home()` ignores there in favour of `USERPROFILE`, so a run reads and writes the real `%USERPROFILE%\.raven` and tests can see each other's records. The Windows numbers above were taken with `USERPROFILE` pointed at an empty folder. That gap predates this change and is not addressed here. ```text uv run --frozen --no-sync ruff check <the 10 changed Python files> uv run --frozen --no-sync ruff format --check <the 10 changed Python files> uv run --frozen --no-sync ty check raven/utils/scratch.py raven/agent/subagent/probe.py raven/agent/subagent/kimi_code.py raven/acp_client/capabilities.py uv run --frozen --no-sync lint-imports ``` All passed, 10 import contracts kept. Not measured: the real `@agentclientprotocol/claude-agent-acp` adapter. The stand-in above takes its place. - [ ] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed The tests box stays unchecked because the slices above carry failures that `main` carries too. No user-facing text or docs change. ## Risk - POSIX behaviour is unchanged apart from order: the directory is removed on the first try, as before, now after the agent has ended rather than before. - On Windows a refused removal adds at most about 0.8 s to a check, and the directory is then left in the temp folder with a warning that names it. Before this change every such check left its directory there and failed. - A check whose agent leaves a process running outside its job, or a CLI that leaves a background worker, now reports the agent's answer where it used to report the removal error. - Rollback: revert the commit. No configuration, schema or stored state changes. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues Fixes #969 Co-authored-by: Claude (claude-opus-5-5[1m]) <noreply@anthropic.com>
636 lines
28 KiB
Python
636 lines
28 KiB
Python
"""Kimi Code's refusals, read the way Kimi Code 2.1.0 reports them.
|
|
|
|
Every sentence here is one Kimi Code printed, measured 2026-09-24 against a
|
|
stand-in endpoint answering each status and against a real account whose plan
|
|
does not include it -- except a membership's own limits, which no account at
|
|
hand could reach, given in its error reference's words. Over ACP it says only
|
|
"Authentication required" (bare, or
|
|
with the provider's status behind it) or nothing at all; `kimi -p` prints the
|
|
reason as ``error: failed to run prompt: ...``. A stand-in `kimi` plays both
|
|
halves: ``acp`` runs the stub ACP server, ``-p`` prints what the test gives it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import os
|
|
import shlex
|
|
import signal
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
import raven.agent.subagent.backends.env as env_mod
|
|
from raven.agent.subagent import kimi_code
|
|
from raven.agent.subagent.probe_state import Remedy
|
|
from raven.config.schema import ThirdPartyAcpSubagentConfig
|
|
from raven.utils.pid import pid_alive
|
|
from tests._windows_dirs import scratch_root, windows_removal
|
|
from tests._windows_kill import PYTHON, windows_kill_path
|
|
|
|
_STUB = Path(__file__).with_name("acp_stub_server.py")
|
|
|
|
PLAN = (
|
|
"Your current subscription does not have access to Kimi Code right now. Upgrade your plan to keep "
|
|
"coding with Kimi Code: https://www.kimi.com/code/#pricing"
|
|
)
|
|
PRICING = "https://www.kimi.com/code/#pricing"
|
|
NO_MODEL = "No model configured. Run `kimi` and use /login to sign in, then retry; or set default_model in config.toml."
|
|
SWITCH = ("kimi", "/model")
|
|
EDIT_PROVIDER = ("kimi", "/provider")
|
|
|
|
|
|
def _cfg(command: str = "kimi acp", **kw: Any) -> ThirdPartyAcpSubagentConfig:
|
|
return ThirdPartyAcpSubagentConfig(name="Kimi Code", preset="kimi_code", command=command, **kw)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _no_login_shell(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Hand the launch and the ask this process's environment rather than a login shell's.
|
|
|
|
What is under test is how Kimi Code's words are read, not the capture; the
|
|
first test to reach it would otherwise spend seconds running the developer's
|
|
own shell, and the stand-in `kimi` travels on each row's own PATH anyway.
|
|
"""
|
|
monkeypatch.setattr("raven.agent.subagent.backends.env._LOGIN_ENV", dict(os.environ))
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"said, remedy",
|
|
[
|
|
# Over ACP, on `session/prompt`.
|
|
(f"Authentication required: 403 {PLAN}", Remedy("plan", PRICING)),
|
|
("Authentication required: 401 Invalid Authentication", Remedy("setup", *EDIT_PROVIDER)),
|
|
# From `kimi -p`.
|
|
(f"provider.auth_error: 403 {PLAN}", Remedy("plan", PRICING)),
|
|
("provider.auth_error: 401 Invalid Authentication", Remedy("setup", *EDIT_PROVIDER)),
|
|
("The provided authorization grant is invalid", Remedy("sign_in", "kimi login")),
|
|
(NO_MODEL, Remedy("sign_in", "kimi login")),
|
|
("provider managed:kimi-code has no credential configured", Remedy("sign_in", "kimi login")),
|
|
("provider moonshotai-cn has no credential configured", Remedy("setup", *EDIT_PROVIDER)),
|
|
('Model "moonshotai-cn/no-such-model" is not configured in config.toml.', Remedy("model", *SWITCH)),
|
|
(
|
|
"provider.api_error: 402 This request requires more credits, or fewer max_tokens. You requested up to "
|
|
"32768 tokens, but can only afford 2048.",
|
|
Remedy("billing", *SWITCH),
|
|
),
|
|
("provider.api_error: 404 Not found the model some-model or Permission denied", Remedy("model", *SWITCH)),
|
|
(
|
|
"provider.rate_limit: 429 Your account org-test request reached organization max RPM: 3, please try "
|
|
"again after 1 seconds",
|
|
Remedy("quota", *SWITCH),
|
|
),
|
|
(
|
|
"provider.api_error: 429 Your account org-test is suspended due to insufficient balance, please "
|
|
"recharge your account or check your plan and billing details",
|
|
Remedy("billing", *SWITCH),
|
|
),
|
|
(
|
|
"no provider configured; complete onboarding via /login or the providers endpoint",
|
|
Remedy("sign_in", "kimi login"),
|
|
),
|
|
# Kimi Code's error reference: a signed-in account's own limits arrive as
|
|
# 401 or 403 too, and none of them is a key to replace.
|
|
(
|
|
"Authentication required: 403 You've reached your 5-hour usage limit. Your quota will reset when the "
|
|
"current 5-hour window ends.",
|
|
Remedy("quota", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 403 You've reached your weekly (7-day) usage limit. Your quota will reset "
|
|
"when the current 7-day window ends.",
|
|
Remedy("quota", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 403 You've reached your monthly usage limit for this billing cycle. Your "
|
|
"quota will be refreshed in the next cycle.",
|
|
Remedy("quota", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 403 You've reached your concurrent request limit. Please wait for your "
|
|
"ongoing requests to finish and try again.",
|
|
Remedy("quota", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 401 Your current subscription does not have access to k3. Upgrade to "
|
|
"higher-tier Kimi Code plans.",
|
|
Remedy("model", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 401 Your current plan supports only k3 up to 256K context. 1M context is "
|
|
"available on higher-tier Kimi Code plans.",
|
|
Remedy("model", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 401 Your current subscription does not have access to "
|
|
"kimi-for-coding-highspeed. Upgrade to higher-tier Kimi Code plans.",
|
|
Remedy("model", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 401 Your model id does not exist, recognized as other:claude-sonnet.",
|
|
Remedy("model", *SWITCH),
|
|
),
|
|
(
|
|
"Authentication required: 401 The API Key appears to be invalid or may have expired. Please verify "
|
|
"your credentials and try again.",
|
|
Remedy("setup", *EDIT_PROVIDER),
|
|
),
|
|
(
|
|
"provider.api_error: 402 We're unable to verify your membership benefits at this time. Please ensure "
|
|
"your membership is active.",
|
|
Remedy("plan"),
|
|
),
|
|
# Measured with a made-up key: one provider's words for a key it does not
|
|
# know. Any other 401 is a refused key however it is worded.
|
|
("Authentication required: 401 Missing Authentication header", Remedy("setup", *EDIT_PROVIDER)),
|
|
# A plan refusal wrapped in the provider's JSON, or ending a sentence,
|
|
# keeps only the address.
|
|
(f'Authentication required: 403 {{"error": {{"message": "{PLAN}"}}}}', Remedy("plan", PRICING)),
|
|
(f"Authentication required: 403 {PLAN}.", Remedy("plan", PRICING)),
|
|
],
|
|
)
|
|
def test_kimi_codes_own_words_name_the_fix(said: str, remedy: Remedy) -> None:
|
|
"""Every refusal Kimi Code names is told as the fix it needs, not as "sign in".
|
|
|
|
Over ACP all of these but the no-reason ones read "Authentication
|
|
required", which the general reading takes for a missing sign-in: measured
|
|
on a signed-in account, the page said "sign in to Kimi Code first" for a
|
|
plan that does not include it, and signing in again changed nothing.
|
|
"""
|
|
verdict = kimi_code.read(_cfg(), said)
|
|
assert verdict is not None
|
|
assert verdict[2] == remedy
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"said",
|
|
[
|
|
"Authentication required",
|
|
"provider.api_error: 400 Invalid request: the max_tokens is too large",
|
|
"Provider safety policy blocked the response.",
|
|
"provider.connection_error: Connection error.",
|
|
# A 401 or 403 that does not read as a credential is not told as one.
|
|
"Authentication required: 403 We consider the current URL poses a security risk",
|
|
# A port is not a status.
|
|
"connect ECONNREFUSED 127.0.0.1:403",
|
|
],
|
|
)
|
|
def test_words_that_name_no_fix_are_left_as_they_came(said: str) -> None:
|
|
"""A bare refusal is asked about; one with a reason this cannot fix keeps its own words."""
|
|
assert kimi_code.read(_cfg(), said) is None
|
|
|
|
|
|
def _through_kimi(
|
|
tmp_path: Path,
|
|
mode: str,
|
|
*,
|
|
says: str | None = None,
|
|
doctor: str | None = None,
|
|
message: str | None = None,
|
|
no_acp: bool = False,
|
|
slow_ask: bool = False,
|
|
ask_worker: bool = False,
|
|
ansi: bool = False,
|
|
command: str = "kimi acp",
|
|
**kw: Any,
|
|
) -> ThirdPartyAcpSubagentConfig:
|
|
"""The Kimi Code preset, run by a stand-in `kimi` on PATH.
|
|
|
|
``acp`` starts the stub server in ``mode``; ``-p`` prints ``says`` the way
|
|
Kimi Code 2.1.0 does (in colour with ``ansi``), or answers when there is
|
|
nothing to say, and notes that it was asked; with ``slow_ask`` it notes its
|
|
pid and never finishes, and with ``ask_worker`` as well it first starts a
|
|
process of its own and notes that pid too. ``doctor config`` reports
|
|
``doctor`` as its one issue.
|
|
"""
|
|
bin_dir = tmp_path / "bin"
|
|
bin_dir.mkdir(exist_ok=True)
|
|
asked = tmp_path / "asked"
|
|
start = (
|
|
"printf '%s\\n' \"error: unknown command 'acp'\" >&2; exit 1"
|
|
if no_acp
|
|
else f"exec {shlex.quote(sys.executable)} {shlex.quote(str(_STUB))}"
|
|
)
|
|
worker = (
|
|
f" sleep 30 & printf '%s\\n' \"$!\" > {shlex.quote(str(tmp_path / 'worker.pid'))}\n" if ask_worker else ""
|
|
)
|
|
slow = f" printf '%s\\n' \"$$\" > {shlex.quote(str(tmp_path / 'ask.pid'))}; exec sleep 30\n" if slow_ask else ""
|
|
red, plain = ("\\033[91m", "\\033[39m") if ansi else ("", "")
|
|
kimi = bin_dir / "kimi"
|
|
kimi.write_text(
|
|
"#!/bin/sh\n"
|
|
'case "$1" in\n'
|
|
f" acp) {start} ;;\n"
|
|
" -p)\n"
|
|
f" printf '%s\\n' \"$*\" >> {shlex.quote(str(asked))}\n"
|
|
f"{worker}"
|
|
f"{slow}"
|
|
' if [ -n "$KIMI_STUB_SAYS" ]; then\n'
|
|
" printf '%s\\n' 'kimi version 2.1.0' >&2\n"
|
|
f" printf '{red}error: failed to run prompt: %s{plain}\\n' \"$KIMI_STUB_SAYS\" >&2\n"
|
|
" printf '%s\\n' 'See log: /nowhere/logs/kimi-code.log' >&2\n"
|
|
" exit 1\n"
|
|
" fi\n"
|
|
" printf '%s\\n' 'PONG'; exit 0 ;;\n"
|
|
" doctor)\n"
|
|
' if [ -n "$KIMI_STUB_DOCTOR" ]; then\n'
|
|
" printf '%s\\n' 'Kimi doctor found 1 issue.' '' 'ERROR config.toml /nowhere/config.toml' >&2\n"
|
|
" printf ' %s\\n' \"$KIMI_STUB_DOCTOR\" >&2\n"
|
|
" exit 1\n"
|
|
" fi\n"
|
|
" printf '%s\\n' 'OK config.toml /nowhere/config.toml'; exit 0 ;;\n"
|
|
"esac\n"
|
|
"exit 2\n"
|
|
)
|
|
kimi.chmod(0o755)
|
|
env = {"ACP_STUB_MODE": mode, "PATH": f"{bin_dir}{os.pathsep}{os.environ['PATH']}"}
|
|
for key, value in (("KIMI_STUB_SAYS", says), ("KIMI_STUB_DOCTOR", doctor), ("ACP_STUB_MESSAGE", message)):
|
|
if value is not None:
|
|
env[key] = value
|
|
return _cfg(
|
|
command=command.replace("{bin}", str(bin_dir)),
|
|
env=env,
|
|
ready_timeout_ms=kw.pop("ready_timeout_ms", 15000),
|
|
**kw,
|
|
)
|
|
|
|
|
|
def _asked(tmp_path: Path) -> list[str]:
|
|
log = tmp_path / "asked"
|
|
return log.read_text().splitlines() if log.exists() else []
|
|
|
|
|
|
def _alive(pid: int) -> bool:
|
|
try:
|
|
os.kill(pid, 0)
|
|
except ProcessLookupError:
|
|
return False
|
|
return True
|
|
|
|
|
|
async def test_a_plan_that_does_not_include_kimi_code_is_named_with_its_page(tmp_path: Path) -> None:
|
|
"""Measured on a real signed-in account: the refusal carries the reason, so nothing is asked."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "kimi_refuses", message=f"Authentication required: 403 {PLAN}"))
|
|
assert pinged.ok is False
|
|
assert pinged.remedy == Remedy("plan", PRICING)
|
|
assert pinged.detail.startswith("it is signed in, but the account's plan does not include Kimi Code; ")
|
|
assert PLAN in pinged.detail, "Kimi Code's own words are kept for the fold"
|
|
assert _asked(tmp_path) == []
|
|
|
|
|
|
async def test_a_refused_key_is_put_right_in_kimi_codes_provider_list(tmp_path: Path) -> None:
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
cfg = _through_kimi(tmp_path, "kimi_refuses", message="Authentication required: 401 Invalid Authentication")
|
|
pinged = await ping_agent(cfg)
|
|
assert pinged.remedy == Remedy("setup", *EDIT_PROVIDER)
|
|
assert "(401)" in pinged.detail
|
|
assert _asked(tmp_path) == []
|
|
|
|
|
|
async def test_a_session_kimi_code_will_not_start_is_asked_about(tmp_path: Path) -> None:
|
|
"""The bare refusal says nothing, so `kimi -p` is asked, once, with the connect's own prompt."""
|
|
from raven.agent.subagent.probe import PROBE_PROMPT, ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "kimi_not_ready", says=NO_MODEL))
|
|
assert pinged.remedy == Remedy("sign_in", "kimi login")
|
|
assert pinged.detail.startswith("it has no model to use: it is not signed in and has no model provider set up; ")
|
|
assert _asked(tmp_path) == [f"-p {PROBE_PROMPT}"]
|
|
|
|
|
|
async def test_a_config_that_does_not_parse_is_named_by_kimi_doctor(tmp_path: Path) -> None:
|
|
"""Measured: a TOML syntax error reads as "No model configured" to `kimi -p`; signing in would not fix it."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
broken = "Invalid TOML in /nowhere/config.toml: only letter, numbers, dashes and underscores are allowed in keys (line 3, column 12)"
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "kimi_not_ready", says=NO_MODEL, doctor=broken))
|
|
assert pinged.remedy == Remedy("config", "kimi doctor config")
|
|
assert "(line 3, column 12)" in pinged.detail
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"says, remedy",
|
|
[
|
|
("The provided authorization grant is invalid", Remedy("sign_in", "kimi login")),
|
|
('Model "moonshotai-cn/no-such-model" is not configured in config.toml.', Remedy("model", *SWITCH)),
|
|
("provider.api_error: 402 This request requires more credits", Remedy("billing", *SWITCH)),
|
|
],
|
|
)
|
|
async def test_a_turn_kimi_code_ended_with_nothing_is_asked_about(tmp_path: Path, says: str, remedy: Remedy) -> None:
|
|
"""A sign-in that no longer renews, 402, 404: over ACP each is an empty turn and nothing else."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "empty_turn", says=says))
|
|
assert pinged.remedy == remedy
|
|
assert says in pinged.detail
|
|
assert len(_asked(tmp_path)) == 1
|
|
|
|
|
|
async def test_words_kimi_code_gives_that_name_no_fix_are_shown_instead_of_the_empty_turn(tmp_path: Path) -> None:
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(
|
|
_through_kimi(tmp_path, "empty_turn", says="Provider safety policy blocked the response.")
|
|
)
|
|
assert pinged.remedy is None
|
|
assert pinged.detail.endswith("asked again with `kimi -p`, it said: Provider safety policy blocked the response.")
|
|
|
|
|
|
async def test_an_empty_turn_that_answers_when_asked_again_says_so(tmp_path: Path) -> None:
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "empty_turn"))
|
|
assert pinged.ok is False
|
|
assert pinged.remedy is None
|
|
assert "it answered, so the failure may have passed" in pinged.detail
|
|
|
|
|
|
@pytest.mark.production_timing
|
|
async def test_an_ask_that_runs_out_leaves_the_connects_own_verdict(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Retried failures take `kimi -p` about 150 s; the ask gives up rather than hold the connect.
|
|
|
|
It waits the cap out, a second here, to prove it holds.
|
|
"""
|
|
from raven.agent.subagent.probe import PROBE_PROMPT, ping_agent
|
|
|
|
monkeypatch.setattr(kimi_code, "_ASK_TIMEOUT_S", 1.0)
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "kimi_not_ready", says=NO_MODEL, slow_ask=True))
|
|
# The general reading of a bare credential refusal, with the preset's sign-in.
|
|
assert pinged.remedy == Remedy("sign_in", "kimi login")
|
|
assert pinged.detail.startswith("it is installed but has no usable credential; ")
|
|
assert _asked(tmp_path) == [f"-p {PROBE_PROMPT}"]
|
|
assert not _alive(int((tmp_path / "ask.pid").read_text())), "the `kimi` asked goes when the ask does"
|
|
|
|
|
|
async def test_a_connect_that_goes_away_mid_ask_takes_the_kimi_it_asked(tmp_path: Path) -> None:
|
|
"""The page that asked going away cancels the connect; the `kimi -p` it started does not outlive it."""
|
|
asking = asyncio.create_task(kimi_code.ask(_through_kimi(tmp_path, "ok", slow_ask=True), "hi"))
|
|
started = tmp_path / "ask.pid"
|
|
for _ in range(500):
|
|
if started.exists() and started.read_text().strip():
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
pid = int(started.read_text())
|
|
assert _alive(pid)
|
|
asking.cancel()
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await asking
|
|
assert not _alive(pid)
|
|
|
|
|
|
def _python_kimi(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, says: str | None = None
|
|
) -> ThirdPartyAcpSubagentConfig:
|
|
"""The Kimi Code preset, its `kimi -p` a script on ``PYTHON`` that notes its pid and does not finish.
|
|
|
|
With ``says``, it prints that as Kimi Code's failure line and exits 1 instead.
|
|
Found by name through ``resolve_program``, the way the ask finds the real one;
|
|
a script on PATH would not start on Windows, where a bare name means ``.exe``.
|
|
"""
|
|
stub, noted = tmp_path / "kimi.py", tmp_path / "ask.pid"
|
|
stub.write_text(
|
|
"import os, time\n"
|
|
f"with open({str(noted)!r} + '.tmp', 'w') as out:\n"
|
|
" out.write(str(os.getpid()))\n"
|
|
f"os.replace({str(noted)!r} + '.tmp', {str(noted)!r})\n"
|
|
"time.sleep(30)\n"
|
|
if says is None
|
|
else f"import sys\nprint({f'error: failed to run prompt: {says}'!r}, file=sys.stderr)\nsys.exit(1)\n",
|
|
encoding="utf-8",
|
|
)
|
|
real = env_mod.resolve_program
|
|
|
|
def resolve(argv, env, **kwargs):
|
|
if argv and argv[0] == "kimi":
|
|
return [PYTHON, str(stub), *argv[1:]]
|
|
return real(argv, env, **kwargs)
|
|
|
|
monkeypatch.setattr(env_mod, "resolve_program", resolve)
|
|
return _cfg()
|
|
|
|
|
|
async def test_a_connect_that_goes_away_mid_ask_ends_its_kimi_through_taskkill_without_killpg(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Windows has no ``os.killpg``: calling it raised ``AttributeError`` in place of the cancel."""
|
|
calls = windows_kill_path(monkeypatch)
|
|
asking = asyncio.create_task(kimi_code.ask(_python_kimi(tmp_path, monkeypatch), "hi"))
|
|
started = tmp_path / "ask.pid"
|
|
for _ in range(200):
|
|
if started.exists():
|
|
break
|
|
await asyncio.sleep(0.05)
|
|
pid = int(started.read_text())
|
|
|
|
asking.cancel()
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await asking
|
|
|
|
assert calls == [["/T", "/F", "/PID", str(pid)]]
|
|
assert not pid_alive(pid)
|
|
|
|
|
|
async def test_an_ask_that_runs_out_without_killpg_still_says_nothing(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""A ``taskkill`` that does not answer still leaves the `kimi` ended, and the ask still says nothing."""
|
|
calls = windows_kill_path(monkeypatch, fails=subprocess.TimeoutExpired("taskkill", 5))
|
|
monkeypatch.setattr(kimi_code, "_ASK_TIMEOUT_S", 1.0)
|
|
|
|
assert await kimi_code.ask(_python_kimi(tmp_path, monkeypatch), "hi") == (None, "")
|
|
|
|
assert [call[:3] for call in calls] == [["/T", "/F", "/PID"]]
|
|
assert not pid_alive(int(calls[0][3]))
|
|
|
|
|
|
async def test_an_ask_whose_directory_stays_in_use_returns_what_kimi_code_said(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""On Windows a process the `kimi` left running keeps its directory, and the refusal used to raise.
|
|
|
|
It raised out of the ask, and so out of the connect's own ping, which
|
|
promises never to.
|
|
"""
|
|
cfg = _python_kimi(tmp_path, monkeypatch, says="No model configured.")
|
|
root = scratch_root(tmp_path, monkeypatch)
|
|
refused = windows_removal(monkeypatch, lambda path: path.parent == root)
|
|
|
|
assert await kimi_code.ask(cfg, "hi") == (1, "No model configured.")
|
|
assert refused, "no removal was refused, so this tested nothing"
|
|
|
|
|
|
@pytest.mark.skipif(not hasattr(os, "killpg"), reason="process groups are POSIX")
|
|
async def test_ask_cancelled_once_running_takes_what_its_kimi_started(tmp_path: Path) -> None:
|
|
"""Ending an ask takes the group, not only the `kimi`, so what the `kimi` started goes with it.
|
|
|
|
Cancelled once the `kimi` is up rather than on the clock: the timeout runs
|
|
from the spawn, so an ask given a sliver of the cap is killed while the
|
|
`kimi` is still starting and has no worker to take. Readiness comes from
|
|
`ask.pid`, which the stand-in writes only after the worker is noted.
|
|
"""
|
|
asking = asyncio.create_task(kimi_code.ask(_through_kimi(tmp_path, "ok", slow_ask=True, ask_worker=True), "hi"))
|
|
started = tmp_path / "ask.pid"
|
|
for _ in range(500):
|
|
if started.exists() and started.read_text().strip():
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
assert started.exists() and started.read_text().strip(), "the `kimi` never came up"
|
|
|
|
worker = int((tmp_path / "worker.pid").read_text())
|
|
assert _alive(worker), "the ask was cancelled before its `kimi` started a worker to take"
|
|
asking.cancel()
|
|
try:
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await asking
|
|
# The group kill is signalled, not awaited: give the worker time to go.
|
|
for _ in range(40):
|
|
if not _alive(worker):
|
|
break
|
|
await asyncio.sleep(0.05)
|
|
assert not _alive(worker), "what the `kimi` started outlives the ask"
|
|
finally:
|
|
if _alive(worker):
|
|
os.kill(worker, signal.SIGKILL)
|
|
|
|
|
|
async def test_a_coloured_answer_is_read_like_a_plain_one(tmp_path: Path) -> None:
|
|
"""With FORCE_COLOR set, `kimi -p` prints its error line in colour (measured); the colour is not the reason."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "kimi_not_ready", says=NO_MODEL, ansi=True))
|
|
assert pinged.remedy == Remedy("sign_in", "kimi login")
|
|
assert pinged.detail.startswith("it has no model to use: ")
|
|
assert "\x1b" not in pinged.detail
|
|
|
|
|
|
async def test_a_bare_refusal_asked_about_in_words_it_does_not_know_still_reads_as_a_sign_in(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Asking only adds to the general reading: words it cannot place (made up here) keep "sign in" and are shown."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "kimi_not_ready", says="Something unexpected happened."))
|
|
assert pinged.remedy == Remedy("sign_in", "kimi login")
|
|
assert pinged.detail.endswith("asked again with `kimi -p`, it said: Something unexpected happened.")
|
|
|
|
|
|
async def test_a_prompt_kimi_code_never_finishes_names_the_command_that_says_why(tmp_path: Path) -> None:
|
|
"""Its retried failures outlast the wait: the general `silent` verdict, with Kimi Code's own diagnose command."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "prompt_hangs", timeout=1))
|
|
assert pinged.remedy == Remedy("silent", "kimi -p hi")
|
|
assert _asked(tmp_path) == [], "a retry that outlasts the connect would outlast the ask too"
|
|
|
|
|
|
async def test_a_start_that_never_finishes_its_handshake_is_not_a_working_kimi_code(tmp_path: Path) -> None:
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "silent", ready_timeout_ms=1000))
|
|
assert pinged.remedy == Remedy("upgrade")
|
|
assert pinged.detail.startswith("it started but never finished its ACP handshake")
|
|
|
|
|
|
async def test_a_kimi_that_does_not_know_acp_is_told_to_upgrade(tmp_path: Path) -> None:
|
|
"""commander's spelling, "error: unknown command 'acp'", read as the too-old verdict yargs' already is."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "ok", no_acp=True))
|
|
assert pinged.remedy == Remedy("upgrade")
|
|
assert pinged.detail.startswith("it is too old to be connected: it does not know the flag or command")
|
|
|
|
|
|
async def test_a_row_whose_command_was_edited_is_not_asked_about(tmp_path: Path) -> None:
|
|
"""Its command is its operator's: the general reading stands, and its executable is not run a second way."""
|
|
from raven.agent.subagent.probe import ping_agent
|
|
|
|
pinged = await ping_agent(_through_kimi(tmp_path, "kimi_not_ready", says=NO_MODEL, command="{bin}/kimi acp"))
|
|
assert pinged.remedy == Remedy("sign_in", "kimi login")
|
|
assert _asked(tmp_path) == []
|
|
|
|
|
|
async def test_the_test_button_asks_about_a_refused_handshake_too(tmp_path: Path) -> None:
|
|
"""Test stops at the handshake that Connect's ping would have failed on, and names the same fix."""
|
|
from raven.agent.subagent.probe import run_test
|
|
|
|
cfg = _through_kimi(
|
|
tmp_path, "kimi_not_ready", says='Model "moonshotai-cn/no-such-model" is not configured in config.toml.'
|
|
)
|
|
tested = await run_test(cfg, source="config")
|
|
assert tested.ok is False
|
|
assert tested.remedy == Remedy("model", *SWITCH)
|
|
|
|
|
|
async def test_the_test_button_names_an_unfinished_handshake_the_way_the_connect_does(tmp_path: Path) -> None:
|
|
from raven.agent.subagent.probe import run_test
|
|
|
|
tested = await run_test(_through_kimi(tmp_path, "silent", ready_timeout_ms=1000), source="config")
|
|
assert tested.remedy == Remedy("upgrade")
|
|
|
|
|
|
def test_kimi_code_is_in_each_table_a_verdict_reads_its_commands_from() -> None:
|
|
from raven.agent.subagent.presets import diagnose_hint_for, model_switch_hint_for, sign_in_hint_for
|
|
|
|
cfg = _cfg()
|
|
sign_in = sign_in_hint_for(cfg)
|
|
assert sign_in is not None and sign_in.local == "kimi login"
|
|
switch = model_switch_hint_for(cfg)
|
|
assert switch is not None and (switch.command, switch.then) == SWITCH
|
|
assert diagnose_hint_for(cfg) == "kimi -p hi"
|
|
|
|
|
|
def test_the_plan_and_config_kinds_travel_to_the_page() -> None:
|
|
assert Remedy.from_wire(Remedy("plan", PRICING).to_wire()) == Remedy("plan", PRICING)
|
|
assert Remedy.from_wire(Remedy("config", "kimi doctor config").to_wire()) == Remedy("config", "kimi doctor config")
|
|
assert Remedy.from_wire({"kind": "plan"}) == Remedy("plan")
|
|
|
|
|
|
async def test_on_windows_the_kimi_asked_is_the_one_on_the_childs_path(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The ask runs the ``kimi`` the connect started, so it is looked up the way the launch looks it up.
|
|
|
|
On Windows that is the child's own PATH, under CreateProcess's rule that a name
|
|
with no extension means ``.exe``; the gateway's PATH is the one CreateProcess
|
|
would have searched, and an agent installed after raven started is not on it.
|
|
"""
|
|
bin_dir = tmp_path / "bin"
|
|
bin_dir.mkdir()
|
|
kimi = bin_dir / "kimi.exe"
|
|
kimi.write_text("#!/bin/sh\nprintf '%s\\n' 'kimi, version 9.9.9'\n", encoding="utf-8")
|
|
kimi.chmod(0o755)
|
|
monkeypatch.setattr(env_mod, "_on_windows", lambda: True)
|
|
monkeypatch.setattr(env_mod, "_LOGIN_ENV", {"PATH": str(bin_dir)})
|
|
|
|
assert await kimi_code._run(ThirdPartyAcpSubagentConfig(name="Kimi Code", command="kimi acp"), "--version") == (
|
|
0,
|
|
"kimi, version 9.9.9\n",
|
|
)
|
|
|
|
|
|
async def test_on_windows_the_kimi_ask_never_starts_a_batch_file(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The ask puts its prompt in argv (``-p``), which a ``.cmd`` would hand to cmd.exe's parser.
|
|
|
|
So with only a batch file on the child's PATH the ask finds no program and
|
|
says nothing, rather than running it.
|
|
"""
|
|
bin_dir = tmp_path / "bin"
|
|
bin_dir.mkdir()
|
|
batch = bin_dir / "kimi.cmd"
|
|
batch.write_text("#!/bin/sh\nprintf '%s\\n' 'the batch file ran'\n", encoding="utf-8")
|
|
batch.chmod(0o755)
|
|
monkeypatch.setattr(env_mod, "_on_windows", lambda: True)
|
|
monkeypatch.setattr(env_mod, "_LOGIN_ENV", {"PATH": str(bin_dir)})
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="Kimi Code", command="kimi acp")
|
|
assert await kimi_code._run(cfg, "-p", "a & b") == (None, "")
|