mirror of
https://github.com/microsoft/vscode.git
synced 2026-08-03 01:13:12 +08:00
018354116a
* Restructure agent host E2E tests into conformance and parity tiers The E2E suite ran every test once per provider, so 52 provider-invariant tests were executed three times on each of three operating systems for a single meaningful assertion. Split the suite into two tiers: - conformance: provider-invariant Agent Host Protocol behavior, run once against a single reference provider - parity: behavior that must be verified separately for Claude, Copilot, and Codex Add an IAgentHostTarget seam so the suite can launch a non-VS Code Agent Host Protocol implementation, keeping the tests external to the implementation under test. Track protocol-surface coverage (commands, notifications, and action types observed on the wire) alongside the existing line coverage, and check the stats in so gaps are visible in review. Freeze the protocol/ suite: it side-loads a mock agent into the production server, so it cannot be run against an alternate implementation. Record the migration backlog in the E2E README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Record request-assertion and line-ending gaps in E2E known issues Document two structural gaps found while reviewing the record/replay design: - The recorded model request is normalized on write but never read back, so replay cannot detect regressions in prompt assembly, history retention, or attachment marshalling. Records the projected-assertion approach and why tool result text must be elided from it. - Snapshot normalization does not handle line endings, so any snapshot carrying literal text can fail on Windows for reasons unrelated to the behavior under test. Also note the portable-command guidance (`node -e` / a seeded script) for scenarios that genuinely need to run a command. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback on E2E coverage tooling - Derive the conformance suite title from the provider config instead of passing it separately, matching defineAgentHostE2ETests and removing a second source of truth for the suite name. - Warn once when protocol-surface observations cannot be written. The write is still non-fatal, but a silent failure previously surfaced much later as the coverage script reporting a missing observation file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Gate the Claude side-chat context test on a server-tool wiring race `side chat receives bounded source context without copied history` fails intermittently on Claude with `Server not found: host`, raised by the Claude CLI when a replayed turn calls a server tool before the host's server-tool MCP server is registered for that session. Separating the conformance and parity tiers concentrated the Claude parity suite into consecutive model-backed turns where interleaved host-only tests previously spaced them out, which loses this race about half the time. Measured at ~4 failures in 8 full-suite runs, against 0 in 6 runs before the split; the test passes in isolation and with a fresh server per test, so it is a materialization race rather than replay or shared-server state. Gate it behind `sideChatServerToolWiringUnstable` and record the measurements in KNOWN_ISSUES.md, including a note not to re-record the capture to make it pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Link the side-chat race to its tracking issue Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop the side-chat gate now that the underlying race is fixed #327560 fixed the root cause: the Claude session published the host's own in-process `host` and `client` MCP bridges into session-scoped state, so a peer or side chat whose query had not yet reported `host` tried to toggle it and the CLI answered `Server not found: host`. That is the same failure this branch quarantined, so remove the `sideChatServerToolWiringUnstable` gate and its known-issue entry rather than carrying a stale workaround. Verified with the gate removed: 4 clean Claude runs and 2 clean full-suite runs (150 passing, 0 failing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>