552 lines
25 KiB
Markdown
Raw Permalink Normal View History

2025-10-23 09:27:04 +08:00
# Observability with OpenTelemetry
Learn how to enable and setup OpenTelemetry for Qwen Code.
- [Observability with OpenTelemetry](#observability-with-opentelemetry)
- [Key Benefits](#key-benefits)
- [OpenTelemetry Integration](#opentelemetry-integration)
- [Configuration](#configuration)
2025-12-15 09:47:03 +08:00
- [Aliyun Telemetry](#aliyun-telemetry)
- [Manual OTLP Export](#manual-otlp-export)
2025-10-23 09:27:04 +08:00
- [Local Telemetry](#local-telemetry)
- [File-based Output (Recommended)](#file-based-output-recommended)
2025-12-15 09:47:03 +08:00
- [Collector-Based Export (Advanced)](#collector-based-export-advanced)
2025-10-23 09:27:04 +08:00
- [Logs and Metrics](#logs-and-metrics)
- [Logs](#logs)
- [Metrics](#metrics)
## Key Benefits
- **๐Ÿ” Usage Analytics**: Understand interaction patterns and feature adoption
across your team
- **โšก Performance Monitoring**: Track response times, token consumption, and
resource utilization
- **๐Ÿ› Real-time Debugging**: Identify bottlenecks, failures, and error patterns
as they occur
- **๐Ÿ“Š Workflow Optimization**: Make informed decisions to improve
configurations and processes
- **๐Ÿข Enterprise Governance**: Monitor usage across teams, track costs, ensure
compliance, and integrate with existing monitoring infrastructure
## OpenTelemetry Integration
Built on **[OpenTelemetry]** โ€” the vendor-neutral, industry-standard
observability framework โ€” Qwen Code's observability system provides:
2025-12-15 09:47:03 +08:00
- **Universal Compatibility**: Export to any OpenTelemetry backend (Aliyun,
Jaeger, Prometheus, Datadog, etc.)
2025-10-23 09:27:04 +08:00
- **Standardized Data**: Use consistent formats and collection methods across
your toolchain
- **Future-Proof Integration**: Connect with existing and future observability
infrastructure
- **No Vendor Lock-in**: Switch between backends without changing your
instrumentation
[OpenTelemetry]: https://opentelemetry.io/
[aliyun-opentelemetry-overview]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/product-overview/what-is-tracing-analysis
[aliyun-opentelemetry-get-started]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/before-you-begin
[aliyun-opentelemetry-console-cn]: https://trace.console.aliyun.com
[aliyun-opentelemetry-console-cn-legacy]: https://tracing.console.aliyun.com
[aliyun-opentelemetry-console-intl]: https://arms.console.alibabacloud.com
2025-10-23 09:27:04 +08:00
## Configuration
All telemetry behavior is controlled through your `.qwen/settings.json` file.
These settings can be overridden by environment variables or CLI flags.
docs(telemetry): align config and docs semantics for target, outfile, and CLI flags (#4066) * docs(telemetry): align config and docs semantics for target, outfile, and CLI flags - Remove stale warning note "This feature requires corresponding code changes" โ€” the OTLP implementation is now complete (#3779, #4061) - Clarify that `target` is an informational destination label and does not control exporter routing; `otlpEndpoint` or `outfile` must be set to configure where data is sent - Mark `--telemetry-target` CLI flag as deprecated in the configuration table to match the deprecateOption() call in cli/src/config/config.ts - Fix `outfile` / `QWEN_TELEMETRY_OUTFILE` descriptions: remove the incorrect "when target is local" qualifier โ€” outfile overrides OTLP export regardless of the target value - Simplify the file-based output example by removing the now-redundant `"target": "local"` and `"otlpEndpoint": ""` fields Closes the "Align telemetry config and docs semantics for target, useCollector, otlpEndpoint, otlpProtocol, and outfile" checklist item in #3731. ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): address Copilot review comments on outfile and target descriptions - Fix outfile table row in telemetry.md: "overrides `otlpEndpoint`" โ†’ "overrides OTLP export" (outfile disables all OTLP exporting, not just the base endpoint) - Use fully-qualified setting names (`telemetry.otlpEndpoint`, `telemetry.outfile`) in the target description in settings.md for consistency with the rest of the table ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): update QWEN_TELEMETRY_TARGET env var description and add outfile note - Align QWEN_TELEMETRY_TARGET env var description with the updated telemetry.target setting semantics (informational label, not routing) - Add a note after the file-based output example clarifying that outfile automatically disables OTLP export ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)
2026-05-13 08:27:41 +08:00
| Setting | Environment Variable | CLI Flag | Description | Values | Default |
| -------------------------------- | -------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ----------------------- |
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing โ€” set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` |
docs(serve): v0.16-alpha known limits + SDK QWEN_SERVER_TOKEN env fallback (PR 27) (#4473) * docs(serve): v0.16-alpha known limits + SDK QWEN_SERVER_TOKEN env fallback (PR 27) First PR in the F5 release chain (PR 27 โ†’ 28 โ†’ 30a โ†’ 31) per the 2026-05-24 v0.16-alpha scope freeze in #4175 (text-only chat / coding + local-only deployment). ## SDK ergonomic micro-change (~50 LOC + 4 tests) `DaemonClient` constructor falls back to `QWEN_SERVER_TOKEN` env var when `opts.token` is absent โ€” closes the asymmetry where the daemon side already honors this var (--token CLI flag fallback, already in main since PR 15) but the SDK forced clients to thread it through every construction. Properties: - Browser-safe via `globalThis.process` indirection (the SDK is imported by @qwen-code/webui; literal process.env access would explode at module load on browser bundles) - Whitespace stripped (matches daemon-side trim โ€” handy for `export QWEN_SERVER_TOKEN=\"\$(cat token.txt)\"` where cat adds a trailing newline) - Empty / whitespace-only treated as unset (a stale `export QWEN_SERVER_TOKEN=\"\"` won't accidentally send Authorization: Bearer with no token) - Resolved at construction, not lazily per-request (later process.env mutations don't affect already-built clients) - Explicit opts.token wins over env Tests: 4 new in DaemonClient.test.ts `bearer auth` describe covering env fallback / explicit-wins / empty-treated-unset / whitespace-stripped. Plus a defensive snapshot/restore on the existing 'omits Authorization when no token' test so an inherited test-runner export of QWEN_SERVER_TOKEN doesn't turn that assertion into a false positive. This SDK fallback is the entire ergonomic replacement for PR 29's SDK env/file fallback. PR 29's other features (auto-gen daemon token, instance-path keying, stale cleanup) remain deferred to v0.16.x โ€” all are DX improvements over the boot-time security gate already shipped in PR 15. ## v0.16-alpha docs (~120 LOC markdown) - docs/users/qwen-serve.md: new "v0.16-alpha known limits" section enumerating product surface (text-only โœ…, multimodal โŒ), deployment surface (local launchers โœ…, containerized โŒ, multi- daemon โŒ, BYO-token โœ…), and hardening posture (boot security gate โœ…, mutation gate โœ…, MCP guardrails โœ…, prompt absolute deadline โธ๏ธ, rate limiting โธ๏ธ, --max-body-size โธ๏ธ). Adds an alpha banner at the top of the file. - docs/developers/examples/daemon-client-quickstart.md: documents the SDK env fallback in both the Hello-daemon intro and the Authentication section, with the "export + no-token-arg" recommended path called out for local dev. Verification: 125/125 DaemonClient.test.ts pass (121 existing + 4 new); 4/4 daemon-public-surface.test.ts pass (constructor signature unchanged); tsc clean on packages/sdk-typescript; eslint --max-warnings 0 clean on touched .ts files. Part of #4175. * fix(sdk): #4473 round 1 fold-in โ€” 2 copilot doc threads adopted T1 [copilot DaemonClient.ts:144 โ€” stale line refs in readTokenFromEnv JSDoc]: removed `runQwenServe.ts:175` (token resolution actually lives at line 302-318 today, would drift again on next refactor) and `docs/users/qwen-serve.md:173`. Replaced with stable symbol/section references ("runQwenServe token-resolution path"; "qwen-serve user guide CLI flags section"). T2 [copilot daemon-client-quickstart.md:33 โ€” `~/.qwen/server-token` implies built-in path that doesn't exist]: PR 27 explicitly defers token auto-generation + file-store fallback (PR 29 deferred features). The example incorrectly suggested a standard file location. Replaced with two explicit user-managed alternatives: - `openssl rand -hex 32` one-shot - `cat ./my-token-file` user-managed file Both threads were accurate suggestions caught at the right time (zero behavior change; pure docstring/example accuracy). Verification: 125/125 DaemonClient tests pass; tsc + eslint clean on touched files.
2026-05-24 23:57:39 +08:00
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,โ€ฆ` | `{}` |
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |
2025-10-23 09:27:04 +08:00
**Note on boolean environment variables:** For the boolean settings (`enabled`,
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
corresponding environment variable to `true` or `1` will enable the feature. Any
other value will disable it.
**Sensitive span attributes:** When `includeSensitiveSpanAttributes` is enabled,
two things happen:
1. **Native span attributes (`qwen-code.interaction`, `api.generateContent*`,
`tool.<name>`)** carry verbatim conversation content:
- User prompts (`new_context`)
- System prompts (`system_prompt` โ€” full text once per session, deduped by
SHA-256 hash; subsequent spans only carry `system_prompt_hash` +
`system_prompt_preview` + `system_prompt_length`)
- Tool schemas (emitted as `tool_schema` events, also hash-deduped)
- Tool inputs (`tool_input`) and tool results (`tool_result`)
- Model output (`response.model_output`)
Each value is truncated at 60 KB; `*_truncated` and `*_original_length`
flags surface when truncation occurs.
2. **Log-to-span bridge spans** (used when HTTP traces are exported without a
logs endpoint) keep their existing `prompt`, `function_args`, and
`response_text` fields, instead of being dropped.
โš ๏ธ **Security warning:** enabling this flag streams full conversation history,
file contents read by `read_file`, shell commands and their output (including
secrets in env vars or arguments), and model responses to the configured OTLP
backend. Treat the backend as a privileged data sink. The flag defaults to
`false`.
**Cost / payload size:** A heavy turn (60 KB system prompt + 10 tool calls,
each up to 60 KB input + 60 KB result, plus 60 KB model output) can produce up
to ~1.5 MB of attribute payload before OTLP compression. When pointing tools
that read large files (`read_file`, etc.) at long-running sessions, monitor
exporter throughput.
This setting does not disable sensitive data in OTel logs or other telemetry
sinks; non-internal API response telemetry can populate `response_text`, so
OTel logs, UI telemetry, and chat recording may receive response text
independently of this setting. QwenLogger does not include `response_text`.
2025-10-23 09:27:04 +08:00
feat(telemetry): define HTTP OTLP endpoint behavior and signal routing (#3779) * feat(telemetry): define HTTP OTLP endpoint behavior and signal routing - Add resolveHttpOtlpUrl() that appends /v1/traces, /v1/logs, /v1/metrics to base HTTP OTLP endpoints per the OpenTelemetry specification - Add per-signal endpoint overrides (otlpTracesEndpoint, otlpLogsEndpoint, otlpMetricsEndpoint) for backends with non-standard paths (e.g. Alibaba Cloud) - Add LogToSpanProcessor that bridges OTel log records to spans for traces-only backends, with session-based traceId correlation and error status propagation - Auto-wire LogToSpanProcessor when traces URL exists but logs URL doesn't - Validate per-signal URLs gracefully (log error + skip, don't crash) - Preserve query strings when appending signal paths to URLs - Guard gRPC branch against missing base endpoint with per-signal config - Update telemetry documentation with signal routing semantics and Alibaba Cloud HTTP per-signal endpoint examples Closes #3734 Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): fix TS noPropertyAccessFromIndexSignature errors in tests Use typed ExportedSpan interface and bracket notation for index signature properties to satisfy strict TypeScript checks in CI. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): replace MD5 with SHA-256 for traceId derivation CodeQL flagged MD5 as a weak cryptographic algorithm when used with session.id (considered sensitive data). Switch to SHA-256 truncated to 32 hex chars to satisfy CodeQL while maintaining the same traceId format required by the OTel specification. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): address review feedback for LogToSpanProcessor robustness - Wrap JSON.stringify in try/catch to handle circular refs and BigInt - Add export timeout (30s) and try/catch to prevent hung shutdown - Track in-flight exports to avoid interval-vs-shutdown race condition - Fix deriveSpanStatus: use truthy checks (!!), drop success===false heuristic since declined tool calls are normal, not errors - Enforce http(s) scheme in validateUrl to reject file:/javascript: URLs - Change DiagLogLevel from ERROR to WARN to preserve operational diagnostics - Preserve logRecord.instrumentationScope instead of hardcoding - Forward severityNumber/severityText as span attributes - Add tests for circular refs, error status edge cases, severity Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): flush sdk shutdown through cleanup Remove async process exit handlers from telemetry initialization and route SDK shutdown through Config cleanup so normal CLI exit paths await pending telemetry exports. Keep shutdown idempotent while an SDK shutdown is in flight. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): harden bridged log shutdown Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): address review follow-ups Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> --------- Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
2026-05-01 22:47:01 +08:00
**HTTP OTLP signal routing:** When using HTTP protocol (`otlpProtocol: "http"`),
Qwen Code automatically appends signal-specific paths (`/v1/traces`, `/v1/logs`,
`/v1/metrics`) to the base `otlpEndpoint`. For example, `http://collector:4318`
becomes `http://collector:4318/v1/traces` for traces. If the URL already ends
with a signal path, it is used as-is. Per-signal endpoint overrides
(`otlpTracesEndpoint`, etc.) take precedence over the base endpoint and are used
verbatim. gRPC protocol uses service-based routing and does not append paths.
The per-signal endpoint environment variables also accept the standard
OpenTelemetry names: `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`,
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`.
The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
2025-10-23 09:27:04 +08:00
For detailed information about all configuration options, see the
[Configuration Guide](./cli/configuration.md).
docs(serve): v0.16-alpha known limits + SDK QWEN_SERVER_TOKEN env fallback (PR 27) (#4473) * docs(serve): v0.16-alpha known limits + SDK QWEN_SERVER_TOKEN env fallback (PR 27) First PR in the F5 release chain (PR 27 โ†’ 28 โ†’ 30a โ†’ 31) per the 2026-05-24 v0.16-alpha scope freeze in #4175 (text-only chat / coding + local-only deployment). ## SDK ergonomic micro-change (~50 LOC + 4 tests) `DaemonClient` constructor falls back to `QWEN_SERVER_TOKEN` env var when `opts.token` is absent โ€” closes the asymmetry where the daemon side already honors this var (--token CLI flag fallback, already in main since PR 15) but the SDK forced clients to thread it through every construction. Properties: - Browser-safe via `globalThis.process` indirection (the SDK is imported by @qwen-code/webui; literal process.env access would explode at module load on browser bundles) - Whitespace stripped (matches daemon-side trim โ€” handy for `export QWEN_SERVER_TOKEN=\"\$(cat token.txt)\"` where cat adds a trailing newline) - Empty / whitespace-only treated as unset (a stale `export QWEN_SERVER_TOKEN=\"\"` won't accidentally send Authorization: Bearer with no token) - Resolved at construction, not lazily per-request (later process.env mutations don't affect already-built clients) - Explicit opts.token wins over env Tests: 4 new in DaemonClient.test.ts `bearer auth` describe covering env fallback / explicit-wins / empty-treated-unset / whitespace-stripped. Plus a defensive snapshot/restore on the existing 'omits Authorization when no token' test so an inherited test-runner export of QWEN_SERVER_TOKEN doesn't turn that assertion into a false positive. This SDK fallback is the entire ergonomic replacement for PR 29's SDK env/file fallback. PR 29's other features (auto-gen daemon token, instance-path keying, stale cleanup) remain deferred to v0.16.x โ€” all are DX improvements over the boot-time security gate already shipped in PR 15. ## v0.16-alpha docs (~120 LOC markdown) - docs/users/qwen-serve.md: new "v0.16-alpha known limits" section enumerating product surface (text-only โœ…, multimodal โŒ), deployment surface (local launchers โœ…, containerized โŒ, multi- daemon โŒ, BYO-token โœ…), and hardening posture (boot security gate โœ…, mutation gate โœ…, MCP guardrails โœ…, prompt absolute deadline โธ๏ธ, rate limiting โธ๏ธ, --max-body-size โธ๏ธ). Adds an alpha banner at the top of the file. - docs/developers/examples/daemon-client-quickstart.md: documents the SDK env fallback in both the Hello-daemon intro and the Authentication section, with the "export + no-token-arg" recommended path called out for local dev. Verification: 125/125 DaemonClient.test.ts pass (121 existing + 4 new); 4/4 daemon-public-surface.test.ts pass (constructor signature unchanged); tsc clean on packages/sdk-typescript; eslint --max-warnings 0 clean on touched .ts files. Part of #4175. * fix(sdk): #4473 round 1 fold-in โ€” 2 copilot doc threads adopted T1 [copilot DaemonClient.ts:144 โ€” stale line refs in readTokenFromEnv JSDoc]: removed `runQwenServe.ts:175` (token resolution actually lives at line 302-318 today, would drift again on next refactor) and `docs/users/qwen-serve.md:173`. Replaced with stable symbol/section references ("runQwenServe token-resolution path"; "qwen-serve user guide CLI flags section"). T2 [copilot daemon-client-quickstart.md:33 โ€” `~/.qwen/server-token` implies built-in path that doesn't exist]: PR 27 explicitly defers token auto-generation + file-store fallback (PR 29 deferred features). The example incorrectly suggested a standard file location. Replaced with two explicit user-managed alternatives: - `openssl rand -hex 32` one-shot - `cat ./my-token-file` user-managed file Both threads were accurate suggestions caught at the right time (zero behavior change; pure docstring/example accuracy). Verification: 125/125 DaemonClient tests pass; tsc + eslint clean on touched files.
2026-05-24 23:57:39 +08:00
### Resource attributes
Resource attributes are static key-value pairs attached to every span, log,
and metric exported via OTLP. Use them to slice telemetry by team, environment,
deployment region, or any other dimension your backend cares about.
Two sources, merged in priority order (lowest โ†’ highest):
1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
key conflict)
`OTEL_SERVICE_NAME` is a separate escape hatch โ€” when set, it overrides
`service.name` from any other source (per the OpenTelemetry spec).
#### Examples
**Slice all telemetry by team / environment:**
```bash
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
```
**Route to a per-tenant collector via `service.name`:**
```bash
export OTEL_SERVICE_NAME=qwen-code-ci
```
**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**
```json
{
"telemetry": {
"resourceAttributes": {
"deployment.environment": "production",
"service.namespace": "engineering-tooling"
}
}
}
```
```bash
# Add a one-off tag without touching settings:
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
```
#### Reserved keys
Some keys are runtime-controlled and cannot be overridden:
- `service.version` โ€” always set to the running CLI version. Setting it from
any source is silently dropped with a warning.
- `session.id` โ€” runtime-injected per session. User-provided values from
either env or settings are dropped with a warning. The reason is that
Resource attributes auto-attach to every metric data point; allowing user
override would bypass [Cardinality controls](#cardinality-controls) below.
Spans and logs always carry `session.id`.
`service.name` is **not** reserved; it follows the precedence chain above.
#### Format
`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at
the wrong boundary and the second half is dropped as malformed). Malformed
pairs are skipped with a warning rather than failing telemetry startup.
#### Troubleshooting: when a user-provided attribute appears not to take effect
Reserved keys (`service.version`, `session.id`), malformed pairs, non-string
settings values, and invalid percent-encoding are all silently dropped with a
warning logged via the OpenTelemetry diagnostics channel. That channel routes
to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the
behavior can look like silent failure.
If a custom resource attribute isn't appearing on exported telemetry:
1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved
key dropped), `Skipping malformed` (bad env var pair), or `must be a string`
(non-string settings value).
2. Verify the env var is set in the qwen-code process's environment (not just
your shell) and that values are percent-encoded.
3. Confirm `telemetry.enabled` is `true` โ€” telemetry init only runs if enabled.
### Cardinality controls
Metrics are aggregated by attribute set at the backend โ€” every distinct
combination of attribute values produces a new time series. Attaching a
high-cardinality field like `session.id` to a metric causes time-series fan-out
proportional to the number of sessions, which quickly exhausts metric backend
storage.
To prevent this, Qwen Code keeps high-cardinality attributes off metric data
points by default. Spans and logs are per-event and unaffected, so they
continue to carry `session.id` for trace and log correlation.
#### `telemetry.metrics.includeSessionId` (default: `false`)
Setting this to `true` (via settings or
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
every metric data point.
โš ๏ธ **Warning:** each CLI session creates a new value. Leaving this on for a
fleet will blow up metric storage. Recommended only for short-term debugging.
For long-term session correlation, query trace or log backends instead.
#### Migration from earlier versions
Prior to this release, `session.id` was attached to metrics by default. If
your Prometheus queries / Grafana dashboards / alert rules reference
`session_id` on a metric, you have two options:
**Option A** โ€” restore the previous behavior for short-term debugging:
```bash
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
```
or:
```json
{
"telemetry": {
"metrics": { "includeSessionId": true }
}
}
```
**Option B (recommended)** โ€” move session-level analysis off metrics. Spans
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
cardinality pressure.
2025-12-15 09:47:03 +08:00
## Aliyun Telemetry
2025-10-23 09:27:04 +08:00
### Manual OTLP Export
2025-10-23 09:27:04 +08:00
To view Qwen Code telemetry in Alibaba Cloud Managed Service for
OpenTelemetry, configure Qwen Code to export to the OTLP endpoint
provided by ARMS.
2025-10-23 09:27:04 +08:00
Setting `"target": "gcp"` alone does not configure the export
destination. If `otlpEndpoint` is not set, Qwen Code still defaults to
`http://localhost:4317`. If `outfile` is set, it overrides
`otlpEndpoint` and telemetry is written to the file instead of being
sent to Alibaba Cloud.
1. Enable telemetry in your `.qwen/settings.json` and set the OTLP
endpoint:
feat(telemetry): define HTTP OTLP endpoint behavior and signal routing (#3779) * feat(telemetry): define HTTP OTLP endpoint behavior and signal routing - Add resolveHttpOtlpUrl() that appends /v1/traces, /v1/logs, /v1/metrics to base HTTP OTLP endpoints per the OpenTelemetry specification - Add per-signal endpoint overrides (otlpTracesEndpoint, otlpLogsEndpoint, otlpMetricsEndpoint) for backends with non-standard paths (e.g. Alibaba Cloud) - Add LogToSpanProcessor that bridges OTel log records to spans for traces-only backends, with session-based traceId correlation and error status propagation - Auto-wire LogToSpanProcessor when traces URL exists but logs URL doesn't - Validate per-signal URLs gracefully (log error + skip, don't crash) - Preserve query strings when appending signal paths to URLs - Guard gRPC branch against missing base endpoint with per-signal config - Update telemetry documentation with signal routing semantics and Alibaba Cloud HTTP per-signal endpoint examples Closes #3734 Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): fix TS noPropertyAccessFromIndexSignature errors in tests Use typed ExportedSpan interface and bracket notation for index signature properties to satisfy strict TypeScript checks in CI. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): replace MD5 with SHA-256 for traceId derivation CodeQL flagged MD5 as a weak cryptographic algorithm when used with session.id (considered sensitive data). Switch to SHA-256 truncated to 32 hex chars to satisfy CodeQL while maintaining the same traceId format required by the OTel specification. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): address review feedback for LogToSpanProcessor robustness - Wrap JSON.stringify in try/catch to handle circular refs and BigInt - Add export timeout (30s) and try/catch to prevent hung shutdown - Track in-flight exports to avoid interval-vs-shutdown race condition - Fix deriveSpanStatus: use truthy checks (!!), drop success===false heuristic since declined tool calls are normal, not errors - Enforce http(s) scheme in validateUrl to reject file:/javascript: URLs - Change DiagLogLevel from ERROR to WARN to preserve operational diagnostics - Preserve logRecord.instrumentationScope instead of hardcoding - Forward severityNumber/severityText as span attributes - Add tests for circular refs, error status edge cases, severity Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): flush sdk shutdown through cleanup Remove async process exit handlers from telemetry initialization and route SDK shutdown through Config cleanup so normal CLI exit paths await pending telemetry exports. Keep shutdown idempotent while an SDK shutdown is in flight. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): harden bridged log shutdown Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): address review follow-ups Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> --------- Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
2026-05-01 22:47:01 +08:00
**Option A: gRPC protocol** (standard OTLP endpoint):
2025-10-23 09:27:04 +08:00
```json
{
"telemetry": {
"enabled": true,
"target": "gcp",
"otlpEndpoint": "https://<your-otlp-endpoint>",
"otlpProtocol": "grpc"
2025-10-23 09:27:04 +08:00
}
}
```
feat(telemetry): define HTTP OTLP endpoint behavior and signal routing (#3779) * feat(telemetry): define HTTP OTLP endpoint behavior and signal routing - Add resolveHttpOtlpUrl() that appends /v1/traces, /v1/logs, /v1/metrics to base HTTP OTLP endpoints per the OpenTelemetry specification - Add per-signal endpoint overrides (otlpTracesEndpoint, otlpLogsEndpoint, otlpMetricsEndpoint) for backends with non-standard paths (e.g. Alibaba Cloud) - Add LogToSpanProcessor that bridges OTel log records to spans for traces-only backends, with session-based traceId correlation and error status propagation - Auto-wire LogToSpanProcessor when traces URL exists but logs URL doesn't - Validate per-signal URLs gracefully (log error + skip, don't crash) - Preserve query strings when appending signal paths to URLs - Guard gRPC branch against missing base endpoint with per-signal config - Update telemetry documentation with signal routing semantics and Alibaba Cloud HTTP per-signal endpoint examples Closes #3734 Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): fix TS noPropertyAccessFromIndexSignature errors in tests Use typed ExportedSpan interface and bracket notation for index signature properties to satisfy strict TypeScript checks in CI. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): replace MD5 with SHA-256 for traceId derivation CodeQL flagged MD5 as a weak cryptographic algorithm when used with session.id (considered sensitive data). Switch to SHA-256 truncated to 32 hex chars to satisfy CodeQL while maintaining the same traceId format required by the OTel specification. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): address review feedback for LogToSpanProcessor robustness - Wrap JSON.stringify in try/catch to handle circular refs and BigInt - Add export timeout (30s) and try/catch to prevent hung shutdown - Track in-flight exports to avoid interval-vs-shutdown race condition - Fix deriveSpanStatus: use truthy checks (!!), drop success===false heuristic since declined tool calls are normal, not errors - Enforce http(s) scheme in validateUrl to reject file:/javascript: URLs - Change DiagLogLevel from ERROR to WARN to preserve operational diagnostics - Preserve logRecord.instrumentationScope instead of hardcoding - Forward severityNumber/severityText as span attributes - Add tests for circular refs, error status edge cases, severity Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): flush sdk shutdown through cleanup Remove async process exit handlers from telemetry initialization and route SDK shutdown through Config cleanup so normal CLI exit paths await pending telemetry exports. Keep shutdown idempotent while an SDK shutdown is in flight. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): harden bridged log shutdown Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): address review follow-ups Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> --------- Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
2026-05-01 22:47:01 +08:00
**Option B: HTTP protocol with per-signal endpoints** (for backends
that use non-standard paths, e.g., `/api/otlp/traces` instead of
`/v1/traces`):
```json
{
"telemetry": {
"enabled": true,
"otlpProtocol": "http",
"otlpTracesEndpoint": "http://<host>/<token>/api/otlp/traces",
"otlpLogsEndpoint": "http://<host>/<token>/api/otlp/logs",
"otlpMetricsEndpoint": "http://<host>/<token>/api/otlp/metrics"
}
}
```
> **Note:** When using HTTP protocol with only `otlpEndpoint` (no
> per-signal overrides), Qwen Code appends standard OTLP paths
> (`/v1/traces`, `/v1/logs`, `/v1/metrics`) to the base URL. If your
> backend uses different paths, use per-signal endpoint overrides as
> shown in Option B.
2. If your Alibaba Cloud endpoint requires authentication, provide OTLP
headers through standard OpenTelemetry environment variables such as
`OTEL_EXPORTER_OTLP_HEADERS` (or the signal-specific variants). Qwen
Code does not currently expose OTLP auth headers directly in
`.qwen/settings.json`.
3. Run Qwen Code and send prompts.
4. View telemetry in Managed Service for OpenTelemetry:
- Product overview:
[What is Managed Service for OpenTelemetry?][aliyun-opentelemetry-overview]
- Getting started:
[Get started with Managed Service for OpenTelemetry][aliyun-opentelemetry-get-started]
- Console entry points:
- China mainland:
[trace.console.aliyun.com][aliyun-opentelemetry-console-cn]
(legacy console:
[tracing.console.aliyun.com][aliyun-opentelemetry-console-cn-legacy])
- International:
[arms.console.alibabacloud.com][aliyun-opentelemetry-console-intl]
- In the console, use `Applications` to inspect traces and service
topology.
- To locate the OTLP endpoint and access information:
- **New console** (`trace.console.aliyun.com` or international):
navigate to `Integration Center`.
- **Legacy console** (`tracing.console.aliyun.com`): navigate to
`Cluster Configurations` โ†’ `Access point information`.
2025-10-23 09:27:04 +08:00
## Local Telemetry
For local development and debugging, you can capture telemetry data locally:
### File-based Output (Recommended)
1. Enable telemetry in your `.qwen/settings.json`:
docs(telemetry): align config and docs semantics for target, outfile, and CLI flags (#4066) * docs(telemetry): align config and docs semantics for target, outfile, and CLI flags - Remove stale warning note "This feature requires corresponding code changes" โ€” the OTLP implementation is now complete (#3779, #4061) - Clarify that `target` is an informational destination label and does not control exporter routing; `otlpEndpoint` or `outfile` must be set to configure where data is sent - Mark `--telemetry-target` CLI flag as deprecated in the configuration table to match the deprecateOption() call in cli/src/config/config.ts - Fix `outfile` / `QWEN_TELEMETRY_OUTFILE` descriptions: remove the incorrect "when target is local" qualifier โ€” outfile overrides OTLP export regardless of the target value - Simplify the file-based output example by removing the now-redundant `"target": "local"` and `"otlpEndpoint": ""` fields Closes the "Align telemetry config and docs semantics for target, useCollector, otlpEndpoint, otlpProtocol, and outfile" checklist item in #3731. ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): address Copilot review comments on outfile and target descriptions - Fix outfile table row in telemetry.md: "overrides `otlpEndpoint`" โ†’ "overrides OTLP export" (outfile disables all OTLP exporting, not just the base endpoint) - Use fully-qualified setting names (`telemetry.otlpEndpoint`, `telemetry.outfile`) in the target description in settings.md for consistency with the rest of the table ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): update QWEN_TELEMETRY_TARGET env var description and add outfile note - Align QWEN_TELEMETRY_TARGET env var description with the updated telemetry.target setting semantics (informational label, not routing) - Add a note after the file-based output example clarifying that outfile automatically disables OTLP export ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)
2026-05-13 08:27:41 +08:00
2025-10-23 09:27:04 +08:00
```json
{
"telemetry": {
"enabled": true,
"outfile": ".qwen/telemetry.log"
}
}
```
docs(telemetry): align config and docs semantics for target, outfile, and CLI flags (#4066) * docs(telemetry): align config and docs semantics for target, outfile, and CLI flags - Remove stale warning note "This feature requires corresponding code changes" โ€” the OTLP implementation is now complete (#3779, #4061) - Clarify that `target` is an informational destination label and does not control exporter routing; `otlpEndpoint` or `outfile` must be set to configure where data is sent - Mark `--telemetry-target` CLI flag as deprecated in the configuration table to match the deprecateOption() call in cli/src/config/config.ts - Fix `outfile` / `QWEN_TELEMETRY_OUTFILE` descriptions: remove the incorrect "when target is local" qualifier โ€” outfile overrides OTLP export regardless of the target value - Simplify the file-based output example by removing the now-redundant `"target": "local"` and `"otlpEndpoint": ""` fields Closes the "Align telemetry config and docs semantics for target, useCollector, otlpEndpoint, otlpProtocol, and outfile" checklist item in #3731. ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): address Copilot review comments on outfile and target descriptions - Fix outfile table row in telemetry.md: "overrides `otlpEndpoint`" โ†’ "overrides OTLP export" (outfile disables all OTLP exporting, not just the base endpoint) - Use fully-qualified setting names (`telemetry.otlpEndpoint`, `telemetry.outfile`) in the target description in settings.md for consistency with the rest of the table ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): update QWEN_TELEMETRY_TARGET env var description and add outfile note - Align QWEN_TELEMETRY_TARGET env var description with the updated telemetry.target setting semantics (informational label, not routing) - Add a note after the file-based output example clarifying that outfile automatically disables OTLP export ๐Ÿค– Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)
2026-05-13 08:27:41 +08:00
> **Note:** When `outfile` is set, OTLP export is automatically disabled.
> The `target` and `otlpEndpoint` settings are not needed for file-only
> output and can be safely omitted from your config.
2025-10-23 09:27:04 +08:00
2. Run Qwen Code and send prompts.
3. View logs and metrics in the specified file (e.g., `.qwen/telemetry.log`).
### Collector-Based Export (Advanced)
1. Run the automation script:
```bash
npm run telemetry -- --target=local
```
This will:
- Download and start Jaeger and OTEL collector
- Configure your workspace for local telemetry
- Provide a Jaeger UI at http://localhost:16686
- Save logs/metrics to `~/.qwen/tmp/<projectHash>/otel/collector.log`
- Stop collector on exit (e.g. `Ctrl+C`)
2. Run Qwen Code and send prompts.
3. View traces at http://localhost:16686 and logs/metrics in the collector log
file.
## Logs and Metrics
The following section describes the structure of logs and metrics generated for
Qwen Code.
- A `sessionId` is included as a common attribute on all logs and metrics.
### Logs
Logs are timestamped records of specific events. The following events are logged for Qwen Code:
- `qwen-code.config`: This event occurs once at startup with the CLI's configuration.
- **Attributes**:
- `model` (string)
- `sandbox_enabled` (boolean)
- `core_tools_enabled` (string)
- `approval_mode` (string)
- `file_filtering_respect_git_ignore` (boolean)
- `debug_mode` (boolean)
- `truncate_tool_output_threshold` (number)
- `truncate_tool_output_lines` (number)
- `hooks` (string, comma-separated hook event types, omitted if hooks disabled)
- `ide_enabled` (boolean)
- `interactive_shell_enabled` (boolean)
- `mcp_servers` (string)
2025-10-23 09:27:04 +08:00
- `output_format` (string: "text" or "json")
- `qwen-code.user_prompt`: This event occurs when a user submits a prompt.
- **Attributes**:
- `prompt_length` (int)
- `prompt_id` (string)
2025-10-23 09:27:04 +08:00
- `prompt` (string, this attribute is excluded if `log_prompts_enabled` is
configured to be `false`)
- `auth_type` (string)
- `qwen-code.tool_call`: This event occurs for each function call.
- **Attributes**:
- `function_name`
- `function_args`
- `duration_ms`
- `success` (boolean)
2025-10-23 09:27:04 +08:00
- `decision` (string: "accept", "reject", "auto_accept", or "modify", if
applicable)
- `error` (if applicable)
- `error_type` (if applicable)
2025-10-23 09:27:04 +08:00
- `content_length` (int, if applicable)
- `metadata` (if applicable, dictionary of string -> any)
2025-10-23 09:27:04 +08:00
- `qwen-code.file_operation`: This event occurs for each file operation.
- **Attributes**:
- `tool_name` (string)
- `operation` (string: "create", "read", "update")
- `lines` (int, if applicable)
- `mimetype` (string, if applicable)
- `extension` (string, if applicable)
- `programming_language` (string, if applicable)
- `diff_stat` (json string, if applicable): A JSON string with the following members:
- `ai_added_lines` (int)
- `ai_removed_lines` (int)
- `user_added_lines` (int)
- `user_removed_lines` (int)
2025-09-12 15:27:42 +08:00
- `qwen-code.api_request`: This event occurs when making a request to Qwen API.
- **Attributes**:
- `model`
- `request_text` (if applicable)
- `qwen-code.api_error`: This event occurs if the API request fails.
- **Attributes**:
- `model`
- `error`
- `error_type`
- `status_code`
- `duration_ms`
- `auth_type`
2025-09-12 15:27:42 +08:00
- `qwen-code.api_response`: This event occurs upon receiving a response from Qwen API.
- **Attributes**:
- `model`
- `status_code`
- `duration_ms`
- `error` (optional)
- `input_token_count`
- `output_token_count`
- `cached_content_token_count`
- `thoughts_token_count`
- `response_text` (if applicable)
- `auth_type`
2025-10-23 09:27:04 +08:00
- `qwen-code.tool_output_truncated`: This event occurs when the output of a tool call is too large and gets truncated.
- **Attributes**:
- `tool_name` (string)
- `original_content_length` (int)
- `truncated_content_length` (int)
- `threshold` (int)
- `lines` (int)
- `prompt_id` (string)
- `qwen-code.malformed_json_response`: This event occurs when a `generateJson` response from Qwen API cannot be parsed as a json.
- **Attributes**:
- `model`
- `qwen-code.flash_fallback`: This event occurs when Qwen Code switches to flash as fallback.
- **Attributes**:
- `auth_type`
- `qwen-code.slash_command`: This event occurs when a user executes a slash command.
- **Attributes**:
- `command` (string)
- `subcommand` (string, if applicable)
2025-10-23 09:27:04 +08:00
- `qwen-code.extension_enable`: This event occurs when an extension is enabled
- `qwen-code.extension_install`: This event occurs when an extension is installed
- **Attributes**:
- `extension_name` (string)
- `extension_version` (string)
- `extension_source` (string)
- `status` (string)
- `qwen-code.extension_uninstall`: This event occurs when an extension is uninstalled
### Metrics
Metrics are numerical measurements of behavior over time. The following metrics are collected for Qwen Code (metric names remain `qwen-code.*` for compatibility):
- `qwen-code.session.count` (Counter, Int): Incremented once per CLI startup.
- `qwen-code.tool.call.count` (Counter, Int): Counts tool calls.
- **Attributes**:
- `function_name`
- `success` (boolean)
- `decision` (string: "accept", "reject", or "modify", if applicable)
- `tool_type` (string: "mcp", or "native", if applicable)
- `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency.
- **Attributes**:
- `function_name`
- `decision` (string: "accept", "reject", or "modify", if applicable)
- `qwen-code.api.request.count` (Counter, Int): Counts all API requests.
- **Attributes**:
- `model`
- `status_code`
- `error_type` (if applicable)
- `qwen-code.api.request.latency` (Histogram, ms): Measures API request latency.
- **Attributes**:
- `model`
- `qwen-code.token.usage` (Counter, Int): Counts the number of tokens used.
- **Attributes**:
- `model`
- `type` (string: "input", "output", "thought", or "cache")
- `qwen-code.file.operation.count` (Counter, Int): Counts file operations.
- **Attributes**:
- `operation` (string: "create", "read", "update"): The type of file operation.
- `lines` (Int, if applicable): Number of lines in the file.
- `mimetype` (string, if applicable): Mimetype of the file.
- `extension` (string, if applicable): File extension of the file.
2025-10-23 09:27:04 +08:00
- `model_added_lines` (Int, if applicable): Number of lines added/changed by the model.
- `model_removed_lines` (Int, if applicable): Number of lines removed/changed by the model.
- `user_added_lines` (Int, if applicable): Number of lines added/changed by user in AI proposed changes.
- `user_removed_lines` (Int, if applicable): Number of lines removed/changed by user in AI proposed changes.
- `programming_language` (string, if applicable): The programming language of the file.
2025-09-12 16:01:38 +08:00
- `qwen-code.chat_compression` (Counter, Int): Counts chat compression operations
- **Attributes**:
- `tokens_before`: (Int): Number of tokens in context prior to compression
- `tokens_after`: (Int): Number of tokens in context after compression