* feat(telemetry): support custom resource attributes and add metric cardinality controls Resolves #4365. Adds two coupled OpenTelemetry capabilities to make qwen-code's telemetry production-ready in multi-team / multi-tenant deployments: 1. Custom resource attributes via standard `OTEL_RESOURCE_ATTRIBUTES` and `OTEL_SERVICE_NAME` env vars and a new `telemetry.resourceAttributes` setting. Operators can now tag every span / log / metric with `team`, `env`, `cost_center`, or anything else their backend needs. 2. Metric cardinality controls. `session.id` is moved off the OpenTelemetry Resource (where it auto-attached to every metric data point and caused unbounded time-series fan-out on Prometheus / ARMS Metric / etc.) and gated behind a new opt-in `telemetry.metrics.includeSessionId` toggle. Spans and logs still carry `session.id` for trace and log correlation. Reserved keys (`service.version`, `session.id`) are stripped from both env and settings sources with a `diag.warn`. `OTEL_SERVICE_NAME` follows the OTel spec precedence (highest priority for `service.name`). Settings JSON values are runtime-coerced to strings as defense against hand-edited non-conforming JSON. Breaking change: metrics no longer carry `session.id` by default. Operators who need it can restore the previous behavior with `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true` or `telemetry.metrics.includeSessionId: true` in settings.json; recommended only for short-term debugging since it re-introduces the cardinality problem. For long-term session-level analysis, prefer trace and log backends which handle per-event data without cardinality pressure. Design doc: docs/design/telemetry-resource-attributes-design.md ๐ค Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): align reserved-key descriptions with implementation Round 1 review fixes (#4367). After session.id was added to RESERVED_RESOURCE_ATTRIBUTE_KEYS in Codex review, four user-facing descriptions still claimed only service.version was reserved: - packages/core/src/telemetry/config.ts (merge comment) - packages/core/src/config/config.ts (TelemetrySettings JSDoc) - packages/cli/src/config/settingsSchema.ts (schema description) - packages/vscode-ide-companion/schemas/settings.schema.json (regenerated) Also corrects scope claim: resource attributes apply to every signal the SDK exports (OTLP and file outfile share the same Resource), not just OTLP. ๐ค Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * docs(telemetry): clarify warning destination and surface percent-encoding hint Round 2 self-review fixes (#4367). Two small but real UX gaps: 1. Reserved-key / malformed-pair / coerce warnings route to the debug log (per #3986), not the console โ so a user who types `OTEL_RESOURCE_ATTRIBUTES=service.version=2.0` sees no feedback that the value was silently dropped. Adds a "Troubleshooting" section in telemetry.md telling users where to look, and a note in the parser docstring documenting where warns go. 2. A literal (unencoded) comma in an env var value is a common foot-gun: the parser splits on it, producing a malformed second half that is silently dropped. Updates the warn text to include a "hint: percent-encode literal commas as %2C" callout, and adds the same guidance to the docs. Deferred to a follow-up: startup-time stderr summary of dropped attributes. Stderr during TUI render could break Ink rendering, so the right surface needs separate design. ๐ค Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * test(telemetry): cover first-`=` split contract in OTEL_RESOURCE_ATTRIBUTES parser Per review feedback on #4367. The parser uses `indexOf('=')` so the first `=` separates key and value while subsequent `=` stay in the value. The behavior was correct but untested; a future refactor to `split('=')` would silently break base64-padded, JWT, or connection-string values. ๐ค Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * feat(telemetry): tighten resource-attribute input validation + startup summary Adopts review feedback from #4367 (wenshao via Qwen Code /review). Five accepted suggestions, bundled because they all touch the same parse/coerce/strip pipeline: 1. Key percent-decoding (CRITICAL). `parseOtelResourceAttributes` now percent-decodes both keys and values per the OTel / W3C Baggage spec. Without this, `OTEL_RESOURCE_ATTRIBUTES=service%2Eversion=99` lands on Resource as the literal key `service%2Eversion`, bypassing the reserved-key filter; a collector that decodes keys downstream could then resurrect `service.version` and spoof the version label. 2. Startup summary of dropped attributes. Every `diag.warn` in resource-attributes.ts routes only to the OTel debug log (per #3986), giving operators zero feedback when their attributes are silently dropped. Helpers now optionally accumulate diagnostics into a `ResourceAttributeWarnings` array; the resolver collects them and the SDK emits a one-time console summary at init (before Ink renders, so no TUI conflict). 3. `||` instead of `??` for service.name fallback. Settings can put an empty string through `??`, producing a blank `service.name` that some backends reject. `||` falls through to the default. 4. `coerceStringResourceAttributes` now trims keys and skips empty/whitespace-only keys, matching `parseOtelResourceAttributes`. Previously `{" ": "x"}` or `{"team ": "y"}` from settings.json would land as malformed Resource attributes. 5. `OTEL_SERVICE_NAME` is trimmed before the truthy check, so values like `' '` or `'\t'` are treated as unset rather than producing a whitespace-only service name on Resource. One suggestion declined (in-thread reply on PR): - "Redundant `?? {}` in sdk.ts:160" โ intentional defense-in-depth for `vi.mock('../config/config.js')` callers in `telemetry.test.ts` where auto-stub returns undefined. The reviewer is right that production code paths never hit it, but tests do. ๐ค Generated with [Qwen Code](https://github.com/QwenLM/qwen-code) * fix(telemetry): trim whitespace-only service.name + add invalid-key-encoding test Adopts two review suggestions on #4367 (wenshao via Qwen Code /review): 1. `service.name` fallback uses `.trim() || SERVICE_NAME` instead of plain `||`. Plain `||` lets whitespace-only values (`" "`, `"\t"`) through as truthy, producing a blank service name on Resource that some backends reject. Both settings (no value trimming) and env (`%20` decodes to `" "`) can deliver such values. Test added. 2. Adds `key%ZZ=val` to the parameterized parser test to cover the invalid-percent-encoding-on-key catch branch. Previously only the value-side catch was tested. ๐ค Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)
552 lines
25 KiB
Markdown
552 lines
25 KiB
Markdown
# Observability with OpenTelemetry
|
|
|
|
Learn how to enable and setup OpenTelemetry for Qwen Code.
|
|
|
|
- [Observability with OpenTelemetry](#observability-with-opentelemetry)
|
|
- [Key Benefits](#key-benefits)
|
|
- [OpenTelemetry Integration](#opentelemetry-integration)
|
|
- [Configuration](#configuration)
|
|
- [Aliyun Telemetry](#aliyun-telemetry)
|
|
- [Manual OTLP Export](#manual-otlp-export)
|
|
- [Local Telemetry](#local-telemetry)
|
|
- [File-based Output (Recommended)](#file-based-output-recommended)
|
|
- [Collector-Based Export (Advanced)](#collector-based-export-advanced)
|
|
- [Logs and Metrics](#logs-and-metrics)
|
|
- [Logs](#logs)
|
|
- [Metrics](#metrics)
|
|
|
|
## Key Benefits
|
|
|
|
- **๐ Usage Analytics**: Understand interaction patterns and feature adoption
|
|
across your team
|
|
- **โก Performance Monitoring**: Track response times, token consumption, and
|
|
resource utilization
|
|
- **๐ Real-time Debugging**: Identify bottlenecks, failures, and error patterns
|
|
as they occur
|
|
- **๐ Workflow Optimization**: Make informed decisions to improve
|
|
configurations and processes
|
|
- **๐ข Enterprise Governance**: Monitor usage across teams, track costs, ensure
|
|
compliance, and integrate with existing monitoring infrastructure
|
|
|
|
## OpenTelemetry Integration
|
|
|
|
Built on **[OpenTelemetry]** โ the vendor-neutral, industry-standard
|
|
observability framework โ Qwen Code's observability system provides:
|
|
|
|
- **Universal Compatibility**: Export to any OpenTelemetry backend (Aliyun,
|
|
Jaeger, Prometheus, Datadog, etc.)
|
|
- **Standardized Data**: Use consistent formats and collection methods across
|
|
your toolchain
|
|
- **Future-Proof Integration**: Connect with existing and future observability
|
|
infrastructure
|
|
- **No Vendor Lock-in**: Switch between backends without changing your
|
|
instrumentation
|
|
|
|
[OpenTelemetry]: https://opentelemetry.io/
|
|
[aliyun-opentelemetry-overview]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/product-overview/what-is-tracing-analysis
|
|
[aliyun-opentelemetry-get-started]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/before-you-begin
|
|
[aliyun-opentelemetry-console-cn]: https://trace.console.aliyun.com
|
|
[aliyun-opentelemetry-console-cn-legacy]: https://tracing.console.aliyun.com
|
|
[aliyun-opentelemetry-console-intl]: https://arms.console.alibabacloud.com
|
|
|
|
## Configuration
|
|
|
|
All telemetry behavior is controlled through your `.qwen/settings.json` file.
|
|
These settings can be overridden by environment variables or CLI flags.
|
|
|
|
| Setting | Environment Variable | CLI Flag | Description | Values | Default |
|
|
| -------------------------------- | -------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ----------------------- |
|
|
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
|
|
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing โ set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
|
|
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
|
|
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
|
|
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
|
|
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
|
|
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
|
|
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
|
|
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
|
|
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` |
|
|
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,โฆ` | `{}` |
|
|
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |
|
|
|
|
**Note on boolean environment variables:** For the boolean settings (`enabled`,
|
|
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
|
|
corresponding environment variable to `true` or `1` will enable the feature. Any
|
|
other value will disable it.
|
|
|
|
**Sensitive span attributes:** When `includeSensitiveSpanAttributes` is enabled,
|
|
two things happen:
|
|
|
|
1. **Native span attributes (`qwen-code.interaction`, `api.generateContent*`,
|
|
`tool.<name>`)** carry verbatim conversation content:
|
|
- User prompts (`new_context`)
|
|
- System prompts (`system_prompt` โ full text once per session, deduped by
|
|
SHA-256 hash; subsequent spans only carry `system_prompt_hash` +
|
|
`system_prompt_preview` + `system_prompt_length`)
|
|
- Tool schemas (emitted as `tool_schema` events, also hash-deduped)
|
|
- Tool inputs (`tool_input`) and tool results (`tool_result`)
|
|
- Model output (`response.model_output`)
|
|
|
|
Each value is truncated at 60 KB; `*_truncated` and `*_original_length`
|
|
flags surface when truncation occurs.
|
|
|
|
2. **Log-to-span bridge spans** (used when HTTP traces are exported without a
|
|
logs endpoint) keep their existing `prompt`, `function_args`, and
|
|
`response_text` fields, instead of being dropped.
|
|
|
|
โ ๏ธ **Security warning:** enabling this flag streams full conversation history,
|
|
file contents read by `read_file`, shell commands and their output (including
|
|
secrets in env vars or arguments), and model responses to the configured OTLP
|
|
backend. Treat the backend as a privileged data sink. The flag defaults to
|
|
`false`.
|
|
|
|
**Cost / payload size:** A heavy turn (60 KB system prompt + 10 tool calls,
|
|
each up to 60 KB input + 60 KB result, plus 60 KB model output) can produce up
|
|
to ~1.5 MB of attribute payload before OTLP compression. When pointing tools
|
|
that read large files (`read_file`, etc.) at long-running sessions, monitor
|
|
exporter throughput.
|
|
|
|
This setting does not disable sensitive data in OTel logs or other telemetry
|
|
sinks; non-internal API response telemetry can populate `response_text`, so
|
|
OTel logs, UI telemetry, and chat recording may receive response text
|
|
independently of this setting. QwenLogger does not include `response_text`.
|
|
|
|
**HTTP OTLP signal routing:** When using HTTP protocol (`otlpProtocol: "http"`),
|
|
Qwen Code automatically appends signal-specific paths (`/v1/traces`, `/v1/logs`,
|
|
`/v1/metrics`) to the base `otlpEndpoint`. For example, `http://collector:4318`
|
|
becomes `http://collector:4318/v1/traces` for traces. If the URL already ends
|
|
with a signal path, it is used as-is. Per-signal endpoint overrides
|
|
(`otlpTracesEndpoint`, etc.) take precedence over the base endpoint and are used
|
|
verbatim. gRPC protocol uses service-based routing and does not append paths.
|
|
|
|
The per-signal endpoint environment variables also accept the standard
|
|
OpenTelemetry names: `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`,
|
|
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`.
|
|
The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
|
|
|
|
For detailed information about all configuration options, see the
|
|
[Configuration Guide](./cli/configuration.md).
|
|
|
|
### Resource attributes
|
|
|
|
Resource attributes are static key-value pairs attached to every span, log,
|
|
and metric exported via OTLP. Use them to slice telemetry by team, environment,
|
|
deployment region, or any other dimension your backend cares about.
|
|
|
|
Two sources, merged in priority order (lowest โ highest):
|
|
|
|
1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
|
|
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
|
|
key conflict)
|
|
|
|
`OTEL_SERVICE_NAME` is a separate escape hatch โ when set, it overrides
|
|
`service.name` from any other source (per the OpenTelemetry spec).
|
|
|
|
#### Examples
|
|
|
|
**Slice all telemetry by team / environment:**
|
|
|
|
```bash
|
|
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
|
|
```
|
|
|
|
**Route to a per-tenant collector via `service.name`:**
|
|
|
|
```bash
|
|
export OTEL_SERVICE_NAME=qwen-code-ci
|
|
```
|
|
|
|
**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"resourceAttributes": {
|
|
"deployment.environment": "production",
|
|
"service.namespace": "engineering-tooling"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
```bash
|
|
# Add a one-off tag without touching settings:
|
|
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
|
|
```
|
|
|
|
#### Reserved keys
|
|
|
|
Some keys are runtime-controlled and cannot be overridden:
|
|
|
|
- `service.version` โ always set to the running CLI version. Setting it from
|
|
any source is silently dropped with a warning.
|
|
- `session.id` โ runtime-injected per session. User-provided values from
|
|
either env or settings are dropped with a warning. The reason is that
|
|
Resource attributes auto-attach to every metric data point; allowing user
|
|
override would bypass [Cardinality controls](#cardinality-controls) below.
|
|
Spans and logs always carry `session.id`.
|
|
|
|
`service.name` is **not** reserved; it follows the precedence chain above.
|
|
|
|
#### Format
|
|
|
|
`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
|
|
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
|
|
be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at
|
|
the wrong boundary and the second half is dropped as malformed). Malformed
|
|
pairs are skipped with a warning rather than failing telemetry startup.
|
|
|
|
#### Troubleshooting: when a user-provided attribute appears not to take effect
|
|
|
|
Reserved keys (`service.version`, `session.id`), malformed pairs, non-string
|
|
settings values, and invalid percent-encoding are all silently dropped with a
|
|
warning logged via the OpenTelemetry diagnostics channel. That channel routes
|
|
to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the
|
|
behavior can look like silent failure.
|
|
|
|
If a custom resource attribute isn't appearing on exported telemetry:
|
|
|
|
1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved
|
|
key dropped), `Skipping malformed` (bad env var pair), or `must be a string`
|
|
(non-string settings value).
|
|
2. Verify the env var is set in the qwen-code process's environment (not just
|
|
your shell) and that values are percent-encoded.
|
|
3. Confirm `telemetry.enabled` is `true` โ telemetry init only runs if enabled.
|
|
|
|
### Cardinality controls
|
|
|
|
Metrics are aggregated by attribute set at the backend โ every distinct
|
|
combination of attribute values produces a new time series. Attaching a
|
|
high-cardinality field like `session.id` to a metric causes time-series fan-out
|
|
proportional to the number of sessions, which quickly exhausts metric backend
|
|
storage.
|
|
|
|
To prevent this, Qwen Code keeps high-cardinality attributes off metric data
|
|
points by default. Spans and logs are per-event and unaffected, so they
|
|
continue to carry `session.id` for trace and log correlation.
|
|
|
|
#### `telemetry.metrics.includeSessionId` (default: `false`)
|
|
|
|
Setting this to `true` (via settings or
|
|
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
|
|
every metric data point.
|
|
|
|
โ ๏ธ **Warning:** each CLI session creates a new value. Leaving this on for a
|
|
fleet will blow up metric storage. Recommended only for short-term debugging.
|
|
For long-term session correlation, query trace or log backends instead.
|
|
|
|
#### Migration from earlier versions
|
|
|
|
Prior to this release, `session.id` was attached to metrics by default. If
|
|
your Prometheus queries / Grafana dashboards / alert rules reference
|
|
`session_id` on a metric, you have two options:
|
|
|
|
**Option A** โ restore the previous behavior for short-term debugging:
|
|
|
|
```bash
|
|
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
|
|
```
|
|
|
|
or:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"metrics": { "includeSessionId": true }
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option B (recommended)** โ move session-level analysis off metrics. Spans
|
|
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
|
|
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
|
|
cardinality pressure.
|
|
|
|
## Aliyun Telemetry
|
|
|
|
### Manual OTLP Export
|
|
|
|
To view Qwen Code telemetry in Alibaba Cloud Managed Service for
|
|
OpenTelemetry, configure Qwen Code to export to the OTLP endpoint
|
|
provided by ARMS.
|
|
|
|
Setting `"target": "gcp"` alone does not configure the export
|
|
destination. If `otlpEndpoint` is not set, Qwen Code still defaults to
|
|
`http://localhost:4317`. If `outfile` is set, it overrides
|
|
`otlpEndpoint` and telemetry is written to the file instead of being
|
|
sent to Alibaba Cloud.
|
|
|
|
1. Enable telemetry in your `.qwen/settings.json` and set the OTLP
|
|
endpoint:
|
|
|
|
**Option A: gRPC protocol** (standard OTLP endpoint):
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"target": "gcp",
|
|
"otlpEndpoint": "https://<your-otlp-endpoint>",
|
|
"otlpProtocol": "grpc"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option B: HTTP protocol with per-signal endpoints** (for backends
|
|
that use non-standard paths, e.g., `/api/otlp/traces` instead of
|
|
`/v1/traces`):
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"otlpProtocol": "http",
|
|
"otlpTracesEndpoint": "http://<host>/<token>/api/otlp/traces",
|
|
"otlpLogsEndpoint": "http://<host>/<token>/api/otlp/logs",
|
|
"otlpMetricsEndpoint": "http://<host>/<token>/api/otlp/metrics"
|
|
}
|
|
}
|
|
```
|
|
|
|
> **Note:** When using HTTP protocol with only `otlpEndpoint` (no
|
|
> per-signal overrides), Qwen Code appends standard OTLP paths
|
|
> (`/v1/traces`, `/v1/logs`, `/v1/metrics`) to the base URL. If your
|
|
> backend uses different paths, use per-signal endpoint overrides as
|
|
> shown in Option B.
|
|
|
|
2. If your Alibaba Cloud endpoint requires authentication, provide OTLP
|
|
headers through standard OpenTelemetry environment variables such as
|
|
`OTEL_EXPORTER_OTLP_HEADERS` (or the signal-specific variants). Qwen
|
|
Code does not currently expose OTLP auth headers directly in
|
|
`.qwen/settings.json`.
|
|
3. Run Qwen Code and send prompts.
|
|
4. View telemetry in Managed Service for OpenTelemetry:
|
|
- Product overview:
|
|
[What is Managed Service for OpenTelemetry?][aliyun-opentelemetry-overview]
|
|
- Getting started:
|
|
[Get started with Managed Service for OpenTelemetry][aliyun-opentelemetry-get-started]
|
|
- Console entry points:
|
|
- China mainland:
|
|
[trace.console.aliyun.com][aliyun-opentelemetry-console-cn]
|
|
(legacy console:
|
|
[tracing.console.aliyun.com][aliyun-opentelemetry-console-cn-legacy])
|
|
- International:
|
|
[arms.console.alibabacloud.com][aliyun-opentelemetry-console-intl]
|
|
- In the console, use `Applications` to inspect traces and service
|
|
topology.
|
|
- To locate the OTLP endpoint and access information:
|
|
- **New console** (`trace.console.aliyun.com` or international):
|
|
navigate to `Integration Center`.
|
|
- **Legacy console** (`tracing.console.aliyun.com`): navigate to
|
|
`Cluster Configurations` โ `Access point information`.
|
|
|
|
## Local Telemetry
|
|
|
|
For local development and debugging, you can capture telemetry data locally:
|
|
|
|
### File-based Output (Recommended)
|
|
|
|
1. Enable telemetry in your `.qwen/settings.json`:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"outfile": ".qwen/telemetry.log"
|
|
}
|
|
}
|
|
```
|
|
|
|
> **Note:** When `outfile` is set, OTLP export is automatically disabled.
|
|
> The `target` and `otlpEndpoint` settings are not needed for file-only
|
|
> output and can be safely omitted from your config.
|
|
|
|
2. Run Qwen Code and send prompts.
|
|
3. View logs and metrics in the specified file (e.g., `.qwen/telemetry.log`).
|
|
|
|
### Collector-Based Export (Advanced)
|
|
|
|
1. Run the automation script:
|
|
```bash
|
|
npm run telemetry -- --target=local
|
|
```
|
|
This will:
|
|
- Download and start Jaeger and OTEL collector
|
|
- Configure your workspace for local telemetry
|
|
- Provide a Jaeger UI at http://localhost:16686
|
|
- Save logs/metrics to `~/.qwen/tmp/<projectHash>/otel/collector.log`
|
|
- Stop collector on exit (e.g. `Ctrl+C`)
|
|
2. Run Qwen Code and send prompts.
|
|
3. View traces at http://localhost:16686 and logs/metrics in the collector log
|
|
file.
|
|
|
|
## Logs and Metrics
|
|
|
|
The following section describes the structure of logs and metrics generated for
|
|
Qwen Code.
|
|
|
|
- A `sessionId` is included as a common attribute on all logs and metrics.
|
|
|
|
### Logs
|
|
|
|
Logs are timestamped records of specific events. The following events are logged for Qwen Code:
|
|
|
|
- `qwen-code.config`: This event occurs once at startup with the CLI's configuration.
|
|
- **Attributes**:
|
|
- `model` (string)
|
|
- `sandbox_enabled` (boolean)
|
|
- `core_tools_enabled` (string)
|
|
- `approval_mode` (string)
|
|
- `file_filtering_respect_git_ignore` (boolean)
|
|
- `debug_mode` (boolean)
|
|
- `truncate_tool_output_threshold` (number)
|
|
- `truncate_tool_output_lines` (number)
|
|
- `hooks` (string, comma-separated hook event types, omitted if hooks disabled)
|
|
- `ide_enabled` (boolean)
|
|
- `interactive_shell_enabled` (boolean)
|
|
- `mcp_servers` (string)
|
|
- `output_format` (string: "text" or "json")
|
|
|
|
- `qwen-code.user_prompt`: This event occurs when a user submits a prompt.
|
|
- **Attributes**:
|
|
- `prompt_length` (int)
|
|
- `prompt_id` (string)
|
|
- `prompt` (string, this attribute is excluded if `log_prompts_enabled` is
|
|
configured to be `false`)
|
|
- `auth_type` (string)
|
|
|
|
- `qwen-code.tool_call`: This event occurs for each function call.
|
|
- **Attributes**:
|
|
- `function_name`
|
|
- `function_args`
|
|
- `duration_ms`
|
|
- `success` (boolean)
|
|
- `decision` (string: "accept", "reject", "auto_accept", or "modify", if
|
|
applicable)
|
|
- `error` (if applicable)
|
|
- `error_type` (if applicable)
|
|
- `content_length` (int, if applicable)
|
|
- `metadata` (if applicable, dictionary of string -> any)
|
|
|
|
- `qwen-code.file_operation`: This event occurs for each file operation.
|
|
- **Attributes**:
|
|
- `tool_name` (string)
|
|
- `operation` (string: "create", "read", "update")
|
|
- `lines` (int, if applicable)
|
|
- `mimetype` (string, if applicable)
|
|
- `extension` (string, if applicable)
|
|
- `programming_language` (string, if applicable)
|
|
- `diff_stat` (json string, if applicable): A JSON string with the following members:
|
|
- `ai_added_lines` (int)
|
|
- `ai_removed_lines` (int)
|
|
- `user_added_lines` (int)
|
|
- `user_removed_lines` (int)
|
|
|
|
- `qwen-code.api_request`: This event occurs when making a request to Qwen API.
|
|
- **Attributes**:
|
|
- `model`
|
|
- `request_text` (if applicable)
|
|
|
|
- `qwen-code.api_error`: This event occurs if the API request fails.
|
|
- **Attributes**:
|
|
- `model`
|
|
- `error`
|
|
- `error_type`
|
|
- `status_code`
|
|
- `duration_ms`
|
|
- `auth_type`
|
|
|
|
- `qwen-code.api_response`: This event occurs upon receiving a response from Qwen API.
|
|
- **Attributes**:
|
|
- `model`
|
|
- `status_code`
|
|
- `duration_ms`
|
|
- `error` (optional)
|
|
- `input_token_count`
|
|
- `output_token_count`
|
|
- `cached_content_token_count`
|
|
- `thoughts_token_count`
|
|
- `response_text` (if applicable)
|
|
- `auth_type`
|
|
|
|
- `qwen-code.tool_output_truncated`: This event occurs when the output of a tool call is too large and gets truncated.
|
|
- **Attributes**:
|
|
- `tool_name` (string)
|
|
- `original_content_length` (int)
|
|
- `truncated_content_length` (int)
|
|
- `threshold` (int)
|
|
- `lines` (int)
|
|
- `prompt_id` (string)
|
|
|
|
- `qwen-code.malformed_json_response`: This event occurs when a `generateJson` response from Qwen API cannot be parsed as a json.
|
|
- **Attributes**:
|
|
- `model`
|
|
|
|
- `qwen-code.flash_fallback`: This event occurs when Qwen Code switches to flash as fallback.
|
|
- **Attributes**:
|
|
- `auth_type`
|
|
|
|
- `qwen-code.slash_command`: This event occurs when a user executes a slash command.
|
|
- **Attributes**:
|
|
- `command` (string)
|
|
- `subcommand` (string, if applicable)
|
|
|
|
- `qwen-code.extension_enable`: This event occurs when an extension is enabled
|
|
- `qwen-code.extension_install`: This event occurs when an extension is installed
|
|
- **Attributes**:
|
|
- `extension_name` (string)
|
|
- `extension_version` (string)
|
|
- `extension_source` (string)
|
|
- `status` (string)
|
|
- `qwen-code.extension_uninstall`: This event occurs when an extension is uninstalled
|
|
|
|
### Metrics
|
|
|
|
Metrics are numerical measurements of behavior over time. The following metrics are collected for Qwen Code (metric names remain `qwen-code.*` for compatibility):
|
|
|
|
- `qwen-code.session.count` (Counter, Int): Incremented once per CLI startup.
|
|
|
|
- `qwen-code.tool.call.count` (Counter, Int): Counts tool calls.
|
|
- **Attributes**:
|
|
- `function_name`
|
|
- `success` (boolean)
|
|
- `decision` (string: "accept", "reject", or "modify", if applicable)
|
|
- `tool_type` (string: "mcp", or "native", if applicable)
|
|
|
|
- `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency.
|
|
- **Attributes**:
|
|
- `function_name`
|
|
- `decision` (string: "accept", "reject", or "modify", if applicable)
|
|
|
|
- `qwen-code.api.request.count` (Counter, Int): Counts all API requests.
|
|
- **Attributes**:
|
|
- `model`
|
|
- `status_code`
|
|
- `error_type` (if applicable)
|
|
|
|
- `qwen-code.api.request.latency` (Histogram, ms): Measures API request latency.
|
|
- **Attributes**:
|
|
- `model`
|
|
|
|
- `qwen-code.token.usage` (Counter, Int): Counts the number of tokens used.
|
|
- **Attributes**:
|
|
- `model`
|
|
- `type` (string: "input", "output", "thought", or "cache")
|
|
|
|
- `qwen-code.file.operation.count` (Counter, Int): Counts file operations.
|
|
- **Attributes**:
|
|
- `operation` (string: "create", "read", "update"): The type of file operation.
|
|
- `lines` (Int, if applicable): Number of lines in the file.
|
|
- `mimetype` (string, if applicable): Mimetype of the file.
|
|
- `extension` (string, if applicable): File extension of the file.
|
|
- `model_added_lines` (Int, if applicable): Number of lines added/changed by the model.
|
|
- `model_removed_lines` (Int, if applicable): Number of lines removed/changed by the model.
|
|
- `user_added_lines` (Int, if applicable): Number of lines added/changed by user in AI proposed changes.
|
|
- `user_removed_lines` (Int, if applicable): Number of lines removed/changed by user in AI proposed changes.
|
|
- `programming_language` (string, if applicable): The programming language of the file.
|
|
|
|
- `qwen-code.chat_compression` (Counter, Int): Counts chat compression operations
|
|
- **Attributes**:
|
|
- `tokens_before`: (Int): Number of tokens in context prior to compression
|
|
- `tokens_after`: (Int): Number of tokens in context after compression
|