OpenTelemetry

Agent SDK sends traces, metrics, and logs to an OTLP collector when you set the OTEL_* variables. Record your own with @opentelemetry/api.

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$OTEL_TOKEN"
export OTEL_SERVICE_NAME=incident-agent
agent-sdk serve --dir . --dev

• The default protocol is http/protobuf. http/json also works, and grpc logs a warning and falls back to http/protobuf.
• Metrics use delta temporality. Grafana Cloud and Prometheus reject delta, so set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative when you export to either one.
• If you don't set an endpoint, Agent SDK doesn't export anything.
• You can create a meter, tracer, or logger at the top of bot/agent.ts, and it will use Agent SDK's providers.

Variables

Variable	Effect
OTEL_EXPORTER_OTLP_ENDPOINT	Base URL. Appends /v1/traces, /v1/metrics, /v1/logs.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	Traces URL, used as written. Overrides the base URL.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT	Metrics URL, used as written. Overrides the base URL.
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT	Logs URL, used as written. Overrides the base URL.
OTEL_EXPORTER_OTLP_HEADERS	Sent on every export. URL-encode values. Reread on each export.
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf (default) or http/json.
OTEL_SERVICE_NAME	service.name. Defaults to the agent name, then unknown_service:node.
OTEL_RESOURCE_ATTRIBUTES	service.namespace, service.version, deployment.environment.name.
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE	delta (default), cumulative, or lowmemory.
OTEL_LOGS_EXPORTER	otlp exports logs. Unset, none, and console do not.
OTEL_SDK_DISABLED	true turns export off.
OTEL_TRACES_EXPORTER	none turns traces off.
OTEL_METRICS_EXPORTER	none turns metrics off.
OTEL_LOG_USER_PROMPTS	1 adds prompt text to log records.
OTEL_LOG_TOOL_CONTENT	1 adds tool payloads to log records.

Hosted agents

export default defineAgent({
  hosting: {
    egressDomains: ["otlp-gateway-prod-us-east-0.grafana.net"],
    secretNames: [
      "OTEL_EXPORTER_OTLP_ENDPOINT",
      "OTEL_EXPORTER_OTLP_HEADERS",
      "OTEL_SERVICE_NAME",
    ],
  },
});

service.name is fixed by the first export, so redeploy to change it.

Record and tag

import { metrics, trace } from "@opentelemetry/api";
import { logs } from "@opentelemetry/api-logs";
import { currentConversationId } from "@grok/agent-sdk/otel";

const tracer = trace.getTracer("incident-agent");
const lookups = metrics.getMeter("incident-agent").createCounter("incident.lookups");
const logger = logs.getLogger("incident-agent");

lookups.add(1, { "cursor.conversation.id": currentConversationId() });

Spans started during a turn get cursor.conversation.id, and a metric gets it only when you set it.

To tag a whole turn, call ctx.telemetry from a tool. Spans in the turn get the trace keys, Agent SDK's log records get the log keys, and cursor.token.usage gets the metric keys. Outside a tool, use telemetry from @grok/agent-sdk/telemetry.

async execute({ key }, ctx) {
  ctx.telemetry.setTraceAttributes({ "work.item": key });
  ctx.telemetry.setMetricAttributes({ "work.item": key });
  ctx.telemetry.setLogAttributes({ "work.item": key });
  return await jira.getIssue(key);
}

Grafana renames a . in a metric key to _.

sum by (work_item) (cursor_token_usage_total)

• Each distinct metric value is a new series, so never use a per-turn ID, a secret, an email, or a raw user ID.
• A session holds up to 32 keys of each kind, with keys up to 128 characters and values up to 256.

Limits

• Agent SDK doesn't support custom span processors, samplers, views, your own OpenTelemetry SDK, or library auto-instrumentation. It reads OTEL_TRACES_SAMPLER and OTEL_METRIC_EXPORT_INTERVAL once at startup, so restart the process after you change either one.
• Import Agent SDK before any other OpenTelemetry setup. Otherwise the turn's tags may not reach your spans.

Migrating from serve({ otel })

Delete serve({ otel }), bot/otel.ts, bot/instrumentation.ts, and any defineOtel call. Set these variables instead.

Old option	Set this instead
serviceName	OTEL_SERVICE_NAME
serviceVersion	service.version in OTEL_RESOURCE_ATTRIBUTES
resourceAttributes	OTEL_RESOURCE_ATTRIBUTES, comma-separated, URL-encoded values
exporters[].url	OTEL_EXPORTER_OTLP_ENDPOINT (appends /v1/traces, /v1/metrics, /v1/logs)
exporters[].url with signals	OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, or OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, used as written
exporters[].headers	OTEL_EXPORTER_OTLP_HEADERS, comma-separated, URL-encoded values
exporters[].protocol	OTEL_EXPORTER_OTLP_PROTOCOL, or OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_METRICS_PROTOCOL, OTEL_EXPORTER_OTLP_LOGS_PROTOCOL
exporters[].signals	The per-signal URL. OTEL_TRACES_EXPORTER=none, OTEL_METRICS_EXPORTER=none, or OTEL_LOGS_EXPORTER=none stops that signal.
traces: false	OTEL_TRACES_EXPORTER=none
traces: true	Leave OTEL_TRACES_EXPORTER unset. Traces start when an endpoint is set.
traces.sampleRatio	OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG, before process start
metrics: false	OTEL_METRICS_EXPORTER=none
metrics: true	Leave OTEL_METRICS_EXPORTER unset. Metrics start when a metrics URL is set.
metrics.exportIntervalMillis	OTEL_METRIC_EXPORT_INTERVAL, milliseconds, before process start
logs: false	Unset OTEL_LOGS_EXPORTER, or set none
logs: true	OTEL_LOGS_EXPORTER=otlp
logs.userPrompts	OTEL_LOGS_EXPORTER=otlp and OTEL_LOG_USER_PROMPTS=1
logs.toolContent	OTEL_LOGS_EXPORTER=otlp and OTEL_LOG_TOOL_CONTENT=1
otel: false	No endpoint, or OTEL_SDK_DISABLED=true

Set the hosted values after deploy:

agent-sdk secret set my-agent OTEL_SERVICE_NAME OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS