# Observability Standard: Distributed Tracing

## Overview
OpenTelemetry distributed tracing for cross-service correlation and performance analysis.

## OpenTelemetry Setup

### Packages
```xml
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.*" />
<PackageReference Include="Azure.Monitor.OpenTelemetry.AspNetCore" Version="1.*" />
```

### Program.cs Configuration
```csharp
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation(opts =>
            {
                opts.RecordException = true;
                opts.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health");
            })
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation(opts => opts.SetDbStatementForText = true)
            .AddSource(ActivitySource.Name) // Custom activities
            .AddAzureMonitorTraceExporter(opts =>
            {
                opts.ConnectionString = config["ApplicationInsights:ConnectionString"];
            });
    });

// Custom ActivitySource for domain operations
public static class Telemetry
{
    public static readonly ActivitySource ActivitySource = new("MyApp", "1.0.0");
}
```

## Custom Span Creation

### Service Method Tracing
```csharp
public async Task<Order> ProcessOrderAsync(CreateOrderCommand command)
{
    using var activity = Telemetry.ActivitySource.StartActivity("ProcessOrder");
    activity?.SetTag("order.userId", command.UserId.ToString());
    activity?.SetTag("order.itemCount", command.Items.Count.ToString());

    try
    {
        var order = await _repository.CreateAsync(command);
        activity?.SetTag("order.id", order.Id.ToString());
        activity?.SetStatus(ActivityStatusCode.Ok);
        return order;
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
        activity?.RecordException(ex);
        throw;
    }
}
```

### Baggage (Cross-Service Context)
```csharp
// Set correlation ID in baggage (propagates across HTTP calls)
Activity.Current?.SetBaggage("correlationId", correlationId);
Activity.Current?.SetBaggage("userId", userId.ToString());

// Read baggage downstream
var correlationId = Activity.Current?.GetBaggageItem("correlationId");
```

## Trace Context Propagation

### HTTP Client (auto-propagated)
```csharp
// HttpClient automatically propagates W3C trace context
// Configure in DI:
builder.Services.AddHttpClient("ExternalApi")
    .AddOpenTelemetryInstrumentation();
```

### Service Bus (manual propagation)
```csharp
// Sender: inject trace context into message properties
var message = new ServiceBusMessage(body);
var activity = Activity.Current;
if (activity != null)
{
    message.ApplicationProperties["traceparent"] = activity.Id;
    message.ApplicationProperties["tracestate"] = activity.TraceStateString;
}

// Receiver: restore trace context from message
if (message.ApplicationProperties.TryGetValue("traceparent", out var traceparent))
{
    using var linkedActivity = Telemetry.ActivitySource.StartActivity(
        "ProcessMessage",
        ActivityKind.Consumer,
        parentId: traceparent.ToString()!);
    await ProcessAsync(message);
}
```

## Trace Sampling Strategy

| Environment | Strategy | Rate |
|-------------|----------|------|
| Development | Always | 100% |
| Staging | Adaptive | 10% + errors |
| Production | Adaptive | 5% + errors |

```csharp
// Production adaptive sampling (errors always sampled)
.AddAspNetCoreInstrumentation(opts =>
{
    opts.Enrich = (activity, eventName, obj) =>
    {
        if (obj is HttpResponse response && response.StatusCode >= 400)
        {
            activity.SetSampled(true); // Always sample errors
        }
    };
})
```
