IronAlpine.Framework.Observability 3.1.1

IronAlpine.Framework.Observability

Serilog host setup with correlation enrichment and sensitive-data masking, the mediator logging behaviour, OpenTelemetry tracing/metrics with a Prometheus endpoint, and named Polly resilience pipelines.

• Package id: IronAlpine.Framework.Observability (3.1.1, net10.0, references Microsoft.AspNetCore.App)
• Depends on: Kernel, Mediator, Serilog.AspNetCore 9, OpenTelemetry 1.20, Polly 8

Installation

<PackageReference Include="IronAlpine.Framework.Observability" Version="3.1.1" />

Quick start

using IronAlpine.Observability;              // UseLoggingBehavior
using IronAlpine.Observability.Extensions;   // UseIronAlpineSerilog, AddIronAlpineTelemetry, AddIronAlpineResilience, UseIronAlpineTelemetry

var builder = WebApplication.CreateBuilder(args);

builder.Host.UseIronAlpineSerilog("orders-service");

builder.Services
    .AddIronAlpineTelemetry(builder.Configuration, "orders-service")
    .AddIronAlpineResilience(builder.Configuration);

builder.Services.AddIronAlpineMediator(cfg =>
{
    cfg.RegisterServicesFromAssembly(applicationAssembly);
    cfg.UseLoggingBehavior();     // first = outermost
});

var app = builder.Build();
app.UseIronAlpineTelemetry();     // Prometheus scraping endpoint

Logging (Serilog)

IHostBuilder.UseIronAlpineSerilog(serviceName = null, configure = null):

• registers IronAlpineSerilogOptions (section IronAlpine:Logging:Serilog), the masking destructuring policy and the event-metadata enricher (AddIronAlpineSerilog);
• configures Serilog with ReadFrom.Configuration (the standard Serilog section: Using, MinimumLevel, WriteTo, Enrich, Properties), Enrich.FromLogContext(), Application property (service name), EnvironmentName, ThreadId;
• adds TraceId, CorrelationId, CausationId, TenantId to every event when an IEventMetadataProvider is registered (the Web package registers one);
• adds an OpenTelemetry sink to IronAlpine:Observability:OpenTelemetry:OtlpEndpoint (or OpenTelemetry:OtlpEndpoint) unless Serilog:WriteTo already contains an OpenTelemetry sink;
• adds a Seq sink to Serilog:WriteTo:Seq:Args:serverUrl (or Seq:ServerUrl) unless Serilog:WriteTo already contains a Seq sink.

Service name precedence: IronAlpine:Logging:Serilog:ServiceName → serviceName argument → entry assembly name.

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console" ],
    "MinimumLevel": { "Default": "Information", "Override": { "Microsoft": "Warning", "System": "Warning" } },
    "WriteTo": [ { "Name": "Console", "Args": { "formatter": "Serilog.Formatting.Json.JsonFormatter, Serilog" } } ]
  },
  "IronAlpine": {
    "Logging": {
      "Serilog": {
        "ServiceName": "orders-service",
        "LogRequestPayloads": true,
        "MaskValue": "***",
        "SensitivePropertyNames": [ "iban", "nationalId" ]
      }
    }
  }
}

Sinks other than OpenTelemetry and Seq (for example Console or File) must be configured in the Serilog section; the package references Serilog.Sinks.OpenTelemetry and Serilog.Sinks.Seq only (Console comes with Serilog.AspNetCore).

Masking

SensitiveDataDestructuringPolicy replaces the value of properties whose name equals an entry of SensitivePropertyNames (case-insensitive) with MaskValue when an object is destructured ({@Request}). Defaults: password, secret, token, accessToken, refreshToken, apiKey, authorization, cookie, clientSecret, connectionString; configured names are added to them. Names are not matched as substrings (newPassword is not masked unless listed). Types in System.* namespaces, value types and collections themselves are not handled by the policy (their elements are).

Mediator logging behaviour

cfg.UseLoggingBehavior() adds LoggingBehavior<TRequest, TResponse> (TResponse : class):

• Information "Handling request {RequestName}" - with Payload: {@Request} when LogRequestPayloads is true (the default);
• Information "completed successfully in {ElapsedMilliseconds}ms" for successful Results and non-Result responses;
• Warning "failed with error {ErrorCode} ... {@Error}" for failed Results;
• exceptions are not caught (they are logged by the exception middleware of the Web package).

Set LogRequestPayloads to false when requests carry personal data.

Tracing and metrics (OpenTelemetry)

AddIronAlpineTelemetry(configuration, serviceName = null, configure = null) (an overload without configuration exists):

• forces W3C trace ids (Activity.DefaultIdFormat);
• resource service.name = IronAlpine:Observability:OpenTelemetry:ServiceName → serviceName → entry assembly;
• tracing: ASP.NET Core, HttpClient, EF Core instrumentation, the event bus ActivitySource (IronAlpine.Eventbus, TelemetryExtensions.EventBusActivitySourceName), an activity processor that adds correlation.id, causation.id, tenant.id tags from IEventMetadataProvider; OTLP exporter when OtlpEndpoint is set (IronAlpine:Observability:OpenTelemetry:OtlpEndpoint, fallback OpenTelemetry:OtlpEndpoint);
• metrics: ASP.NET Core (with Kestrel), HttpClient, runtime (with process CPU/memory), EF Core and the IronAlpine event bus and caching meters (see Metrics); Prometheus exporter (only when EnablePrometheusScrapingEndpoint is true).

The resolved options are registered both as IronAlpineTelemetryOptions and as IOptions<IronAlpineTelemetryOptions>. app.UseIronAlpineTelemetry() maps the Prometheus scraping endpoint at PrometheusScrapingEndpointPath (default /metrics) only when EnablePrometheusScrapingEndpoint is true (default).

{
  "IronAlpine": {
    "Observability": {
      "OpenTelemetry": {
        "ServiceName": "orders-service",
        "OtlpEndpoint": "http://otel-collector:4317",
        "EnablePrometheusScrapingEndpoint": true,
        "PrometheusScrapingEndpointPath": "/metrics"
      }
    }
  }
}

Add more sources with the standard OpenTelemetry builder; AddOpenTelemetry() can be called again:

builder.Services.AddOpenTelemetry().WithTracing(tracing => tracing.AddSource("MyCompany.Orders"));

Metrics

AddIronAlpineTelemetry listens to these meters (names are constants on TelemetryExtensions):

Process CPU and memory come from the System.Runtime meter on .NET 9+, so no process instrumentation package is referenced.

Prometheus names. The exporter turns dots into underscores, appends the unit (s → _seconds, By → _bytes) and adds _total to counters: eventbus.outbox.published → eventbus_outbox_published_total, eventbus.outbox.oldest_pending_age (unit s) → eventbus_outbox_oldest_pending_age_seconds, tag cache.definition → label cache_definition.

Event bus (IronAlpine.EventBus)

Tags are low-cardinality: topic, and for dead letters direction and reason. The event type is deliberately not a tag (every new event type would add time series); use logs or traces for per-type detail.

The four gauges never query the database during a scrape. A background sampler runs one count and one min query per table every 30 seconds (both served by the ProcessedOnUtc indexes) and keeps the result in memory; the age is computed from the stored timestamp at scrape time, so it keeps growing while the oldest message waits. The gauges report nothing until the first sample. Every pod reads the same tables and reports the same value: aggregate the gauges with max(), not sum().

Caching (IronAlpine.Caching)

cache_definition is the [CacheDefinition(CacheKey = ...)] key (for example product-price), or the query type name when the attribute is missing; never the concrete key (product-price:42).

Example PromQL

# Outbox publish failures per second, by topic
sum by (topic) (rate(eventbus_outbox_publish_failed_total[5m]))

# Oldest waiting outbox message (same value on every pod -> max)
max(eventbus_outbox_oldest_pending_age_seconds)

# Dead letters in the last hour, by direction and reason
sum by (direction, reason) (increase(eventbus_deadletter_created_total[1h]))

# Cache hit ratio per definition
sum by (cache_definition) (rate(cache_hit_total[5m]))
  / (sum by (cache_definition) (rate(cache_hit_total[5m])) + sum by (cache_definition) (rate(cache_miss_total[5m])))

# Share of 5xx responses
sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
  / sum(rate(http_server_request_duration_seconds_count[5m]))

Suggested alert rules

groups:
  - name: ironalpine
    rules:
      - alert: OutboxBacklogStuck
        expr: max(eventbus_outbox_oldest_pending_age_seconds) > 300
        for: 5m
      - alert: InboxBacklogStuck
        expr: max(eventbus_inbox_oldest_pending_age_seconds) > 300
        for: 5m
      - alert: DeadLettersCreated
        expr: sum by (direction, reason) (increase(eventbus_deadletter_created_total[15m])) > 0
      - alert: HighServerErrorRate
        expr: |
          sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
            / sum(rate(http_server_request_duration_seconds_count[5m])) > 0.05
        for: 10m

Prometheus exporter and dotnet_exceptions_total

OpenTelemetry.Exporter.Prometheus.AspNetCore up to 1.16.0-beta.1 threw and caught IndexOutOfRangeException while writing labels on every scrape, so dotnet_exceptions_total grew by a few on each scrape. 3.1.0 references the 1.20 line, where a scrape throws no exceptions (measured; the exporter changelog does not list it as a separate fix). The exporter still grows its response buffer by catching IndexOutOfRangeException/ArgumentException; this happens only when a response outgrows the buffer (initially 85 KB) and the grown buffer is kept, so it is rare. If an exception alert based on dotnet_exceptions_total fires after a deployment, check these two types first, or exclude them in the alert expression.

Event metadata

EventMetadataContext(TraceId, CorrelationId, CausationId, TenantId, TraceParent, TraceState, Baggage) describes the current request or message. IEventMetadataProvider.GetCurrent() reads it, IEventMetadataAccessor also sets and clears it. AsyncLocalEventMetadataAccessor (IronAlpine.Observability.Runtime) is an async-local implementation for non-HTTP hosts:

services.AddSingleton<AsyncLocalEventMetadataAccessor>();
services.AddSingleton<IEventMetadataAccessor>(sp => sp.GetRequiredService<AsyncLocalEventMetadataAccessor>());
services.AddSingleton<IEventMetadataProvider>(sp => sp.GetRequiredService<AsyncLocalEventMetadataAccessor>());

The Web package registers an HTTP-aware implementation (HttpEventMetadataProvider) that reads X-Correlation-ID, X-Causation-ID, X-Tenant-ID, traceparent, tracestate and baggage.

Resilience (Polly)

builder.Services.AddIronAlpineResilience(builder.Configuration, policies => policies
    .AddHttpPipeline("payments-api", b => b
        .AddRetry(new RetryStrategyOptions<HttpResponseMessage> { MaxRetryAttempts = 2 })
        .AddTimeout(TimeSpan.FromSeconds(5))));

public sealed class PaymentsClient(HttpClient http, IResiliencePipelineProvider resilience)
{
    public async Task<HttpResponseMessage> CaptureAsync(Guid id, CancellationToken ct)
        => await resilience.ExecuteHttpAsync(
            ResiliencePolicyNames.HttpDefault,
            token => new ValueTask<HttpResponseMessage>(http.PostAsync($"payments/{id}/capture", null, token)),
            ct);

    public ValueTask<int> CountAsync(CancellationToken ct)
        => resilience.ExecuteAsync("default", token => new ValueTask<int>(42), ct);
}

Overloads of AddIronAlpineResilience: (configuration), (Action<ResilienceOptions>), (configuration, Action<ResilienceOptions>), (configuration, Action<IResiliencePolicyRegistry>), (configuration, Action<ResilienceOptions>, Action<IResiliencePolicyRegistry>), (Action<ResilienceOptions>, Action<IResiliencePolicyRegistry>). Only one call may register custom pipelines.

IResiliencePipelineProvider:

• GetPipeline(name) / GetHttpPipeline(name) return a custom pipeline from the registry if one exists, otherwise build one from ResilienceOptions.Policies[name], falling back to the default policy for unknown names. Pipelines are cached and rebuilt when options change.
• Built pipelines: retry (outermost; handles every exception except OperationCanceledException; HTTP pipelines also retry 408, 429, 500, 502, 503, 504) → circuit breaker (when enabled) → timeout (per attempt, when enabled).
• ExecuteAsync, ExecuteAsync<TResult>, ExecuteHttpAsync extension methods (ResiliencePipelineProviderExtensions).

Section IronAlpine:Resilience (validated on start). Built-in policy names (ResiliencePolicyNames): default, http-default, kafka-producer, kafka-consumer, outbox-processing, inbox-processing, save-changes. Configured policies are merged into these defaults.

{
  "IronAlpine": {
    "Resilience": {
      "Policies": {
        "default": {
          "Enabled": true,
          "Retry": { "MaxRetryAttempts": 3, "BaseDelaySeconds": 1, "MaxDelaySeconds": 30, "BackoffType": "Exponential", "UseJitter": true },
          "CircuitBreaker": { "Enabled": false, "FailureRatio": 0.5, "MinimumThroughput": 20, "SamplingDurationSeconds": 30, "BreakDurationSeconds": 30 },
          "Timeout": { "Enabled": true, "TimeoutSeconds": 30 }
        },
        "payments-api": { "Retry": { "MaxRetryAttempts": 2 }, "Timeout": { "TimeoutSeconds": 5 } }
      }
    }
  }
}

The named Kafka, outbox, inbox and save-changes policies are defaults for application code; the EventBus and Data packages do not use them (they have their own retry settings).

Other contracts

• IExecutionModeDetector / DefaultExecutionModeDetector (IronAlpine.Observability.Runtime) detects EF Core design-time execution (dotnet ef) so start-up code can skip work that needs live infrastructure. It is not registered automatically.
• IronAlpine.Observability.Contracts.IAuditUser is [Obsolete] since 3.0.5; use IronAlpine.Kernel.Primitives.IAuditUser (the only contract the data layer and Security use).
• IronAlpine.Observability.Runtime.AsyncLocalEventMetadataAccessor and DefaultExecutionModeDetector are the canonical implementations; the copies in IronAlpine.Web.Runtime are [Obsolete] since 3.0.5.

Pitfalls

• eventbus_*_pending and eventbus_*_oldest_pending_age_seconds are the same table value on every pod; sum() multiplies them by the pod count. Use max().
• Up to 3.0.4 EnablePrometheusScrapingEndpoint and PrometheusScrapingEndpointPath had no effect (the endpoint was always /metrics) and event bus activities were not traced unless the source was added manually. Both are fixed in 3.0.5; check scrape configurations that relied on /metrics while a different path was configured.
• LogRequestPayloads defaults to true; request payloads are logged at Information.
• Masking matches exact property names only.
• Retry.MaxRetryAttempts = 0 means "no retries": the pipeline is built without a retry strategy (circuit breaker and timeout still apply). Negative values fail validation. Use "Enabled": false to turn the whole policy off.
• Without the Web package (or your own IEventMetadataProvider), logs and spans carry no correlation properties.