Serilog.Sinks.AzureDataExplorer 1.1.2

dotnet add package Serilog.Sinks.AzureDataExplorer --version 1.1.2                
NuGet\Install-Package Serilog.Sinks.AzureDataExplorer -Version 1.1.2                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Serilog.Sinks.AzureDataExplorer" Version="1.1.2" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Serilog.Sinks.AzureDataExplorer --version 1.1.2                
#r "nuget: Serilog.Sinks.AzureDataExplorer, 1.1.2"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Serilog.Sinks.AzureDataExplorer as a Cake Addin
#addin nuget:?package=Serilog.Sinks.AzureDataExplorer&version=1.1.2

// Install Serilog.Sinks.AzureDataExplorer as a Cake Tool
#tool nuget:?package=Serilog.Sinks.AzureDataExplorer&version=1.1.2                

Serilog.Sinks.AzureDataExplorer

.NET Nuget

A Serilog sink that writes events to an Azure Data Explorer (Kusto) cluster.

Package - Serilog.Sinks.AzureDataExplorer | Platforms - .Net 6.0

Getting started

Install from NuGet:

Install-Package Serilog.Sinks.AzureDataExplorer

How to use

This sink supports the durable mode where the logs are written to a file first and then flushed to the specified ADX table. This durable mode prevents data loss when the ADX connection couldnt be established. Durable mode can be turned on when we specify the bufferFileName in the LoggerConfiguration as mentioned below.

Configuration when durable mode is not required

var log = new LoggerConfiguration()
    .WriteTo.AzureDataExplorer(new AzureDataExplorerSinkOptions
    {
        IngestionEndpointUri = "https://ingest-mycluster.northeurope.kusto.windows.net",
        DatabaseName = "MyDatabase",
        TableName = "Serilogs"
    })
    .CreateLogger();

Configuration when durable mode is required

var log = new LoggerConfiguration()
    .WriteTo.AzureDataExplorer(new AzureDataExplorerSinkOptions
    {
        IngestionEndpointUri = "https://ingest-mycluster.northeurope.kusto.windows.net",
        DatabaseName = "MyDatabase",
        TableName = "Serilogs",
        BufferBaseFileName = "BufferBaseFileName",
        BufferFileRollingInterval = RollingInterval.Minute
    })
    .CreateLogger();

Note: Inorder to get the exception as string mapped to kusto column such as Exception, it is recommended to use ExceptionEx Attribute. For example:

new SinkColumnMapping { ColumnName ="Exception", ColumnType ="string", ValuePath = "$.ExceptionEx" }

Configuration of Azure Data Explorer Serilog sink through appsettings.json

You can configure the Azure Data Explorer Serilog Sink using an appsettings.json file. Below is a sample appsettings.json file that includes the Periodic Batching configuration options: batchPostingLimit, period, and queueSizeLimit.

Sample appsettings.json contents

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.AzureDataExplorer" ],
    "MinimumLevel": "Verbose",
    "WriteTo": [
      {
        "Name": "AzureDataExplorerSink",
        "Args": {
          "ingestionUri": "https://ingest-cluster-name",
          "databaseName": "sample",
          "tableName": "table",
          "applicationClientId": "xxxxxxxx",
          "applicationSecret": "xxxxxxx",
          "tenantId": "xxxxxxx",
          "isManagedIdentity": false,
          "isWorkloadIdentity": false,
          "batchPostingLimit": 1000,  // Optional
          "period": 10.0,             // Optional (in seconds)
          "queueSizeLimit": 100000    // Optional
        }
      }
    ]
  }
}

Parameters for Periodic Batching:

batchPostingLimit: Specifies the maximum number of events to include in a batch. Defaults to 1000.

period: Specifies the time in seconds between checking for event batches to post. Defaults to 10 seconds.

queueSizeLimit: Specifies the maximum number of events in the queue. Once this limit is reached, new events will be dropped until the queue size decreases. Defaults to 100000.

The appsettings.json can be loaded through the following piece of code

var configuration = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
            .Build();

var logger = new LoggerConfiguration()
            .ReadFrom.Configuration(configuration)
            .CreateLogger();

Required NuGet Packages

Ensure you have the following NuGet dependencies to enable configuration through appsettings.json:

  • Microsoft.Extensions.Configuration
  • Microsoft.Extensions.Configuration.Json
  • Serilog.Settings.Configuration

Features

Options

Batching

  • BatchPostingLimit: The maximum number of events to post in a single batch. Defaults to 1000.
  • Period: The time to wait between checking for event batches. Defaults to 10 seconds.
  • QueueSizeLimit: The maximum number of events that will be held in-memory while waiting to ship them to AzureDataExplorer. Beyond this limit, events will be dropped. The default is 100,000.

Target ADX Cluster

  • IngestionEndpointUri: Azure Data Explorer endpoint (Ingestion endpoint for Queued Ingestion, Query endpoint for Streaming Ingestion)
  • DatabaseName: The name of the database to which data should be ingested to
  • TableName: The name of the table to which data should be ingested to
  • FlushImmediately : In case queued ingestion is selected, this property determines if is needed to flush the data immediately to ADX cluster. Not recommended to enable for data with higher workloads. The default is false.

Mapping

Azure Data Explorer provides data mapping capabilities, allowing the ability to extract data rom the ingested JSONs as part of the ingestion. This allows paying a one-time cost of processing the JSON during ingestion, and reduced cost at query time.

By default, the sink uses the following data mapping:

Column Name Column Type JSON Path
Timestamp datetime $.Timestamp
Level string $.Level
Message string $.Message
Exception string $.Exception
Properties dynamic $.Properties

This mapping can be overridden using the following options:

  • MappingName: Use a data mapping configured in ADX.

  • ColumnsMapping: Use an ingestion-time data mapping.

    Note: If we need detailed exception messages along with stackTrace, innerExceptionDetails etc please use | Column Name | Column Type | JSON Path | |-------------|-------------|------------------| | Exception | string | $.ExceptionEx |

Durable Mode

Durable mode can be turned on when we specify the bufferFileName in the LoggerConfiguration. There are few other options available when the durable mode is enabled.

  • BufferBaseFileName : Enables the durable mode. When specified, the logs are written to the bufferFileName first and then ingested to ADX.

  • BufferFileRollingInterval : The interval at which buffer log files will roll over to a new file. The default is RollingInterval.Hour

  • BufferLogShippingInterval : The interval between checking the buffer files.

  • BufferFileSizeLimitBytes : The maximum size, in bytes, to which the buffer log file for a specific date will be allowed to grow. By default 10 MB is applied

  • BufferFileLoggingLevelSwitch : A switch allowing the pass-through minimum level to be changed at runtime.

  • BufferFileCountLimit : The maximum number of log files that will be retained, including the current log file. For unlimited retention, pass null. The default is 20.

Authentication

The sink supports authentication using various methods. Use one of the following methods to configure the desired authentication methods:

new AzureDataExplorerSinkOptions()
    .WithXXX(...)
Mode Method Notes
AadUserPrompt WithAadUserPrompt Recommended only development!
AadUserToken WithAadUserToken
AadApplicationCertificate WithAadApplicationCertificate
AadApplicationKey WithAadApplicationKey
AadApplicationSubjectName WithAadApplicationSubjectName
AadApplicationThumbprint WithAadApplicationThumbprint
AadApplicationToken WithAadApplicationToken
AadAzureTokenCredentials WithAadAzureTokenCredentials
AadUserManagedIdentity WithAadUserManagedIdentity
AadSystemManagedIdentity WithAadSystemManagedIdentity
AadWorkloadIdentity WithWorkloadIdentity

Note that if none of the authentication options are provided, AzCliIdentity , followed by AadUserPrompt will be attempted.

Product Compatible and additional computed target framework versions.
.NET net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.1.2 4,928 8/14/2024
1.1.1 1,477 8/2/2024
1.1.0 5,959 4/30/2024
1.0.9 432 3/19/2024
1.0.8 154 3/12/2024
1.0.7 155 3/5/2024
1.0.6 160 2/21/2024
1.0.5 495 1/26/2024
1.0.4 12,430 8/24/2023
1.0.3 5,485 3/9/2023
1.0.2 558 7/21/2022
1.0.1 493 7/11/2022
1.0.0 499 7/11/2022

Added GZip compression when sending data
     Improved options default values
     Added durability to the Azure Data Explorer Sink
     Added Sample project
     Added unit Tests
     Added End To End Test cases