ObDb 0.0.1

ObDb

A high-performance, zero-allocation L3 order book database for .NET.

Record individual order mutations (Add, Remove, Update) in a custom binary format optimized for sequential writes and fast random-access reads via timestamp seek.

Features

• L3 Fidelity — individual order tracking, not just aggregated price levels
• Zero-Allocation Write Path — Span<T> and pre-allocated buffers throughout
• Custom Binary Format — self-describing .obdb files with CRC-verified chunks
• Timestamp Seek — binary search the index to jump to any point in time
• Multi-Target — supports .NET 8, .NET 9, and .NET 10
• Crash Recovery — detect and repair incomplete files, rebuild indexes
• Server Mode — gRPC + REST APIs with live streaming via SSE
• CLI Tool — inspect, verify, repair, export, compact, prune data files

Installation

# Core library
dotnet add package ObDb

# CLI tool
dotnet tool install --global ObDb.Cli

Quick Start

using ObDb;
using ObDb.Reading;
using ObDb.Writing;
using System.Buffers.Binary;

// 1. Define a series schema
var spec = new SeriesSpec
{
    Name = "AAPL",
    OrderIdSize = 8,
    QuantityType = QuantityType.Long,
    MaxDeltasPerChunk = 10_000
};

var dataDir = "data/AAPL";
var orderIdBuf = new byte[8];

// 2. Write order mutations
using (var writer = new ObDbWriter(new ObDbWriterOptions
{
    OutputDirectory = dataDir,
    SeriesSpec = spec
}))
{
    long ts = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000;

    BinaryPrimitives.WriteInt64LittleEndian(orderIdBuf, 1);
    writer.Add(ts, orderIdBuf, Side.Bid, 185.00m, 100);

    BinaryPrimitives.WriteInt64LittleEndian(orderIdBuf, 2);
    writer.Add(ts + 1000, orderIdBuf, Side.Ask, 185.05m, 200);

    BinaryPrimitives.WriteInt64LittleEndian(orderIdBuf, 1);
    writer.Update(ts + 2000, orderIdBuf, Side.Bid, 185.01m, 150);

    BinaryPrimitives.WriteInt64LittleEndian(orderIdBuf, 2);
    writer.Remove(ts + 3000, orderIdBuf);
}

// 3. Replay from the beginning
using var reader = ObDbSeriesReader.Open(dataDir, "AAPL");
foreach (var update in reader.ReplayAll())
{
    Console.WriteLine($"[{update.Delta.Type}] Orders: {update.Book.OrderCount}, " +
        $"Bid: {update.Book.BestBid}, Ask: {update.Book.BestAsk}");
}

// 4. Seek to a specific timestamp
using var seekReader = ObDbSeriesReader.Open(dataDir, "AAPL");
foreach (var update in seekReader.ReplayFrom(ts + 2000))
{
    // Snapshot restored, replay continues from here
}

Architecture

Write Path                          Read Path
──────────                          ─────────
                                    Load .obdb.idx
Writer.Add/Remove/Update()          Binary search for timestamp
  │                                     │
  ▼                                     ▼
Buffer deltas in memory             Open referenced .obdb file
  │                                     │
  ▼                                     ▼
On chunk boundary:                  Read snapshot → restore book
  Write snapshot + deltas               │
  to new .obdb file                     ▼
  │                                 Replay deltas → yield
  ▼                                 (BookState, Delta) pairs
Append entry to .obdb.idx
  │
  ▼
fsync file + index

File format: Each .obdb file contains: File Header (128B) → Chunk Header (48B) → Snapshot → Deltas → Chunk Footer (12B, CRC-32) → File Footer (16B). One chunk per file, one index per series.

CLI Tool

# Inspect series metadata
obdb inspect --data-dir ./data --series AAPL

# Verify data integrity (CRC checks)
obdb verify --data-dir ./data --series AAPL

# Export to CSV/JSON
obdb export --data-dir ./data --series AAPL --format csv > events.csv

# Repair crashed files
obdb repair --data-dir ./data --series AAPL

# Rebuild index from data files
obdb reindex --data-dir ./data --series AAPL

# Compact files with new chunk size
obdb compact --data-dir ./data --series AAPL --target-deltas 50000

# Prune old data
obdb prune --data-dir ./data --series AAPL --before 2024-01-01

Server

ObDb includes a server with gRPC and REST APIs:

# Run directly
dotnet run --project src/ObDb.Server

# Or via Docker
docker compose up

REST endpoints (port 5000):

• POST /api/v1/{tenant}/series — create series
• POST /api/v1/{tenant}/series/{name}/write — write mutations
• GET /api/v1/{tenant}/series/{name}/snapshot — get book snapshot
• GET /api/v1/{tenant}/series/{name}/replay — replay via SSE
• GET /api/v1/{tenant}/series/{name}/subscribe — live stream via SSE

gRPC services (port 5001): SeriesService, WriteService, ReadService

Health checks: GET /health, GET /health/ready

Building from Source

dotnet build                    # Build all projects
dotnet test                     # Run all tests (all TFMs)
dotnet pack                     # Create NuGet packages

# Benchmarks
dotnet run -c Release --project benchmarks/ObDb.Benchmarks

License

Apache 2.0