PinkRoosterAi.Persistify 1.0.6

<div align="center"> <img src="img/logo_transparent.png" alt="PinkRoosterAi.Persistify Logo" width="300">

PinkRoosterAi.Persistify

A robust, thread-safe, and extensible persistent dictionary library for .NET

</div>

🚀 Overview

Persistify is a production-ready .NET library that seamlessly bridges in-memory dictionaries with persistent storage, offering thread-safe, asynchronous, and batch-optimized data persistence. Built with modern .NET patterns, it provides pluggable storage backends, intelligent caching, and robust error handling with retry logic.

Key Features

• 🔄 Thread-Safe Async Operations - Full async/await support with semaphore-based initialization safety
• 🔌 Pluggable Storage Backends - JSON files, SQLite, or custom providers
• ⚡ Intelligent Batching - Configurable size and time-based batch triggers for optimal performance
• 🎯 TTL-Based Caching - In-memory caching layer with automatic eviction and persistence backing
• 🛡️ Robust Error Handling - Polly-powered retry logic with exponential backoff and jitter
• 🎛️ Fluent Configuration - Builder pattern for clean, chainable setup
• 📊 Event-Driven Monitoring - Comprehensive error events with retry context

📦 Installation

Package Manager

Install-Package PinkRoosterAi.Persistify

.NET CLI

dotnet add package PinkRoosterAi.Persistify

PackageReference

<PackageReference Include="PinkRoosterAi.Persistify" Version="1.0.6" />

🏗️ Architecture

Persistify implements a non-generic provider pattern with pluggable storage backends:

┌─────────────────────────────┐
│    PersistentDictionary     │ ← Thread-safe Dictionary<string,T>
│  ┌─────────────────────────┐│
│  │ CachingPersistentDict.  ││ ← TTL-based caching layer
│  └─────────────────────────┘│
└─────────────────────────────┘
              │
              ▼
┌─────────────────────────────┐
│   IPersistenceProvider      │ ← Pluggable storage abstraction
├─────────────────────────────┤
│ JsonFilePersistenceProvider │ ← JSON file storage
│ DatabasePersistenceProvider │ ← SQLite database storage
│ CustomPersistenceProvider   │ ← Your custom implementation
└─────────────────────────────┘

Core Components

• PersistentDictionary<TValue> - Main thread-safe dictionary with async persistence
• CachingPersistentDictionary<TValue> - Memory cache with TTL-based eviction over persistence
• IPersistenceProvider - Storage abstraction supporting runtime type handling
• PersistenceProviderBuilder - Fluent configuration with method chaining

⚡ Quick Start

JSON File Storage

using PinkRoosterAi.Persistify;
using PinkRoosterAi.Persistify.Builders;

// Configure JSON file provider with batching
var provider = PersistenceProviderBuilder.JsonFile<int>()
    .WithFilePath("./data")
    .WithBatch(batchSize: 10, batchInterval: TimeSpan.FromSeconds(5))
    .WithRetry(maxAttempts: 3, delay: TimeSpan.FromMilliseconds(200))
    .Build();

// Create and initialize dictionary
var dict = provider.CreateDictionary("user-preferences");
await dict.InitializeAsync();

// Mutations are buffered and auto-flushed when batch size is reached
await dict.AddAndSaveAsync("theme", 1);
await dict.AddAndSaveAsync("language", 2);

// Indexer mutations also buffer for batch persistence
dict["setting1"] = 100;
dict["setting2"] = 200;
dict["setting3"] = 300;

// Manual flush for any remaining buffered changes
await dict.FlushAsync();

// Proper async disposal (also flushes pending changes)
await dict.DisposeAsync();

Database Storage

// Configure SQLite database provider
var dbProvider = PersistenceProviderBuilder.Database<string>()
    .WithConnectionString("Data Source=app.db;")
    .WithRetry(maxAttempts: 5, delay: TimeSpan.FromMilliseconds(500))
    .ThrowOnFailure(true) // Throw exceptions on persistent failures
    .WithBatch(batchSize: 20, batchInterval: TimeSpan.FromMinutes(1))
    .Build();

var sessionStore = dbProvider.CreateDictionary("user_sessions");
await sessionStore.InitializeAsync();

// Database operations with automatic retry
await sessionStore.AddAndSaveAsync("user123", "session-token-abc");
await sessionStore.TryAddAndSaveAsync("user456", "session-token-def");
await sessionStore.RemoveAndSaveAsync("user789");

await sessionStore.DisposeAsync();

TTL-Based Caching

// Create caching dictionary with 15-minute TTL
var cacheProvider = PersistenceProviderBuilder.JsonFile<object>()
    .WithFilePath("./cache")
    .Build();

var cache = cacheProvider.CreateCachingDictionary("api-cache", TimeSpan.FromMinutes(15));
await cache.InitializeAsync();

// Cache API responses
await cache.AddAndSaveAsync("user:123", new { Name = "John", Email = "john@example.com" });
await cache.AddAndSaveAsync("config:theme", "dark-mode");

// Access cached data (resets TTL)
var userData = cache["user:123"];

// Automatic eviction after 15 minutes of no access
await Task.Delay(TimeSpan.FromMinutes(16));
// cache["user:123"] will throw KeyNotFoundException

await cache.DisposeAsync();

🎛️ Configuration

JSON File Provider Options

var provider = PersistenceProviderBuilder.JsonFile<MyDataType>()
    .WithFilePath("./persistent-data")                    // Storage directory
    .WithSerializerOptions(new JsonSerializerOptions     // Custom JSON settings
    {
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
        WriteIndented = true
    })
    .WithBatch(
        batchSize: 50,                                    // Auto-flush after 50 changes
        batchInterval: TimeSpan.FromSeconds(30)           // Auto-flush every 30 seconds
    )
    .WithRetry(
        maxAttempts: 5,                                   // Retry up to 5 times
        delay: TimeSpan.FromMilliseconds(100)             // Base delay with exponential backoff
    )
    .ThrowOnFailure(false)                                // Use events instead of exceptions
    .Build();

Database Provider Options

var provider = PersistenceProviderBuilder.Database<ComplexType>()
    .WithConnectionString("Data Source=app.db;")          // SQLite connection string
    .WithBatch(batchSize: 100, batchInterval: TimeSpan.FromMinutes(5))
    .WithRetry(maxAttempts: 10, delay: TimeSpan.FromSeconds(1))
    .ThrowOnFailure(true)
    .Build();

Note: Column names (KeyColumnName, ValueColumnName) can be customized via DatabasePersistenceOptions when constructing the provider directly.

🔧 Advanced Usage

Error Handling & Events

var dict = provider.CreateDictionary("monitored-data");

// Subscribe to persistence error events
dict.PersistenceError += (sender, e) =>
{
    Console.WriteLine($"Persistence error in operation '{e.Operation}'");
    Console.WriteLine($"Retry attempt: {e.RetryAttempt}");
    Console.WriteLine($"Is fatal (all retries exhausted): {e.IsFatal}");
    Console.WriteLine($"Exception: {e.Exception.Message}");

    // Log to your telemetry system
    logger.LogWarning("Persistence failure: {Operation} attempt {Attempt}",
        e.Operation, e.RetryAttempt);
};

await dict.InitializeAsync();

Custom Persistence Provider

public class RedisPersistenceProvider : IPersistenceProvider
{
    private readonly IConnectionMultiplexer _redis;

    public IPersistenceOptions Options { get; }

    public RedisPersistenceProvider(string connectionString, IPersistenceOptions options)
    {
        _redis = ConnectionMultiplexer.Connect(connectionString);
        Options = options;
    }

    public async Task<Dictionary<string, object>> LoadAsync(
        string dictionaryName, Type valueType, CancellationToken ct)
    {
        var db = _redis.GetDatabase();
        var keys = await db.SetMembersAsync($"{dictionaryName}:keys");
        var result = new Dictionary<string, object>();

        foreach (var key in keys)
        {
            var value = await db.StringGetAsync($"{dictionaryName}:{key}");
            if (value.HasValue)
            {
                result[key!] = JsonSerializer.Deserialize(value!, valueType)!;
            }
        }
        return result;
    }

    public async Task SaveAsync(
        string dictionaryName, Type valueType, Dictionary<string, object> data, CancellationToken ct)
    {
        var db = _redis.GetDatabase();
        var transaction = db.CreateTransaction();

        transaction.KeyDeleteAsync($"{dictionaryName}:keys");
        foreach (var kvp in data)
        {
            var serializedValue = JsonSerializer.Serialize(kvp.Value, valueType);
            transaction.StringSetAsync($"{dictionaryName}:{kvp.Key}", serializedValue);
            transaction.SetAddAsync($"{dictionaryName}:keys", kvp.Key);
        }

        await transaction.ExecuteAsync();
    }

    // Implement remaining interface members: ExistsAsync, CreateDictionary, CreateCachingDictionary
}

Batch Operations & Performance

var provider = PersistenceProviderBuilder.JsonFile<int>()
    .WithFilePath("./metrics")
    .WithBatch(batchSize: 1000, batchInterval: TimeSpan.FromMinutes(5))
    .Build();

var highThroughputDict = provider.CreateDictionary("metrics");
await highThroughputDict.InitializeAsync();

// High-frequency updates - automatically batched
for (int i = 0; i < 10000; i++)
{
    highThroughputDict[$"metric_{i}"] = Random.Shared.Next(1, 100);
    // Persistence happens in background when batch size (1000) is reached
}

// Force flush remaining batched changes
await highThroughputDict.FlushAsync();

📊 Performance Characteristics

Batching Performance

• Individual operations (BatchSize=1): ~1ms per key (includes I/O)
• Batched operations: ~0.01ms per key in batch (100x improvement)
• Memory overhead: ~200 bytes per key
• Concurrent writers: Thread-safe with minimal lock contention

Storage Backends

🛠️ Thread Safety

Persistify is designed for high-concurrency scenarios:

• Reader safety: Concurrent reads are synchronized via lightweight locks
• Writer safety: Exclusive locks with minimal contention
• Initialization safety: Semaphore-based async initialization guard (only one thread initializes)
• Batch safety: Snapshot-based flush with _flushInProgress guard preventing concurrent flushes
• Disposal safety: Proper async cleanup with pending change flush

// Safe concurrent usage
var dict = provider.CreateDictionary("concurrent-data");
await dict.InitializeAsync();

// Multiple threads can safely read and write concurrently
var tasks = Enumerable.Range(0, 100).Select(async i =>
{
    await dict.AddAndSaveAsync($"key_{i}", $"value_{i}");
});

await Task.WhenAll(tasks);

🧪 Testing

Unit Testing with Mocks

[Fact]
public async Task Should_Handle_Persistence_Failures()
{
    var mockProvider = new Mock<IPersistenceProvider<int>>(MockBehavior.Strict);
    var mockOptions = new Mock<IPersistenceOptions>();
    mockOptions.Setup(o => o.BatchSize).Returns(1);
    mockOptions.Setup(o => o.BatchInterval).Returns(TimeSpan.Zero);
    mockOptions.Setup(o => o.MaxRetryAttempts).Returns(3);
    mockOptions.Setup(o => o.RetryDelay).Returns(TimeSpan.FromMilliseconds(100));
    mockOptions.Setup(o => o.ThrowOnPersistenceFailure).Returns(false);
    mockProvider.Setup(p => p.Options).Returns(mockOptions.Object);
    mockProvider.Setup(p => p.ExistsAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()))
        .ReturnsAsync(false);
    mockProvider.Setup(p => p.SaveAsync(It.IsAny<string>(),
            It.IsAny<Dictionary<string, int>>(), It.IsAny<CancellationToken>()))
        .ThrowsAsync(new IOException("Disk full"));

    var dict = new PersistentDictionary<int>(mockProvider.Object, "test");

    var errorEventFired = false;
    dict.PersistenceError += (s, e) => errorEventFired = true;

    await dict.InitializeAsync();
    await dict.AddAndSaveAsync("key", 123);

    // Allow background flush to complete
    await Task.Delay(500);
    Assert.True(errorEventFired);
}

Integration Testing

[Fact]
public async Task Should_Persist_Data_Across_Instances()
{
    var tempPath = Path.Combine(Path.GetTempPath(), "persistify-test");
    Directory.CreateDirectory(tempPath);

    var provider = PersistenceProviderBuilder.JsonFile<string>()
        .WithFilePath(tempPath)
        .Build();

    // First instance
    var dict1 = provider.CreateDictionary("integration-test");
    await dict1.InitializeAsync();
    await dict1.AddAndSaveAsync("test-key", "test-value");
    await dict1.FlushAsync();
    await dict1.DisposeAsync();

    // Second instance should load existing data
    var provider2 = PersistenceProviderBuilder.JsonFile<string>()
        .WithFilePath(tempPath)
        .Build();
    var dict2 = provider2.CreateDictionary("integration-test");
    await dict2.InitializeAsync();

    Assert.Equal("test-value", dict2["test-key"]);
    await dict2.DisposeAsync();
}

📋 API Reference

For the full API reference with detailed parameters, return values, and exceptions, see docs/API.md.

PersistentDictionary<TValue>

CachingPersistentDictionary<TValue>

Inherits all PersistentDictionary<TValue> methods. Adds TTL-based automatic eviction: entries not accessed within the configured time-to-live are removed from memory and the change is flushed to storage.

Events

🔧 Requirements

• .NET 9.0 or later
• Dependencies:
  • Microsoft.Extensions.Logging.Abstractions (>=9.0.6)
  • Polly (>=8.6.1) - Retry logic and resilience
  • ServiceStack.OrmLite.Sqlite (>=8.8.0) - SQLite support

📚 Examples

Real-World Usage Scenarios

Application Configuration Store

var configProvider = PersistenceProviderBuilder.JsonFile<object>()
    .WithFilePath("./config")
    .Build();

var appConfig = configProvider.CreateDictionary("app-settings");
await appConfig.InitializeAsync();

// Load/save configuration
appConfig["database.connectionString"] = "Server=...";
appConfig["features.enableAdvancedSearch"] = true;
appConfig["cache.ttlMinutes"] = 30;

await appConfig.FlushAsync();

User Session Management

var sessionProvider = PersistenceProviderBuilder.Database<UserSession>()
    .WithConnectionString("Data Source=sessions.db;")
    .WithBatch(batchSize: 50, batchInterval: TimeSpan.FromMinutes(2))
    .Build();

var sessions = sessionProvider.CreateCachingDictionary("user_sessions", TimeSpan.FromHours(4));
await sessions.InitializeAsync();

// Manage user sessions with automatic TTL eviction
await sessions.AddAndSaveAsync(sessionId, new UserSession
{
    UserId = userId,
    CreatedAt = DateTime.UtcNow,
    LastActivity = DateTime.UtcNow
});

Analytics Data Collection

var analyticsProvider = PersistenceProviderBuilder.JsonFile<AnalyticsEvent>()
    .WithFilePath("./analytics")
    .WithBatch(batchSize: 1000, batchInterval: TimeSpan.FromMinutes(5))
    .WithRetry(maxAttempts: 10, delay: TimeSpan.FromSeconds(1))
    .Build();

var events = analyticsProvider.CreateDictionary("analytics-events");
await events.InitializeAsync();

// High-frequency event collection - automatically batched
events[$"event_{DateTime.UtcNow.Ticks}"] = new AnalyticsEvent
{
    UserId = userId,
    EventType = "page_view",
    Timestamp = DateTime.UtcNow,
    Properties = new { Page = "/dashboard", Duration = 2500 }
};

🤝 Contributing

Contributions are welcome! Please open an issue or pull request on GitHub.

Development Setup

git clone https://github.com/pinkroosterai/Persistify.git
cd Persistify
dotnet restore
dotnet build
dotnet test

Running Samples

dotnet run --project PinkRoosterAi.Persistify.Samples

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

• Issues: GitHub Issues
• API Reference: docs/API.md
• NuGet Package: PinkRoosterAi.Persistify

<div align="center"> _{Built with ❤️ by <a href="https://github.com/pinkroosterai">PinkRoosterAI</a>} </div>