TestDynamo.Serialization 0.8.0

TestDynamo

An in-memory dynamodb client for automated testing

TestDynamo is a rewrite of dynamodb in dotnet designed for testing and debugging. It implements a partial feature set of IAmazonDynamoDb to manage schemas and read and write items.

• Core features
  • Table management (Create/Update/Delete table)
  • Index management (Create/Update/Delete index)
  • Item operations (Put/Delete/Update etc)
  • Queries and Scans
  • Batching and Transactional writes
• Document model and DynamoDBContext
• Multi region setups
• Global tables and replication
• Streams and stream subscribers
• Efficient cloning and deep copying of databases for isolation
• Full database serialization and deserialization for data driven testing
• Basic cloudformation template support for creating Tables and GlobalTables

Installation

• Core functionality: dotnet add package TestDynamo
• Add lambda support: dotnet add package TestDynamo.Lambda
• Add serialization or cloud formation support: dotnet add package TestDynamo.Serialization

The basics

using TestDynamo;

[Test]
public async Task GetPersonById_WithValidId_ReturnsPerson()
{
   // arrange
   using var client = TestDynamoClient.CreateClient<AmazonDynamoDBClient>();

   // create a table and add some items
   await client.CreateTableAsync(...);
   await client.BatchWriteItemAsync(...);

   var testSubject = new MyBeatlesService(client);

   // act
   var beatle = testSubject.GetBeatle("Ringo");

   // assert
   Assert.Equal("Starr", beatle.SecondName);
}

The details

TestDynamo has a suite of features and components to model a dynamodb environment and simplify the process of writing tests.

• Api.Database contains tables from a single region.
  • Full expression engine so you can test your queries, scans, projections and conditions
  • Schema and item change tools make creating and populating test databases easier
  • Database cloning allows you to make copies of entire AWS regions which can be safely used in other tests.
  • Debug properties are optimized for reading in a debugger.
  • Test query tools to get data in and out of the database with as little code as possible
  • Streaming and Subscriptions can model lambdas subscribed to dynamodb streams
• DynamoDBContext works out of the box
• Api.GlobalDatabase models an AWS account spread over multiple regions. It contains Api.Databases.
  • Global databases can be cloned too
• TestDynamoClient is the entry point for linking a database to a AmazonDynamoDBClient.
  • Check the features for a full list of endpoints, request args and responses that are supported.
• DatabaseSerializer is a json serializer/deserializer for entire databases and global databases.
• Cloud Formation Templates can be consumed to to initialize databases and global databases
• Locking and atomic transactions
• Transact write ClientRequestToken
• Interceptors can be used to modify the functionality of the database, either to add more traditional mocking or to polyfill unsupported features
• Logging can be configured at the database level or the AmazonDynamoDBClient level

Using Expressions

All dynamodb expression types are supported

Database

The database is the core of TestDynamo and it models a single region and it's tables. The database is both fast and lightweight, built from simple data structures.

Databases can be injected into an AmazonDynamoDBClient to then be passed into a test

using TestDynamo;

using var db = new Api.Database(new DatabaseId("us-west-1"));
using var client = db.CreateClient<AmazonDynamoDBClient>();

Schema and Item Change

Databases have some convenience methods to make adding tables and items easier

using TestDynamo;

using var database = new Api.Database(new DatabaseId("us-west-1"));

// add a table
database
   .TableBuilder("Beatles", ("FirstName", "S"))
   .WithGlobalSecondaryIndex("SecondNameIndex", ("SecondName", "S"), ("FirstName", "S"))
   .AddTable();

// add some data
database
   .ItemBuilder("Beatles")
   .Attribute("FirstName", "Ringo")
   .Attribute("SecondName", "Starr")
   .AddItem();

Database Cloning

Databases are built for cloning. This allows you to create and populate a database once, and then clone it to be used in any test with full isolation. TestDynamo uses immutable data structures for most things, which means that nothing is actually copied during a clone so cloning has almost no overhead.

using TestDynamo;

private static Api.Database _sharedRootDatabase = BuildDatabase();

private static Api.Database BuildDatabase()
{
   var database = new Api.Database(new DatabaseId("us-west-1"));

   // add a table
   database
      .TableBuilder("Beatles", ("FirstName", "S"))
      .WithGlobalSecondaryIndex("SecondNameIndex", ("SecondName", "S"), ("FirstName", "S"))
      .AddTable();

   // add some data
   database
      .ItemBuilder("Beatles")
      .Attribute("FirstName", "Ringo")
      .Attribute("SecondName", "Starr")
      .AddItem();

   return database;
}

[Test]
public async Task TestSomething()
{
   // clone the database to get working copy
   // without altering the original
   using var database = _sharedRootDatabase.Clone();
   using var client = database.CreateClient<AmazonDynamoDBClient>();

   // act
   ...

   // assert
   ...
}

Debug Properties

Use the handy debug properties on Api.Database and Api.GlobalDatabase in your debugger of choice

Test Tools

Use query tools to find items in a table

using var database = GetMeADatabase();

var ringo = database
    .GetTable("Beatles")
    .GetValues()
    .Single(v => v["FirstName"].S == "Ringo");

Streaming and Subscriptions

If streams are enabled on tables they can be used for global table replication and custom subscribers. TestDynamo differs from dynamodb as follows

• There is no limit to the number of subscribers that you can have on a stream
• Strem settings (e.g. NEW_AND_OLD_IMAGES) are configured per subscriber. If these values are set on a stream they will be ignored

To subscribe to changes with a lambda stream subscription syntax you can import the TestDynamo.Lambda package from nuget

using TestDynamo;
using TestDynamo.Lambda;

var subscription = database.AddSubscription(
   "Beatles",
   (dynamoDbStreamsEvent, cancellationToken) =>
   {
      var added = dynamoDbStreamsEvent.Records.FirstOrDefault()?.Dynamodb.NewImage?["FirstName"]?.S;
      if (added != null)
         Console.WriteLine($"{added} has joined the Beatles");

      var removed = dynamoDbStreamsEvent.Records.FirstOrDefault()?.Dynamodb.OldImage?["FirstName"]?.S;
      if (removed != null)
         Console.WriteLine($"{removed} has left the Beatles");

      return default;
   });

// disposing will remove the subscription
subscription.Dispose();

Subscribe to raw changes

var subscription = database
   .SubscribeToStream("Beatles", (cdcPacket, cancellationToken) =>
   {
      var added = cdcPacket.data.packet.changeResult.OrderedChanges
         .Select(x => x.Put)
         .Where(x => x.IsSome)
         .Select(x => x.Value["FirstName"].S)
         .FirstOrDefault();

      if (added != null)
         Console.WriteLine($"{added} has joined the Beatles");

      var removed = cdcPacket.data.packet.changeResult.OrderedChanges
         .Select(x => x.Deleted)
         .Where(x => x.IsSome)
         .Select(x => x.Value["FirstName"].S)
         .FirstOrDefault();

      if (removed != null)
         Console.WriteLine($"{removed} has left the Beatles");

      return default;
   });

// disposing will remove the subscription
subscription.Dispose();

Subscriptions synchonicity and error handling can be customized with SubscriberBehaviour through the SubscribeToStream and AddSubscription methods.

Subscriptions can be executed synchonously or asynchonously. For example, if a subscription is configured to execute synchronously, and a PUT request is executed, the AmazonDynamoDBClient will not return a PUT response until the subscriber has completed it's work. If the subscription is asynchronous, then subscriber execution is disconnected from the event trigger.

If a subscriber is synchronous, its errors can be propagated back to the trigger method, allowing for more direct test results. Otherwise, errors are cached and can be retrieved in the form of Exceptions when an AwaitAllSubscribers method is called

AwaitAllSubscribers

The Api.Database, Api.GlobalDatabase and AmazonDynamoDBClient have AwaitAllSubscribers methods to pause test execution until all subscribers have executed. This method will throw any exceptions that were experienced within subscribers and were not propagated synchronously. For AmazonDynamoDBClient, the AwaitAllSubscribers method is static and available on the TestDynamoClient class.

DynamoDBContext

using TestDynamo;

using var client = TestDynamoClient.CreateClient<AmazonDynamoDBClient>();

... // initialize the database schema

using var context = new DynamoDbContext(client)
await context.SaveAsync(new Beatle
{
   FirstName = "Ringo",
   SecondName = "Starr"
})

Global Database

The global database models an AWS account with a collection of regions. Each region is an Api.Database. It is used to test global table functionality

Creating global tables is a synchonous operation. The global table will be ready to use as soon as the AmazonDynamoDBClient client returns a response.

using TestDynamo;

using var globalDatabase = new GlobalDatabase();

// create a global table from ap-south-2
using var apSouth2Client = globalDatabase.CreateClient<AmazonDynamoDBClient>(new DatabaseId("ap-south-2"));
await apSouth2Client.CreateGlobalTableAsync(...);

// create a local table in cn-north-1
using var cnNorthClient = globalDatabase.CreateClient<AmazonDynamoDBClient>(new DatabaseId("cn-north-1"));
await cnNorthClient.CreateTableAsync(...);

Global Database Cloning

using TestDynamo;

using var globalDatabase = new GlobalDatabase();
using var db2 = globalDatabase.Clone();

AwaitAllSubscribers

The Api.GlobalDatabase and AmazonDynamoDBClient have AwaitAllSubscribers methods to pause test execution until all data has been replicated between databases. For AmazonDynamoDBClient, the AwaitAllSubscribers method is static and available on the TestDynamoClient class.

TestDynamoClient

TestDynamoClient links an AmazonDynamoDBClient to an Api.Database or an Api.GlobalDatabase. It has several useful extension methods

Create methods

using TestDynamo;

// create a client with an empty database
using var client1 = TestDynamoClient.CreateClient<AmazonDynamoDBClient>();

// create a client from an existing database
using var db1 = new Api.Database();
using var client21 = db1.CreateClient<AmazonDynamoDBClient>();

using var db2 = new Api.GlobalDatabase();
using var client22 = db2.CreateClient<AmazonDynamoDBClient>();

// attach a database to an existing client
using var db3 = new Api.Database();
using var client3 = new AmazonDynamoDBClient();
db3.Attach(client3);

Get methods

using TestDynamo;

using var client1 = TestDynamoClient.CreateClient<AmazonDynamoDBClient>();

// get the underlying database from a client
var db1 = TestDynamoClient.GetDatabase(clientclient);

// get a debug table from a client
var beatles = TestDynamoClient.GetTable(client, "Beatles");

Set methods

using TestDynamo;

using var client = TestDynamoClient.CreateClient<AmazonDynamoDBClient>();

// set an artificial processing delay
TestDynamoClient.SetProcessingDelay(client, TimeSpan.FromSeconds(0.1));

// set paging settings for database
TestDynamoClient.SetScanLimits(client, ...);

// set the AWS account id for the client
TestDynamoClient.SetAwsAccountId(client, "12345678");

Database Serializers

Database serializers are available from the TestDynamo.Serialization nuget package.

Database serializers can serialize or deserialize an entire database or global database to facilitate data driven testing.

using TestDynamo;
using TestDynamo.Serialization;

using var db1 = new Api.Database();
... populate database

DatabaseSerializer.Database.ToFile(db1, @"TestData.json");

using var db2 = DatabaseSerializer.Database.FromFile(@"TestData.json");

// there are also tools to serialize and deserialze global databases
var json = DatabaseSerializer.GlobalDatabase.ToString(globalDb, @"TestData.json");

Serialization is designed to share data between test runs, but ultimately, it scales with the number of items in the database. This means that it may take more time than is ideal for executing fast unit tests. Database cloning is a better solution for large databases which are shared between multiple tests, as it executes instantly for any sized database or global database

Cloud Formation Templates

Static Cloud Formation templates can be consumed to create databases and global databases. Dynamic templates with functions are not supported.

Import the dotnet add package TestDynamo.Serialization package to use cloudformation templates

using TestDynamo.Serialization;

var cfnFile1 = new CloudFormationFile(await File.ReadAllTextAsync("myTemplate1.json"), "eu-north-1");
var cfnFile2 = new CloudFormationFile(await File.ReadAllTextAsync("myTemplate2.json"), "us-west-2");
using var database = await CloudFormationParser.BuildDatabase(new[] { cfnFile1, cfnFile2 }, new CloudFormationSettings(true));
...

Locking and Atomic transactions

Test dynamo is more consistant than DynamoDb. In general, all operations on a single database (region) are atomic. Within the AmazonDynamoDBClient client, BatchRead and BatchWrite operations are executed as several independant operations in order to simulate non consistency.

The biggest differences you will see are

• Reads are always atomic
• Writes from tables to global secondary indexes are always atomic

Transact write ClientRequestToken

Client request tokens are used in transact write operations as an idempotency key. If 2 requests have the same client request token, the second one will not be executed. By default AWS keeps client request tokens for 10 minutes. TestDynamo keeps client request tokens for 10 seconds. This cache time can be updated in Settings.

Interceptors

Interceptors can be added to intercept and override certain database functionality.

For example, this sample implements create and restore backup functionality

Implement backups functionality

using TestDynamo;
using TestDynamo.Api.FSharp;
using TestDynamo.Client;
using TestDynamo.Model;

/// <summary>
/// An interceptor which implements the CreateBackup and RestoreTableFromBackup operations
/// </summary>
public class CreateBackupInterceptor(Dictionary<string, DatabaseCloneData> backupStore) : IRequestInterceptor
{
    public async ValueTask<object?> InterceptRequest(Api.FSharp.Database database, object request, CancellationToken c)
    {
        if (request is CreateBackupRequest create)
            return CreateBackup(database, create.TableName);

        if (request is RestoreTableFromBackupRequest restore)
            return await RestoreBackup(database, restore.BackupArn);

        // return null to allow the client to process other request types as normal
        return null;
    }

    private CreateBackupResponse CreateBackup(Api.FSharp.Database database, string tableName)
    {
        // wrap the database in something that is more C# friendly
        using var csDatabase = new Api.Database(database);

        // clone the required database and remove all other tables
        var cloneData = csDatabase.BuildCloneData();
        cloneData = new DatabaseCloneData(
            cloneData.data.ExtractTables(new [] { tableName }),
            cloneData.databaseId);

        // create a fake arn and store a cloned DB as a backup
        var arn = $"{database.Id.regionId}/{tableName}";
        backupStore.Add(arn, cloneData);

        return new CreateBackupResponse
        {
            BackupDetails = new BackupDetails
            {
                BackupArn = arn,
                BackupStatus = BackupStatus.AVAILABLE
            }
        };
    }

    private async ValueTask<RestoreTableFromBackupResponse> RestoreBackup(Api.FSharp.Database database, string arn)
    {
        // parse fake ARN created in the CreateBackup method
        var arnParts = arn.Split("/");
        if (arnParts.Length != 2)
            throw new AmazonDynamoDBException("Invalid backup arn");

        var tableName = arnParts[1];
        if (!backupStore.TryGetValue(arn, out var backup))
            throw new AmazonDynamoDBException("Invalid backup arn");

        // wrap the database in something that is more C# friendly
        using var csDatabase = new Api.Database(database);

        // delete any existing data to make room for restore data
        if (csDatabase.TryDescribeTable(tableName).IsSome)
            await csDatabase.DeleteTable(tableName);

        csDatabase.Import(backup.data);
        return new RestoreTableFromBackupResponse
        {
            TableDescription = new TableDescription
            {
                TableName = tableName
            }
        };
    }

    // no need to intercept responses
    public ValueTask<object?> InterceptResponse(Api.FSharp.Database database, object request, object response, CancellationToken c) => default;
}

// create an in memory store for backups
var backups = new Dictionary<string, DatabaseCloneData>();

using var database = new Api.Database(new DatabaseId("us-west-1"));

// create an interceptor and use in a client
var interceptor = new CreateBackupInterceptor(backups);
using var client = database.CreateClient<AmazonDynamoDBClient>(interceptor);

// execute some requests which are not intercepted. These will not be intercepted
await client.PutItemAsync(...);
await client.PutItemAsync(...);

// create a backup. This will be intercepted
var backupResponse = await client.CreateBackupAsync(new CreateBackupRequest
{
   TableName = "Beatles"
});

// restore from backup. This will be intercepted
await client.RestoreTableFromBackupAsync(new RestoreTableFromBackupRequest
{
   BackupArn = backupResponse.BackupDetails.BackupArn
});

Implement BillingMode functionality

Here is another example of how to implement some out of scope BillingMode functionality with an interceptor

/// <summary>
/// An interceptor which implements BillingMode functionality for CreateTableAsync
/// </summary>
public class BillingModeInterceptor : IRequestInterceptor
{
    // request interception is not requred
    public ValueTask<object?> InterceptRequest(Api.FSharp.Database database, object request, CancellationToken c) => default;

    public ValueTask<object?> InterceptResponse(Api.FSharp.Database database, object request, object response, CancellationToken c)
    {
        if (request is not CreateTableRequest req || response is not CreateTableResponse resp)
            return default;

        // modify the output
        resp.TableDescription.BillingModeSummary = new BillingModeSummary
        {
            BillingMode = req.BillingMode ?? BillingMode.PAY_PER_REQUEST,
            LastUpdateToPayPerRequestDateTime = DateTime.UtcNow
        };

        // Return default so that the response will be passed on after modification
        // If a non null item is returned here, it will be passed on instead 
        return default;
    }
}

Logging

Logging is implemented by Microsoft.Extensions.Logging.ILogger. Databases can be created with loggers. Clients can also be created with loggers, which will override the datase logging