Blazing.Json.Queryable 1.0.0

Blazing.Json.Queryable

Table of Contents

• Overview
• Key Features
• Why Choose Blazing.Json.Queryable?
• Getting Started
  • Installation
  • Requirements
  • Quick Start
• Usage
  • Basic Queries
  • Streaming Queries
  • Advanced Streaming with .NET 10 Async LINQ
  • UTF-8 Direct Processing
  • JSONPath Queries
• Supported LINQ Methods
  • Filtering Operations
  • Projection Operations
  • Ordering Operations
  • Quantifier Operations
  • Element Access Operations
  • Aggregation Operations
  • Set Operations
  • Partitioning Operations
  • Grouping Operations
  • Conversion Operations
  • Sequence Operations
• How It Works
• Performance Advantages
• Samples
• Give a ⭐
• Support
• History

Overview

Blazing.Json.Queryable is a high-performance LINQ provider for JSON data that processes JSON directly without full deserialization. Unlike traditional approaches that load entire JSON files into memory and then apply LINQ queries, this library processes JSON as a stream, providing dramatic performance improvements and memory efficiency for medium to large JSON files.

This custom JSON LINQ provider supports standard string, UTF-8, streaming, and advanced streaming JSONPath operations. Whether you're working with multi-megabyte API responses, large data exports, or log files, Blazing.Json.Queryable enables you to query JSON data efficiently with the familiar LINQ syntax you already know.

Key Features

• ✅ Direct JSON Processing: Query JSON without full deserialization
• ✅ Memory Efficient: Process files larger than available RAM
• ✅ Early Termination: Stop reading after finding required results (Take, First, Any)
• ✅ Streaming Support: Native IAsyncEnumerable<T> for real-time processing
• ✅ UTF-8 Native: Zero-allocation UTF-8 processing with Span<T>
• ✅ JSONPath Support: Navigate complex JSON structures efficiently
• ✅ Full LINQ: Comprehensive LINQ method support (60+ operations)
• ✅ .NET 10 Async LINQ: Built-in async LINQ with async predicates and transformations
• ✅ Multiple Input Formats: String, Stream, UTF-8 bytes, and more

Why Choose Blazing.Json.Queryable?

Performance Advantages Over Traditional JSON + LINQ

When working with medium to large JSON files, the traditional approach of deserializing to objects and then querying has significant disadvantages:

Traditional Approach:

// Traditional: Load ALL data into memory, then query
var json = File.ReadAllText("large-file.json");          // Load entire file
var all = JsonSerializer.Deserialize<List<Person>>(json); // Deserialize ALL records
var results = all.Where(p => p.Age > 25).Take(10).ToList(); // Query in-memory

Blazing.Json.Queryable Approach:

// Blazing.Json.Queryable: Stream and query simultaneously
await using var stream = File.OpenRead("large-file.json");
var results = await JsonQueryable<Person>.FromStream(stream)
    .Where(p => p.Age > 25)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync(); // Only deserializes matching records!

Key Advantages

When to Use Each Approach

✅ Use Blazing.Json.Queryable When:

• Files are > 10 MB
• You need early termination (Take, First, Any)
• Working in memory-constrained environments
• Processing files larger than available RAM
• Building async/streaming APIs
• Need real-time data processing

✅ Use Traditional JsonSerializer + LINQ When:

• Files are < 1 MB
• You need all data (no early termination)
• Simplicity is the priority
• Working with in-memory collections

Getting Started

Installation

Install via NuGet Package Manager:

<PackageReference Include="Blazing.Json.Queryable" Version="1.0.0" />

Or via the .NET CLI:

dotnet add package Blazing.Json.Queryable

Or via the Package Manager Console:

Install-Package Blazing.Json.Queryable

Requirements

• .NET 10.0 or later
• System.Text.Json (included with .NET)
• Utf8JsonAsyncStreamReader - Provides JSONPath support for token-based filtering (NuGet Package)
  • Automatically included as a dependency
  • Enables efficient JSONPath navigation: $.data[*], $.users[*].orders[*], etc.

Quick Start

using Blazing.Json.Queryable.Providers;

// From JSON string
var jsonString = """[{"Name":"Alice","Age":30},{"Name":"Bob","Age":25}]""";
var results = JsonQueryable<Person>.FromString(jsonString)
    .Where(p => p.Age > 25)
    .ToList();

// From file stream (memory-efficient)
await using var stream = File.OpenRead("data.json");
var streamResults = await JsonQueryable<Person>.FromStream(stream)
    .Where(p => p.Age > 25)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync();

// From UTF-8 bytes (zero-allocation)
byte[] utf8Json = Encoding.UTF8.GetBytes(jsonString);
var utf8Results = JsonQueryable<Person>.FromUtf8(utf8Json)
    .Where(p => p.Age > 25)
    .ToList();

public record Person(string Name, int Age);

Usage

Basic Queries

Standard LINQ operations work as expected:

var json = """
[
    {"Id":1,"Name":"Alice","Age":30,"City":"London","IsActive":true},
    {"Id":2,"Name":"Bob","Age":25,"City":"Paris","IsActive":true},
    {"Id":3,"Name":"Charlie","Age":35,"City":"London","IsActive":false}
]
""";

// Filtering
var adults = JsonQueryable<Person>.FromString(json)
    .Where(p => p.Age >= 18)
    .ToList();

// Projection
var names = JsonQueryable<Person>.FromString(json)
    .Select(p => p.Name)
    .ToList();

// Ordering
var sorted = JsonQueryable<Person>.FromString(json)
    .OrderBy(p => p.Age)
    .ThenBy(p => p.Name)
    .ToList();

// Aggregation
var avgAge = JsonQueryable<Person>.FromString(json)
    .Average(p => p.Age);

var totalActive = JsonQueryable<Person>.FromString(json)
    .Count(p => p.IsActive);

// Combined operations
var results = JsonQueryable<Person>.FromString(json)
    .Where(p => p.Age >= 25 && p.City == "London")
    .OrderByDescending(p => p.Age)
    .Select(p => new { p.Name, p.Age })
    .Take(5)
    .ToList();

Streaming Queries

Process large files with constant memory usage:

// Traditional approach - loads entire file into memory
var traditionalResults = new List<Person>();
var allJson = File.ReadAllText("huge-dataset.json"); // Loads ALL 500MB
var allPeople = JsonSerializer.Deserialize<List<Person>>(allJson); // Deserializes ALL
traditionalResults = allPeople.Where(p => p.Age > 25).Take(100).ToList();

// Blazing.Json.Queryable - streams and stops early
var streamingResults = new List<Person>();
await using (var stream = File.OpenRead("huge-dataset.json"))
{
    await foreach (var person in JsonQueryable<Person>.FromStream(stream)
        .Where(p => p.Age > 25)
        .Take(100) // Stops reading after 100 matches!
        .AsAsyncEnumerable())
    {
        streamingResults.Add(person);
    }
}
// Result: 10-20x faster, uses constant memory

Advanced Streaming with .NET 10 Async LINQ

.NET 10 includes built-in async LINQ support, enabling powerful async transformations:

await using var stream = File.OpenRead("data.json");

// Async predicates - call async methods in Where clauses
await foreach (var person in JsonQueryable<Person>.FromStream(stream)
    .AsAsyncEnumerable()
    .Where(async (p, ct) => await IsValidUserAsync(p, ct)) // Async predicate!
    .OrderBy(p => p.Name)
    .Take(100))
{
    Console.WriteLine(person.Name);
}

// Async transformations with Select
await foreach (var enriched in JsonQueryable<Person>.FromStream(stream)
    .AsAsyncEnumerable()
    .Select(async (p, ct) => await EnrichPersonDataAsync(p, ct)) // Async select!
    .Where(p => p.Score > 50))
{
    await ProcessAsync(enriched);
}

// Cancellation support
using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(5));

await foreach (var item in JsonQueryable<Person>.FromStream(stream)
    .AsAsyncEnumerable()
    .WithCancellation(cts.Token))
{
    // Automatically cancelled after 5 minutes
}

UTF-8 Direct Processing

Process UTF-8 bytes without string conversion overhead:

// From UTF-8 byte array
byte[] utf8Json = Encoding.UTF8.GetBytes(jsonString);
var results = JsonQueryable<Person>.FromUtf8(utf8Json)
    .Where(p => p.Age > 25)
    .ToList();

// From ReadOnlySpan<byte> (zero-allocation)
ReadOnlySpan<byte> utf8Span = utf8Json.AsSpan();
var spanResults = JsonQueryable<Person>.FromUtf8(utf8Span)
    .Where(p => p.Age > 25)
    .ToList();

// From UTF-8 stream (most efficient for large data)
await using var utf8Stream = File.OpenRead("data.json");
await foreach (var person in JsonQueryable<Person>.FromStream(utf8Stream)
    .AsAsyncEnumerable())
{
    // Process each person with zero string allocations
}

JSONPath Queries

Navigate complex nested JSON structures from API responses:

JSONPath Support: Powered by Utf8JsonAsyncStreamReader - a high-performance library for token-based JSON filtering with JSONPath expressions.

// Example: GitHub API-style response
using var httpClient = new HttpClient();
var apiResponse = await httpClient.GetStringAsync("https://api.example.com/v1/users/123");

Sample API response structure:

{
  "status": "success",
  "data": {
    "user": {
      "id": 123,
      "username": "alice_dev",
      "profile": {
        "fullName": "Alice Developer",
        "email": "alice@example.com",
        "address": {
          "city": "London",
          "country": "UK",
          "postalCode": "SW1A 1AA"
        },
        "metadata": {
          "joinDate": "2024-01-15",
          "verified": true
        }
      },
      "repositories": [
        {
          "id": 1001,
          "name": "awesome-project",
          "stars": 1250,
          "language": "C#",
          "isPrivate": false
        },
        {
          "id": 1002,
          "name": "secret-work",
          "stars": 45,
          "language": "TypeScript",
          "isPrivate": true
        }
      ],
        "organizations": [
          {
            "id": 5001,
            "name": "TechCorp",
            "role": "developer",
            "projects": [
              { "id": 101, "name": "Enterprise API", "budget": 500000.00 },
              { "id": 102, "name": "Mobile App", "budget": 250000.00 }
            ]
          }
        ]
      }
    }
  }
}

Query repositories directly using JSONPath:

var publicRepos = JsonQueryable<Repository>
    .FromString(apiResponse, "$.data.user.repositories[*]")
    .Where(r => !r.IsPrivate && r.Stars > 100)
    .OrderByDescending(r => r.Stars)
    .ToList();

Console.WriteLine($"Found {publicRepos.Count} popular public repositories");

// Query nested organization projects
var largeProjects = JsonQueryable<Project>
    .FromString(apiResponse, "$.data.user.organizations[*].projects[*]")
    .Where(p => p.Budget > 200000)
    .OrderByDescending(p => p.Budget)
    .ToList();

Console.WriteLine($"Found {largeProjects.Count} high-budget projects");

// Query user profile data
var verifiedUser = JsonQueryable<UserProfile>
    .FromString(apiResponse, "$.data.user.profile")
    .Where(p => p.Metadata.Verified && p.Address.Country == "UK")
    .FirstOrDefault();

if (verifiedUser != null)
{
    Console.WriteLine($"Verified user: {verifiedUser.FullName} from {verifiedUser.Address.City}");
}

// Model classes
public record Repository(int Id, string Name, int Stars, string Language, bool IsPrivate);
public record Project(int Id, string Name, decimal Budget);
public record UserProfile(string FullName, string Email, Address Address, Metadata Metadata);
public record Address(string City, string Country, string PostalCode);
public record Metadata(string JoinDate, bool Verified);

Stream large API responses efficiently

await using var stream = await httpClient.GetStreamAsync("https://api.example.com/v1/users?page=1&limit=10000");

await foreach (var repo in JsonQueryable<Repository>
    .FromStream(stream, "$.data.items[*].repositories[*]")
    .Where(r => r.Language == "C#")
    .Take(50)
    .AsAsyncEnumerable())
{
    Console.WriteLine($"C# Repository: {repo.Name} ({repo.Stars} stars)");
}

Supported LINQ Methods

Filtering Operations

Projection Operations

Ordering Operations

Quantifier Operations

Element Access Operations

Aggregation Operations

Set Operations

Partitioning Operations

Grouping Operations

Conversion Operations

Note: For async conversion operations, use .AsAsyncEnumerable() followed by .NET 10's built-in async LINQ methods:

• await query.AsAsyncEnumerable().ToListAsync() - Async conversion to List<T>
• await query.AsAsyncEnumerable().ToArrayAsync() - Async conversion to array
• await query.AsAsyncEnumerable().ToDictionaryAsync(...) - Async conversion to dictionary

Sequence Operations

How It Works

Blazing.Json.Queryable uses a custom LINQ provider that translates LINQ expressions into efficient JSON processing operations:

1. Expression Tree Analysis: LINQ queries are analyzed to build an execution plan
2. Streaming JSON Parser: Uses System.Text.Json.Utf8JsonReader for efficient parsing
3. JSONPath Navigation: Leverages Utf8JsonAsyncStreamReader for token-based filtering when JSONPath expressions are used
4. Lazy Evaluation: Only processes JSON elements needed for the query
5. Early Termination: Stops reading JSON as soon as query requirements are met
6. Zero-Allocation UTF-8: Direct UTF-8 processing using Span<byte> and ReadOnlySpan<byte>
7. Async Enumeration: Native IAsyncEnumerable<T> support for non-blocking I/O

Execution Flow Example

// Query: Find first 10 active users over 25 in London
var results = await JsonQueryable<Person>.FromStream(stream)
    .Where(p => p.Age > 25)
    .Where(p => p.City == "London")
    .Where(p => p.IsActive)
    .OrderBy(p => p.Name)
    .Take(10)
    .AsAsyncEnumerable()
    .ToListAsync();

// Internal execution:
// 1. Build execution plan from LINQ expression tree
// 2. Open JSON stream and start parsing
// 3. For each JSON object:
//    a. Check Age > 25 (filter early)
//    b. Check City == "London" (filter)
//    c. Check IsActive (filter)
//    d. If all pass, deserialize to Person object
//    e. Add to ordered buffer
// 4. Stop reading after 10th match (early termination!)
// 5. Return results without processing entire file

Performance Advantages

Real-World Performance Comparison

Scenario: 100,000 record JSON file (52MB), find first 10 matching records

Key Performance Benefits

1. Early Termination:

   • Traditional: Processes ALL 100K records
   • Blazing.Json.Queryable: Stops after finding 10 matches
   • Result: 10-20x faster for Take/First operations

2. Memory Efficiency:

   • Traditional: Loads 52MB file + 450MB objects
   • Blazing.Json.Queryable: Uses ~25MB constant memory
   • Result: Can process files larger than available RAM

3. Zero-Allocation UTF-8:

   • Traditional: UTF-8 → String → Object
   • Blazing.Json.Queryable: UTF-8 → Object (direct)
   • Result: Fewer allocations, less GC pressure

4. Async Streaming:

   • Traditional: Blocking I/O during file read
   • Blazing.Json.Queryable: Non-blocking async enumeration
   • Result: Better scalability in web APIs

When Performance Gains Are Largest

• Large files (> 10MB): Bigger file = larger advantage
• Early termination (Take/First): Smaller N = larger speedup
• Complex filters: More filtering = more benefit from stream processing
• Memory constraints: Limited RAM = Blazing.Json.Queryable enables processing that's impossible traditionally

Samples

The library includes comprehensive sample projects demonstrating different usage patterns:

Sample Projects

All samples are located in the samples/Blazing.Json.Queryable.Samples directory:

• BasicQueries.cs - Fundamental LINQ operations (Where, Select, OrderBy, Take, Skip, First, Count, Any)
• AdvancedLinqOperationsSamples.cs - Advanced operations (Zip, Chunk, DistinctBy, ExceptBy, IntersectBy, UnionBy)
• StreamQueries.cs - Async streaming with IAsyncEnumerable<T>
• Utf8Queries.cs - Zero-allocation UTF-8 processing examples
• JsonPathSamples.cs - Complex JSONPath navigation
• ComplexGroupingSamples.cs - GroupBy, Join, and GroupJoin operations
• LargeDatasetSamples.cs - Memory-efficient large file processing
• PerformanceComparison.cs - Benchmarks comparing traditional vs streaming approaches
• AsyncQueries.cs - .NET 10 async LINQ with async predicates
• AdvancedScenarios.cs - Real-world patterns and best practices

Running the Samples

# Clone the repository
git clone https://github.com/gragra33/Blazing.Json.Queryable.git
cd Blazing.Json.Queryable

# Run the samples project
dotnet run --project samples/Blazing.Json.Queryable.Samples

The samples include:

• Interactive console menu
• Performance benchmarks with timing
• Memory usage comparisons
• Real-world scenarios
• Best practices demonstrations

Give a ⭐

If you like or are using this project to learn or start your solution, please give it a star. Thanks!

Also, if you find this library useful and you're feeling really generous, please consider buying me a coffee ☕.

Support

• Documentation: Full API documentation included in NuGet package
• Samples: Comprehensive samples in the repository
• Issues: Report bugs or request features on GitHub Issues
• Discussions: Ask questions on GitHub Discussions

History

V1.0.0 - 11 January, 2026

Initial Release - Full production release

• Core Features:

  • Custom LINQ provider for JSON with 60+ LINQ methods
  • Direct JSON processing without full deserialization
  • Early termination support for Take, First, Any operations
  • Memory-efficient streaming for files larger than RAM

• Input Format Support:

  • Standard JSON strings
  • UTF-8 byte arrays and spans (zero-allocation)
  • File streams with async enumeration
  • Advanced streaming with IAsyncEnumerable<T>

• LINQ Operations:

  • Filtering: Where, OfType
  • Projection: Select, SelectMany
  • Ordering: OrderBy, OrderByDescending, ThenBy, ThenByDescending, Reverse
  • Quantifiers: All, Any, Contains, SequenceEqual
  • Element Access: First, FirstOrDefault, Last, LastOrDefault, Single, SingleOrDefault, ElementAt, ElementAtOrDefault
  • Aggregation: Count, LongCount, Sum, Average, Min, Max, MinBy, MaxBy, Aggregate
  • Set Operations: Distinct, DistinctBy, Union, UnionBy, Intersect, IntersectBy, Except, ExceptBy
  • Partitioning: Take, TakeLast, TakeWhile, Skip, SkipLast, SkipWhile, Chunk
  • Grouping: GroupBy, GroupJoin, Join
  • Conversion: ToList, ToArray, ToDictionary, ToHashSet, ToLookup, Cast, AsEnumerable, AsQueryable, AsAsyncEnumerable
  • Sequence: Concat, Append, Prepend, Zip, DefaultIfEmpty

• .NET 10 Features:

  • Built-in async LINQ support with async predicates
  • Async transformations with Select
  • Native cancellation token support
  • Enhanced performance with latest runtime optimizations

• Performance Optimizations:

  • Zero-allocation UTF-8 processing with Span<T>
  • Early termination (10-20x faster for Take/First on large files)
  • Constant memory usage regardless of file size
  • Efficient execution plan optimization

• Advanced Capabilities:

  • JSONPath support for nested JSON navigation
  • Complex query translation and optimization
  • Async streaming with proper cancellation

• Documentation:

  • Comprehensive README with examples
  • Full XML documentation in NuGet package
  • sample project with 10+ demonstrations of all features
  • Performance comparison benchmarks

• Quality Assurance:

  • Comprehensive test coverage
  • Real-world performance benchmarks
  • Production-ready error handling
  • Best practices implementation

License: MIT License - see LICENSE file for details

Copyright © 2026 Graeme Grant. All rights reserved.