JsonPathPlus 1.9.0

JsonPathPlus

Lightweight JSONPath-like Stream extensions for System.Text.Json.

Extracts JSON subtrees from a Stream by path expression. Uses streaming selectors for root arrays and root objects, with automatic full-parse fallback when a selector requires full-document evaluation.

Memory-efficient array filtering

Stream inputs are read incrementally, including arrays nested below property selectors such as $.data.items[?(@.score >= 750)].name. Filters evaluate a read-only JsonElement for small candidates; large candidates retain only filter dependencies while their encoded bytes are captured for replay. Only returned values are materialized as JsonNode. Results are yielded as they are found, without collecting the complete result set. Non-seekable streams replay only their small inspected prefix, not a copy of the input.

For these selectors, working memory depends on fixed-size read blocks, the largest individual JSON token, filter operand values, and the selected result, rather than the entire array. Negative/computed indices, ordered nested unions, outer filters and nested queries use bounded replay. Seekable sources replay slices of the original input; non-seekable sources spill captures above 32 KiB to a private temporary file, deleted on disposal, cancellation or error. This trades extra passes and disk I/O for lower RAM. Recursive property searches and field projections traverse containers incrementally. Nested unions preserve requested order and duplicates; existing nonnegative root unions retain document order. Whole-value comparisons and selecting an entire document, object or array still require materializing that value.

Stream overloads accept JsonPathExtractionOptions with MaxDepth (zero preserves the default depth 64), MaxTokenBytes and MaxCapturedValueBytes. Limits are opt-in and fail explicitly; FullParseMaxBytes continues to apply only to full-document fallback, not to streamable recursive or negative-index selectors. Temporary-storage and I/O errors propagate to the caller. Do not modify a seekable source while enumerating it.

To keep memory low, pass a file/network stream directly, process results with await foreach rather than accumulating them, and use ExtractFirstJsonMatchAsync when only one result is needed. String inputs still require a UTF-8 encoding buffer. For an existing JsonNode, results are traversed lazily but the original DOM remains in memory.

Stopping enumeration early stops further parsing; malformed JSON after the consumed portion is not guaranteed to be detected. Seekable input positions are restored and caller-owned streams remain open, including after early disposal or cancellation.

The allocation benchmarks can be reproduced with:

dotnet run --project tests\JsonPathPlus.Benchmarks\JsonPathPlus.Benchmarks.csproj --framework net8.0 --configuration Release -- --filter "*JsonPathFilterAllocationBenchmarks*" "*JsonPathStreamingMemoryBenchmarks*"

In a .NET 8.0.28 ShortRun, the pre-parsed filter benchmark allocated 1,360 bytes for the first match and 1,528 bytes for all matches. Stream first-match allocations were unchanged between 1,000 and 100,000 elements: 2.81 KB for a seekable root array and 3.73 KB for a seekable nested array (2.98 KB and 3.90 KB for non-seekable inputs). These are managed allocations, not peak working set or pooled-buffer capacity. Enumerating an entire array still allocates per-candidate read-only documents, but does not retain all candidates or matches. Regression tests also verify that the first result requires reading no more than 32 KB of a much larger input.

Observed peak-memory results

Isolated .NET 8 Release probes compared the optimized reader with commit f8035c8. Each case ran in three fresh processes after a small-input warmup; the table shows medians. Inputs were non-seekable: either 100,000 small records (about 3.93 MB, including nested collections), or two records with an 8 MiB string in the rejected record. Searches below consume only the first selected result.

For extraction, observed managed peak is the larger of the managed-heap high-water sampled every 1 ms and the retained managed bytes measured at the first yield, relative to the post-warmup baseline. Sampling can miss short peaks; this is not an exact profiler maximum, a process working-set measurement, or a universal RAM limit. Result counts, numeric checksums, final values and returned paths were checked against the baseline.

The ten previously buffered, small-output search cases exceeded an 80% reduction. This does not apply to every workload: already-streaming full-array enumeration and deeply nested path enumeration reduced observed peak by only 2-5%. Whole-value results still require output-sized memory. Throughput also varies: full flat-filter enumeration improved from 524 to 377 ms, while returning 100,000 deeply nested paths slowed from 589 to 643 ms. These diagnostic timings include measurement overhead.

Streaming editing was separately compared with JsonNode.ParseAsync followed by the equivalent DOM transformation and serialization of all 100,000 records. Both implementations produced matching output lengths and full-output checksums. Using the same three-process median and 1 ms managed-heap sampling:

Lower peak does not imply fewer total allocations or faster execution. Callbacks still allocate a small DOM per selected value, and small inputs can be slower than whole-document editing. Temporary-file replay also trades RAM for I/O. Use BenchmarkDotNet to measure editing throughput and total managed allocations on the target machine; these are different metrics from the observed peaks above:

dotnet run --project tests\JsonPathPlus.Benchmarks\JsonPathPlus.Benchmarks.csproj --framework net8.0 --configuration Release -- --filter "*JsonPathEditingMemoryBenchmarks*"

Installation

dotnet add package JsonPathPlus

Quick start

using JsonPathPlus;

// First match
JsonNode? node = await stream.ExtractFirstJsonMatchAsync("$.document_data.items[0].seal");

// All matches
await foreach (var item in stream.ExtractAllJsonMatchesAsync("$.items[*].id"))
{
    Console.WriteLine(item);
}

// Union indices (multiple array elements)
await foreach (var item in stream.ExtractAllJsonMatchesAsync("$.items[0,2,4]"))
{
    Console.WriteLine(item);
}

// Union properties (multiple object fields)
JsonNode? selected = await stream.ExtractFirstJsonMatchAsync("$.book[title,author]");
Console.WriteLine(selected);

Streaming JSON editing and projection write-back

Read a document once and write the modified JSON to a different stream:

using JsonPathPlus;
using System.Text.Json.Nodes;

long edited = await source.EditJsonAsync(destination, "$.items[?(@.id == 42)]", node =>
{
    JsonObject item = node!.AsObject();
    item["name"] = "Updated";
    item["enabled"] = true;
    item["tags"]!.AsArray().Add("reviewed");
    item.Remove("obsolete");
    return item;
}, new JsonPathEditingOptions { MaxMaterializedValueBytes = 64 * 1024 });

The callback receives only a selected value, not a DOM for the entire source. Use small selectors and MaxMaterializedValueBytes to bound callback memory. Selecting $ for a callback explicitly selects and materializes the whole document; use direct set/remove/projection operations instead when that is unacceptable. Returning null writes JSON null; it does not delete the match.

These operations never materialize the original selected container:

await source.SetJsonValueAsync(destination, "$.items[?(@.id == 42)].name", JsonValue.Create("Updated"));
await source.RemoveJsonMatchesAsync(destination, "$.items[?(@.disabled == true)]");

// Keep only id/name in each item, but preserve meta and every other nonselected branch.
await source.ProjectJsonAsync(destination, "$.items[id,name]");
// Or exclude large payload fields without building objects/arrays in memory.
await source.ProjectJsonAsync(destination, "$.items[!payload]");

Each example is a separate rewrite: rewind/reopen the source and use a fresh destination. The JSONPath addresses existing values; set does not create missing paths. Editing accepts property/index/range/union/wildcard/filter/recursive/computed-index selectors. Computed extraction results (schema, counts, nested queries, intermediate projections) are not writable locations and are rejected before output is written. Project paths must end with [field,...] or [!field,...].

Matches are changed once in source order, using original array indices. Duplicate union selectors do not apply the same edit twice. Selecting an ancestor suppresses edits to its descendants. Projection preserves source field order; selected arrays project direct object items and leave nulls/scalars/nested arrays intact. Empty projected objects remain {}. Return values count selected source locations, not whether the replacement differs from the original.

Save into the same stream

await using var file = new FileStream("data.json", FileMode.Open, FileAccess.ReadWrite,
    FileShare.None, 16384, FileOptions.Asynchronous);

await file.SetJsonValueInPlaceAsync("$.items[-1].name", JsonValue.Create("Last item"));
await file.ProjectJsonInPlaceAsync("$.items[id,name]");
await file.RemoveJsonMatchesInPlaceAsync("$.obsolete");
await file.EditJsonInPlaceAsync("$.meta", node =>
{
    node!["updated"] = true;
    return node;
});

An arbitrary length-changing JSON edit cannot safely overwrite unread source bytes. In-place methods first prepare the result in an automatically deleted temporary file, then overwrite the original stream and truncate stale trailing bytes. No document DOM or document-sized RAM staging buffer is used. The stream must support reading, writing, seeking and resizing; a fixed-capacity MemoryStream cannot grow.

Preparation errors, malformed input, callback errors and cancellation leave the source unchanged. Final overwrite is not atomic: a storage/write failure can leave partial data. Cancellation is checked before committing and is not observed during the final overwrite. For crash-safe file updates, write to a different file and use the platform's atomic file-replacement mechanism instead.

All methods leave caller streams open. Seekable source positions are restored. In-place writes replace the JSON from the current position through the end, preserving any preceding bytes. Separate-output methods append at the destination's current position without truncating it; use an empty destination. Input and output must not share underlying storage. Separate-output failures may leave partial output.

Unmodified subtrees are copied as encoded JSON; changed ancestor containers may have different whitespace/escaping. Memory depends on read/write blocks, the largest JSON token, filter operands and any callback/replacement values. Large replay captures use the same bounded temporary storage as extraction. Disk I/O, depth and size-limit errors propagate to the caller. Configure JsonPathEditingOptions.ReadOptions for reader limits.

NuGet publishing (GitHub only)

NuGet publishing is automated in GitHub Actions and does not require local package push commands.

Packages are published to both nuget.org and GitHub Packages.

Workflows

• Build validation: .github/workflows/build.yml
• NuGet publish: .github/workflows/publish-nuget.yml

Required GitHub secret

Add this repository secret before publishing:

• NUGET_API_KEY: API key from nuget.org with push permission for JsonPathPlus

Release branch format

Publishing is triggered when you push a branch with this exact pattern:

• release_1.0.0
• release_1.0.1

The version is taken from the branch name after release_. Only semantic versions in major.minor.patch format are accepted.

Publish flow

When a release branch is pushed, .github/workflows/publish-nuget.yml performs:

1. Add GitHub Packages NuGet source (authenticated with GITHUB_TOKEN)
2. Restore
3. Build (Release) with /p:Version from branch name
4. Test
5. Pack with /p:PackageVersion from branch name
6. Push package to nuget.org with --skip-duplicate
7. Push package to GitHub Packages with --skip-duplicate
8. Create a GitHub Release automatically with --generate-notes

Typical commands

git checkout -b release_1.0.0
git push -u origin release_1.0.0

Notes

• Local nupkg/snupkg outputs are ignored by git.
• If a branch name is invalid (for example release_1.0), the publish workflow fails fast with a validation error.
• If the same package version already exists on nuget.org, publish step is non-failing because --skip-duplicate is enabled.

Supported path syntax

API reference

// Stream input
Task<JsonNode?> ExtractFirstJsonMatchAsync(this Stream stream, string? selectToken, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonNode?> ExtractAllJsonMatchesAsync(this Stream stream, string? selectToken, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonPathMatch> ExtractAllJsonMatchesWithPathsAsync(this Stream stream, string? selectToken, CancellationToken cancellationToken = default)

// Stream input with extraction options
Task<JsonNode?> ExtractFirstJsonMatchAsync(this Stream stream, string? selectToken, JsonPathExtractionOptions options, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonNode?> ExtractAllJsonMatchesAsync(this Stream stream, string? selectToken, JsonPathExtractionOptions options, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonPathMatch> ExtractAllJsonMatchesWithPathsAsync(this Stream stream, string? selectToken, JsonPathExtractionOptions options, CancellationToken cancellationToken = default)

// JsonNode input
Task<JsonNode?> ExtractFirstJsonMatchAsync(this JsonNode? node, string? selectToken, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonNode?> ExtractAllJsonMatchesAsync(this JsonNode? node, string? selectToken, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonPathMatch> ExtractAllJsonMatchesWithPathsAsync(this JsonNode? node, string? selectToken, CancellationToken cancellationToken = default)

// Raw JSON string input
Task<JsonNode?> ExtractFirstJsonMatchAsync(this string json, string? selectToken, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonNode?> ExtractAllJsonMatchesAsync(this string json, string? selectToken, CancellationToken cancellationToken = default)
IAsyncEnumerable<JsonPathMatch> ExtractAllJsonMatchesWithPathsAsync(this string json, string? selectToken, CancellationToken cancellationToken = default)

// JSON Schema extraction
Task<JsonNode?> ExtractJsonSchemaAsync(this Stream stream, string? selectToken, CancellationToken cancellationToken = default)
Task<JsonNode?> ExtractJsonSchemaAsync(this Stream stream, string? selectToken, JsonPathExtractionOptions options, CancellationToken cancellationToken = default)
JsonNode? ExtractJsonSchema(this JsonNode? node, string? selectToken)
Task<JsonNode?> ExtractJsonSchemaAsync(this string json, string? selectToken, CancellationToken cancellationToken = default)

// JSON Schema generation options (control dialect, inference, formats)
// JsonPathSchemaGenerationOptions.FullInference (default, 2020-12)
// JsonPathSchemaGenerationOptions.Draft07Compatible

Passing null or "$" as selectToken returns the entire document.

All extraction APIs accept an optional CancellationToken.

JsonPathExtractionOptions.FullParseMaxBytes lets you cap fallback full-document parsing by stream size. Streaming selectors are still allowed under this cap. This is not a global memory limit: it does not cap a streamed candidate, a selected subtree, results retained by the caller, or buffers used by subtree fallback.

Path validation

Use JsonPathValidator to check a path expression before executing it:

// Quick boolean check
bool ok = JsonPathValidator.IsValid("$.items[?(@.price < 10)]");

// Full result with error message
JsonPathValidationResult result = JsonPathValidator.Validate("$.items[?()]");
if (!result.IsValid)
    Console.WriteLine(result.Error);
// → "Empty filter expression at position 7."

Validate detects the following structural errors:

null and empty strings are considered valid (they select the root document). Bracket characters inside quoted string literals in filter expressions are handled correctly and do not produce false positives.

Field projection and exclusion

Field inclusion projection selects only specified fields:

// Returns only title, author, isbn fields
await foreach (var book in stream.ExtractAllJsonMatchesAsync("$.books[*][title, author, isbn]"))
{
    Console.WriteLine(book);
}
// Output: { "title": "...", "author": "...", "isbn": "..." }

Field exclusion removes specified fields from result objects:

// Returns all fields except title and price
await foreach (var book in stream.ExtractAllJsonMatchesAsync("$.store.book[?(@.price > 2)][!title, !price]"))
{
    Console.WriteLine(book);
}
// Output: { "author": "J.R.R. Tolkien", "isbn": "978-0547928227", "category": "fiction" }

Both features work on individual objects and arrays of objects:

Behavior:

• Exclusion only applies to objects. Arrays and scalar values are returned unchanged.
• A field exclusion segment is applied after all previous path segments complete.
• Excluding all fields returns an empty object {}.

Nested query

Nested queries allow applying sub-paths to specific keys within a single bracket expression and combining the results into a result object:

// Select books under $20 with title/author, bicycle color, and the store name
await foreach (var result in stream.ExtractAllJsonMatchesAsync(
    "$.store[book[?(@.price < 20)][title, author], bicycle[color], name]"))
{
    Console.WriteLine(result);
}
// Output: { "name": "Bookstore", "bicycle": { "color": "red" }, "book": [{ "title": "Book A", "author": "Author 1" }, ...] }

Each key in the bracket expression can have its own sub-path (filters, projections, exclusions, etc.) applied independently:

Behavior:

• Keys that don't exist in the source object are omitted from the result.
• Sub-paths that match nothing cause the key to be omitted.
• A single sub-path match is kept as-is; multiple matches are collected into an array.
• Nested query is triggered when at least one branch has bracket-delimited sub-paths; otherwise, it falls through to existing segment types (field projection, property union, etc.).

JSON Schema extraction

Generate a JSON Schema (2020-12 / draft-07) from JSON data by path. The schema infers types, properties, constraints, formats, enums, and structure from the matched data. The generator follows the JSON Schema Core and JSON Schema Validation specifications.

// Get schema of the entire document (2020-12 by default)
JsonNode? schema = await stream.ExtractJsonSchemaAsync("$");

// Get schema of array items (schemas from all items are merged)
JsonNode? itemSchema = await stream.ExtractJsonSchemaAsync("$.items[*]");
// → { "$schema": "https://json-schema.org/draft/2020-12/schema",
//     "type": "object",
//     "properties": { "id": { "type": "integer", "minimum": 1, "maximum": 3 },
//                    "value": { "type": "string", "minLength": 1, "maxLength": 1 } },
//     "required": ["id", "value"],
//     "minProperties": 2, "maxProperties": 2 }

Using the schema() path function — generates a schema inline in the extraction pipeline:

// schema() as a path segment
JsonNode? schema = await stream.ExtractFirstJsonMatchAsync("$.items.schema()");

// With wildcards — each match gets its own schema
await foreach (var s in stream.ExtractAllJsonMatchesAsync("$.items[*].schema()"))
    Console.WriteLine(s);

Inferred Schema Keywords

The generator automatically infers the following keywords from data samples:

Supported Auto-Detected Formats

When InferFormat is enabled (default), the generator detects semantic formats in string values:

Configuration Options

Control schema generation behavior via JsonPathSchemaGenerationOptions:

var opts = new JsonPathSchemaGenerationOptions
{
    SchemaDialect = JsonSchemaDialect.Draft202012,  // or Draft07
    InferConstraints = true,    // min/max, lengths
    InferEnum = true,           // enum from distinct values
    InferConst = true,          // const from identical values
    InferFormat = true,         // auto-detect format
    InferPattern = true,        // auto-detect pattern (^\d+$, ^[a-zA-Z]+$, etc.)
    InferUniqueItems = true,    // uniqueItems for arrays
    MaxEnumValues = 10,         // max distinct values in enum
    StrictAdditionalProperties = false,   // additionalProperties: false
    StrictUnevaluatedProperties = false,  // unevaluatedProperties: false
    StrictUnevaluatedItems = false,       // unevaluatedItems: false
    InferDependentRequired = true,        // dependentRequired inference
    GenerateMetaData = false,   // title, description, examples
    HighConfidenceFormatOnly = true,  // only high-confidence format detection
    SchemaId = null,            // optional $id
    Comment = null,             // optional $comment
};

Pre-built presets:

• JsonPathSchemaGenerationOptions.FullInference — all inference enabled, 2020-12 dialect (default)
• JsonPathSchemaGenerationOptions.Draft07Compatible — minimal output, draft-07, no auto-inference

Schema merging behavior:

• Same-type values merge into a unified schema (e.g., multiple objects merge their properties)
• Mixed types produce a oneOf (2020-12) or anyOf (draft-07) union
• Required fields are those present in all merged object schemas
• Numeric ranges widen: min takes the smallest, max takes the largest
• String lengths widen: minLength takes the smallest, maxLength takes the largest
• dependentRequired is inferred from 100% co-occurrence of properties across schemas
• Enum is emitted when 2–N distinct const values are seen across merged schemas

Malformed path behavior

The parser is lenient and never throws on invalid or incomplete path expressions. The table below documents the defined behavior for common malformed inputs.

Guarantee: no malformed path will cause an infinite loop or an unhandled exception. At worst, the path is parsed partially and matched against whatever valid segments were extracted before the error.

Implementation roadmap

All planned roadmap phases in this README are implemented.

JSONPath ↔ XPath quick reference

The table below maps common XPath expressions to their JSONPath equivalents as a reference. Entries marked "not present in the original spec" are extensions defined by the JSONPath-Plus JavaScript library and are not guaranteed to be implemented in this library.

License

MIT