ND.FW.DataValidation 1.0.1

ND.FW.DataValidation

A configurable data-validation and metadata-derivation nuget, built on top of ND.FW.RulesEngine. Everything is driven by the same Rule JSON shape — no C# changes needed to add a validation rule, switch rule providers, or add a metadata-derivation stage.

Install

dotnet add package ND.FW.DataValidation

The two halves

Both read rules through the same IRuleRepository and the same Rule JSON shape — a rule's RuleExpressionType decides which half consumes it. Validation types (Required, Regex, NumberRange, DateValidation, Conditional, Lookup, Triangulation, FuzzyMatch, Enrichment, Standardization, Normalization) map to an ND.FW.RulesEngine workflow exactly as before. Metadata-derivation types (AttributeMapping, CandidateGrouping, MultiPageConsolidation, ConfidenceAggregation, CanonicalSelection, MultiValueConsolidation, ConflictDetection, CrossFieldValidation) drive the consolidation pipeline instead. You can mix both kinds of rules in the same rule set returned by a provider — each service simply filters for the types it understands.

Quick start

builder.Services.AddDataValidation(builder.Configuration, options =>
{
    // register any product-specific operators alongside the built-in base set
    options.CustomOperatorTypes.Add(typeof(MyProduct.Operators));
});

// appsettings.json
{
  "DataValidation": {
    "RuleRepository": {
      "UseDatabaseProvider": true,
      "UseLocalJsonProvider": true,
      "Mode": "Additive"
    },
    "Database": {
      "Schema": "dbo",
      "RuleDetailsSpName": "sp_Get_Rule_Details",
      "RuleJsonColumnName": "Rule_Definition_JSON",
      "ParameterMap": {
        "FileTaxonomyId": "@p_File_Taxonomy_Id",
        "DocTaxonomyId": "@p_Doc_Taxonomy_Id",
        "FileDocTaxonomyId": "@p_File_Doc_Taxonomy_Id",
        "FieldId": "@p_File_Taxonomy_X_Doc_Taxonomy_X_Field_Id",
        "RuleEventStage": "@p_Rule_Event_Stage"
      }
    },
    "LocalJson": {
      "RootDirectory": "RuleSets",
      "FileNameTemplate": "{DocTaxonomyId}.json"
    }
  }
}

Validation

var criteria = RuleQueryCriteria.FromKeys(
    ("FileTaxonomyId", 12),
    ("DocTaxonomyId", 4),
    ("FieldId", 101));

var input = new Dictionary<string, object?> { ["phone"] = "+919876543210", ["pan"] = "ABCDE1234F" };

var result = await dataValidationService.ValidateAsync(criteria, input);

foreach (var r in result.RuleResults)
    Console.WriteLine($"{r.RuleName}: {(r.IsSuccess ? "PASS" : $"FAIL - {r.ErrorMessage}")}");

Metadata derivation

var candidates = new List<AttributeCandidate>
{
    new() { AttributeName = "PT_NAME_1", Value = "John A Smith", Page = 1, Position = 0, ValueStrength = 0.81, Source = "ocr" },
    new() { AttributeName = "patient full name", Value = "John Smith", Page = 3, Position = 2, ValueStrength = 0.93, Source = "model-x" },
    new() { AttributeName = "PhoneNumber", Value = "555-0100", Page = 1, ValueStrength = 0.7 },
    new() { AttributeName = "PhoneNumber", Value = "555-0199", Page = 2, ValueStrength = 0.6 },
};

var result = await metadataDerivationService.DeriveAsync(criteria, candidates);

var patientName = result.Attributes["PatientName"]; // canonical value + evidence + confidence
foreach (var conflict in result.Conflicts)
    Console.WriteLine($"Conflict on {conflict.AttributeName}: {conflict.ConflictingValues.Count} candidate values");

Rule providers — multiple, at the same time

IRuleRepository has two built-in implementations:

• DatabaseRuleRepository — calls a stored procedure. SP name, schema, and the mapping from RuleQueryCriteria.Keys to SP parameter names are all configuration (DatabaseRuleProviderOptions), not hardcoded — so this provider works for any product's own key shape, not just taxonomy IDs.
• LocalJsonRuleRepository — reads rule JSON from local files (or inline configuration). Accepts a single Rule object, a Rule array, or a { "Rules": [...] } wrapper per file.

When both are enabled, CompositeRuleRepository merges their results:

• Additive (default) — rules from every enabled provider are combined, de-duplicated by RuleName (earliest-registered provider wins on a collision).
• Fallback — providers are tried in order; the first one that returns any rules wins and the rest are skipped.

Add a third provider (e.g. a remote config service) by implementing IRuleRepository and registering it — CompositeRuleRepository accepts any IEnumerable<IRuleRepository>.

Metadata-derivation pipeline

Stages run in a fixed logical order, but only the stages whose RuleExpressionType is present in the resolved rule set actually run — a rule set with just CandidateGrouping + CanonicalSelection rules skips multi-page consolidation and cross-field validation entirely.

AttributeMapping → CandidateGrouping → MultiPageConsolidation →
ConfidenceAggregation → CanonicalSelection → MultiValueConsolidation →
ConflictDetection → CrossFieldValidation

Note: ToCanonicalDictionary() exposes multi-value attributes (IsMultiValue: true) as a List<object?>, not a scalar. A CrossFieldValidation expression referencing such an attribute must treat it as a list (e.g. ((List<object?>)input1["PhoneNumber"]).Count > 0) rather than comparing it directly — ND.FW.RulesEngine passes the value through unchanged and does not coerce it to a string or number.

Every canonical attribute retains full evidence (EvidenceRef: value, page, position, source location, source, and computed weight) for every candidate that contributed to it, whether it won or lost — satisfying audit/traceability needs without any extra configuration.

Extending with a custom stage

Metadata-derivation stages are just another named plugin point, exactly like ND.FW.RulesEngine's INdContextPlugin:

public sealed class MyCustomStage : IMetadataDerivationStage
{
    public string RuleExpressionType => "MyCustomStageType";
    public Task ExecuteAsync(Rule rule, MetadataDerivationContext context, CancellationToken ct) { ... }
}

services.AddDataValidation(configuration, options =>
{
    options.MetadataDerivationStageTypes.Add(typeof(MyCustomStage));
});

Rule JSON compatibility

Existing rule rows/files are not restructured. Every new capability is added by:

1. New RuleExpressionType string values (data, not schema).
2. A small number of new, optional fields on RuleExpression — AttributeNameMap, GroupByKeys, ConfidenceWeights, ConflictResolutionStrategy, ConflictThreshold, IsMultiValue, MaxValues, PageOrderStrategy, CanonicalFields. All nullable; absent in existing rows and simply unused by validation rule types.
3. Reuse of one existing field, Condition, as the raw cross-field expression string for CrossFieldValidation rules (this field already existed on RuleExpression and was previously unused by any mapped type).

No existing field was renamed, retyped, or removed.

Packaging note

ND.FW.DataValidation references ND.FW.RDBM / Microsoft.Data.SqlClient directly so DatabaseRuleRepository is available out of the box, matching how DVS already depends on ND.FW.RDBM. If a consumer only ever wants the local-JSON provider (no database dependency at all — e.g. an edge/offline scenario), the cleaner long-term shape is to split this into ND.FW.DataValidation.Core (abstractions, local JSON, metadata derivation) + ND.FW.DataValidation.SqlServer (the database provider) so the SQL dependency is opt-in per package reference rather than always-transitive. That split is a bigger packaging change than this version takes on; call it out if a consumer without SQL Server ever needs this nuget.

License

Internal use.