Etymon.Schema 0.1.0-preview.17

Etymon.Schema

Define the type once. Derive everything else.

Part of the Etymon suite. Depends on Etymon.Core and Etymon.Base, and on no NuGet package at all beyond FSharp.Core — System.Text.Json ships in the shared framework for both target frameworks, so it is an in-box API here rather than a dependency you inherit.

Do not install this for the JSON

F# already has good JSON libraries. Thoth.Json's decoder combinators look a great deal like Schema.object, Fleece and Chiron are mature, and FSharp.SystemTextJson handles records and unions with almost no ceremony. Nobody should switch libraries for the codec alone, and this package does not ask you to.

The reason to write a Schema<'T> is what else comes out of it. One constraint declaration —

let ageSchema = Schema.int |> Schema.constrain (Check.intRange (Some 0) (Some 130))

— reaches five places, and none of them restate it:

Schema.fromJson personSchema payload    // "age: must be between 0 and 130"
OpenApi.toJsonSchema personSchema       // "minimum": 0, "maximum": 130
Mapping.tableOf options personSchema    // CHECK ("age" >= 0 AND "age" <= 130)
Generate.valid personSchema             // only ever generates ages in range
Config.load personSchema sources        // "age: expected an integer (from environment)"

If you only want the JSON, use Thoth. If you want the OpenAPI document, the SQL and the generators to be incapable of disagreeing with the decoder, that is what this is for.

One advantage does hold even ignoring all that: a Schema<'T> is a single value, not a separate encoder and decoder that can drift apart. There is no way to change how a value is written without changing how it is read.

The idea

A Schema<'T> is one value that knows how a type is encoded, decoded, validated and documented. You write it once:

type Person = { Name: string; Age: int; Email: Email option; Address: Address }

let personSchema =
    Schema.object "Person" {
        let! name = Schema.required "name" (Schema.string |> Schema.constrain Check.nonEmpty) (fun p -> p.Name)
        and! age = Schema.required "age" ageSchema (fun p -> p.Age)
        and! email = Schema.optional "email" Email.schema (fun p -> p.Email)
        and! address = Schema.required "address" Address.schema (fun p -> p.Address)
        return { Name = name; Age = age; Email = email; Address = address }
    }

and get all of it:

Schema.toJson personSchema person      // encode
Schema.fromJson personSchema payload   // decode, reporting every error at once
Schema.validate personSchema person    // check the rules, no JSON involved
Schema.info personSchema               // the description every other package reads

Errors accumulate, and say where

Schema.fromJson personSchema """{"age":-3,"address":{"zip":"xx"}}"""

name: is required
age: must be between 0 and 130
address.street: is required
address.zip: must be exactly 5 characters
address.zip: must match ^\d+$

Five problems, five errors, each with a path. Nothing stops at the first. A bad element in a list reports its index — items[3].name — and a bad field inside a union case carries its whole path through the recursion.

Constrained types come with their rules

A refinement from Etymon.Core lifts straight in, and brings its constraints:

let emailSchema = Schema.string |> Schema.refine Email.refinement

SchemaInfo.constraints emailSchema.Info
// [ Length(Some 3, Some 254); HasFormat Email ]

That list is the point of the whole suite. Etymon.Schema.OpenApi turns it into JSON Schema keywords, Etymon.Schema.Sql into SQL constraints, Etymon.Invariants.FsCheck into generators — every one of them reading the same declaration rather than restating it.

A value carried as one of a fixed set of codes — a union stored as "accountant" — is a constrained string with a conversion on top:

Schema.string
|> Schema.constrain (Check.oneOf [ "accountant"; "bookkeeper" ])
|> Schema.convert parse render

The constraint runs before the conversion, so parse only ever sees a code it knows, and the code set is published as an enum in every derivation.

What it covers

Primitives (string, int, int64, float, decimal, bool, guid, dateTimeOffset, dateOnly, timeOnly, timeSpan, bytes, raw), nullable, list, array, map, objects, tagged unions, and recursive for a type that contains itself.

Optional and nullable are different things, because JSON Schema, OpenAPI and TypeScript all treat them differently. Schema.optional means the key may be absent; Schema.nullable means the value may be null.

Unions say how the union is written. Schema.union is adjacently tagged — { "kind": "circle", "value": 1 } — which works whatever shape the payload has, including a bare number, and the tag can never collide with a payload field. Schema.unionWith (UnionShape.InternalTag "type") writes the tag first and the payload's own fields beside the tag — { "type": "contractor_bill", "billNumber": "KT-32" } — for bytes already written that way; every case then carries an object payload or none, with no field named like the tag, and the schema refuses anything else when the schema is built (a payload held by reference is checked on the first write). The shape is part of the description, so the OpenAPI document, the TypeScript output, the generator and the compatibility snapshot all read the shape.

Recursion needs Schema.recursive. Without it, building a self-referential schema overflows the stack while it is being constructed, rather than failing in a way that tells you what is wrong.

Two decode modes

Strict is for JSON that arrived as JSON: a string where a number was expected is a mistake by whoever sent it.

Coercing is for the places where everything is a string whether it wants to be or not — environment variables, query strings, form posts:

Schema.fromJson Schema.int "\"8080\""                          // Error
Schema.fromJsonWith DecodeMode.Coercing Schema.int "\"8080\""  // Ok 8080

Etymon.Config reads in coercing mode. Coercion never makes nonsense acceptable: "eighty" is still not a number.

Adopting this over records that came off the wire

If your request and response types are currently bound by System.Text.Json, they are nullable throughout — string | null, Nullable<int>, 'a[] | null — because a decoder that cannot refuse has to put something in every field. And System.Text.Json writes every field, null included: {"name":"x","note":null}, never {"name":"x"}. A Schema speaks in option, so every field crosses.

One rule makes the conversion mechanical: take the record at its word. A field the record declares nullable is present and nullable in the Schema — always written, null when there is nothing, which is what System.Text.Json writes and what every client of yours has been reading. A field it declares non-nullable is Schema.required. A key that may genuinely be absent is Schema.optional, and that is a different statement, about the contract rather than the record.

Without that rule the first question on every single type is "should this be required?", the answer is a judgement each time, and it drifts across a codebase. With it, the conversion is transcription.

WireField crosses in both directions. The getter takes the nullable form, and the value bound in the expression is the nullable form too — so the record is rebuilt by naming its fields, with no Option.toObj at any of them:

let createBill =
    Schema.object "CreateBillRequest" {
        let! billNumber = WireField.text "billNumber" (fun r -> r.BillNumber)
        and! vendorId   = WireField.guid "vendorId" (fun r -> r.VendorId)
        and! lines      = WireField.array "lines" lineSchema (fun r -> r.Lines)
        return { BillNumber = billNumber; VendorId = vendorId; Lines = lines }
    }

The signatures are annotated for F# nullness, so this compiles unchanged in a project with <Nullable>enable</Nullable> — which is the audience the module exists for.

What a WireField field says, in all three places. It encodes null as "x": null, not as an absent key. It is described as required with a nullable type — ["string", "null"], in required — so a generated client types it string | null and receives exactly that, and the compatibility snapshot records it as present, so removing it later is reported as the break it is. It decodes absent and null the same way, as null, because in practice callers mean the same thing by both. It is built on Schema.present, which is the same statement for a record that already speaks in option. WireField.nullable and Schema.nullable therefore mean the same thing: a value that may be null. Schema.optional is the one that means a key that may be missing.

Constraints still apply through the crossing: WireField.constrainedText takes a ConstraintCheck and enforces it, which is the point of going through a Schema at all. For anything the typed members do not cover — a nested object that may be null, a string carrying Schema.sensitive, a Schema.raw payload — WireField.nullable takes any Schema, and the typed members are specialisations of it:

WireField.nullable "cadence"  cadenceSchema                          (fun r -> r.Cadence)
WireField.nullable "password" (Schema.string |> Schema.sensitive)    (fun r -> r.Password)

A request body is the other direction. WireField is what you write. What you accept is read leniently — absent and null are the same thing — so the honest description of the same field is optional and nullable. That is RequestField: the same members, the same signatures, the same crossing, and a description that differs in one word. A request file uses one module, a response file the other, and a DTO shared by both is described twice, once each way, so that neither direction is the other's misuse:

let createBillRequest =                                       // what you accept
    Schema.object "CreateBillRequest" {
        let! billNumber = RequestField.text "billNumber" (fun r -> r.BillNumber)
        and! lines      = RequestField.array "lines" lineSchema (fun r -> r.Lines)
        return { BillNumber = billNumber; Lines = lines }
    }

Coming in, billNumber is optional and nullable, with "default": null in the document; going out through WireField it is required and nullable. A collection coming in additionally admits null, because a request may send it; going out it never does. RequestField.requiredArray and requiredDictionary are WireField's, because required is required in both directions.

Reach for WireField because a field is nullable, never because it is a collection. WireField.array and WireField.dictionary take the nullable collection a deserialiser leaves and hand back a non-null one, so no call site needs a guard: a null collection and an absent key both read as empty, and empty is what is written for null. They are described as required, because that is what is written. WireField.requiredArray and WireField.requiredDictionary are the same crossing for a field the record declares non-nullable, and they refuse an absent key rather than read it as empty. Where the difference between absent and empty genuinely carries meaning, say so with Schema.optional and a nullable element type.

WireField.array         "lines"      lineSchema    (fun r -> r.Lines)       // 'F[] | null
WireField.requiredArray "roles"      Schema.string (fun p -> p.Roles)       // 'F[]

WireField.dictionary takes IDictionary<string, 'V> and hands back a Dictionary<string, 'V>: the interface going in so either field type satisfies it, the concrete type coming back so it assigns without a cast.

Design notes

Encoding cannot fail; decoding accumulates. A value of type 'T already is whatever its type says, so Schema.toJson is total. Decoding reads a JsonElement rather than a Utf8JsonReader — a forward-only reader cannot collect errors from fields it has already passed, and in F# it cannot be captured by a closure at all, which is what a combinator library is made of. So encoding streams and decoding buffers, deliberately.

No reflection. Every combinator is explicit, which means the whole package is trimming- and AOT-friendly. Schema.auto will be a labelled convenience, marked RequiresUnreferencedCode, never the default path.

Fields are written in declaration order, and maps in key order, so encoding the same value twice produces identical bytes. That is what makes a snapshot test worth having.

The default writer escapes HTML-sensitive characters, so a + in a timestamp offset comes out as +. Valid JSON, reads back identically, occasionally startling. Use Schema.toJsonWith with JavaScriptEncoder.UnsafeRelaxedJsonEscaping if the output is for a person and never for a web page. UTC timestamps render as Z, so the common case is clean.

Public surface

The complete public surface of this package, every value with its full signature and its documentation, is in Surface.fsi beside this file. It is written by the build from the implementation, so it is where to learn what a function hands back without compiling anything.

Licence

MIT.