PhoenixmlDb.XQuery 3.0.0

• net10.0

  • Antlr4.Runtime.Standard (>= 4.13.1)
  • Lucene.Net (>= 4.8.0-beta00016)
  • Lucene.Net.Analysis.Common (>= 4.8.0-beta00016)
  • PhoenixmlDb.Core (>= 2.6.0)

• net8.0

  • Antlr4.Runtime.Standard (>= 4.13.1)
  • Lucene.Net (>= 4.8.0-beta00016)
  • Lucene.Net.Analysis.Common (>= 4.8.0-beta00016)
  • PhoenixmlDb.Core (>= 2.6.0)

XPath mode, schema-aware function results and types, bounded schema fetches, tail calls, and a
large drop in QT3 failures. QT3: 31 failing of 31,325 (2.9.0: 207 of 31,331). The count includes
corrections to the test harness as well as engine changes, and 5 Unicode cases that depend on
another Unicode version are skipped. Requires PhoenixmlDb.Core 2.6.0.

### Security: deeply nested or chained expressions with untrusted queries (GHSA-9qvr-vjmg-m3pf)

Only hosts that compile or evaluate queries from untrusted parties are affected. A deeply nested
or long chained expression compiles and evaluates within the thread's stack and in linear time, or
raises XPDY0130. In browser WebAssembly and WASI, which cannot start a thread, a deep evaluation
continues on the current stack as in 2.9.0, and the workarounds of the advisory still apply. See
[GHSA-9qvr-vjmg-m3pf](https://github.com/phoenixmldb/phoenixmldb-xquery/security/advisories/GHSA-9qvr-vjmg-m3pf)
for the details and workarounds.

### Breaking — read this before upgrading

No public type or signature is removed or changed. These behaviour changes can affect a host:

- **`E => f(A)` parses to the function call `f(E, A)`**: a `FunctionCallExpression`, or a
 `DynamicFunctionCallExpression` when `f` is a variable or an expression. The parser makes an
 `ArrowExpression` only for the thin arrow `->`, and its `IsThinArrow` is always true. Code that
 walks the AST and looks for `ArrowExpression` to find `=>` must look for the call.
- **XPDY0130 for an expression nested too deeply for the thread's stack.** A caller can catch it.
 A deep chain of arithmetic, comparison, `and`, `or` or `||` operators does not raise it: its
 evaluation continues on a new stack. Chains of predicates, `!` maps, unary operators, lookups
 and FLWOR clauses can raise it on a small stack. The parser needs a stack of at least 768 KB.
- **`CostModel.EstimateCardinality` keeps each result per `CostModel` instance.** Do not use one
 instance from more than one thread at the same time. A change in `IContainerStatistics` after
 the first estimate does not change the kept results. The library makes a new `CostModel` for
 each query plan.
- **`XQueryExpressionRewriter.Rewrite` and `XQueryExpressionWalker.Walk` can continue on another
 thread when the stack is low.** A subclass that keeps per-thread state (`[ThreadStatic]`) in its
 `Visit` methods can see a different thread during a deep walk.
- **A nested `load-xquery-module` (depth more than 1) runs on a new 16 MB stack**, so the limit of
 64 nested loads gives FOQM0003 on any thread stack size.
- **FODC0005 for a colon in the first segment of a relative `fn:doc` reference** (#340), for
 example `doc('2026-01-02T10:30.xml')`. Such a name is not a valid relative URI. A stored
 document with such a name cannot be read with `doc('name')`; start the reference with `./`
 (`doc('./2026-01-02T10:30.xml')`).

### API

- **XPath mode: `CompilationOptions.XPath` and `XQueryParserFacade.XPath`** (#328). The text is
 parsed as an XPath 3.1 expression. XQuery-only syntax (prolog, FLWOR clauses beyond one `for` or
 `let`, positional and typed bindings, `switch`, `typeswitch`, `try`/`catch`, update expressions,
 `validate`, extension and ordered expressions, constructors, string constructors, annotations)
 is XPST0003. `<` is always an operator, string literals are verbatim, line endings are not
 normalized, and the namespace axis is available. The default is off, and XQuery behaviour does
 not change.
- `XdmSequenceType.OptionalNumeric` (#330).

### Schema fetches and locations

- **A schema fetch is bounded, and a timeout is XQST0059** (#348). With no `ResourcePolicy`, a
 schema location is read through the engine's HTTP client, which gives up on a connect after
 10 s. A timeout is reported as XQST0059 from `Compile`, and the schema provider is released.
- **A schema location is resolved as a URI** (#265): a location with a scheme that is not a URI is
 not read as a path, and a schema hint is not made absolute against the working directory.

### Deep trees

- **A query that builds a deeply nested tree does not end the process** (#301). A constructed
 element copies its descendants with an explicit stack. `fn:serialize`, the result serializer
 (xml, adaptive, json, html, xhtml), `fn:deep-equal`, `fn:base-uri` and copy-modify continue on a
 16 MB thread when the stack is short. `validate {}` over such a tree raises a catchable
 `InsufficientExecutionStackException`.

### Types and function conversion

- **`fn:analyze-string` results carry the types of their schema** (#345) when the schema provider
 can take the built-in F&O schema; otherwise they stay untyped. A carriage return in the text
 survives.
- **`element(*, xs:numeric)` and `attribute(*, xs:numeric)` match nodes of a numeric type** (#346).
- **`element(*, U)` is a subtype of `element(*, V)` in a function type only if U derives from V**
 (#342), when a schema provider is present.
- **A function result of a schema type casts an untyped value through the schema** (#343), so a
 union tries its members in order.
- **`fn:abs`, `ceiling`, `floor`, `round` and `round-half-to-even` return `xs:numeric?`** (#330).
 The two-argument forms take `xs:anyAtomicType?`, and the result keeps the argument's type.
- **Node arguments of atomic-typed parameters are atomized** (#306). `escape-html-uri`,
 `encode-for-uri` and `iri-to-uri` accept nodes; `escape-html-uri` and `iri-to-uri` encode by code
 point. `fn:id` splits each string value into IDREFs.
- **An xs:string parameter takes a string, an untyped value or an xs:anyURI** (#317, #332). A
 number, boolean, date, time, duration, g* value or QName is XPTY0004.
- **An untyped value that does not cast raises the cast's error** (#321): `local:f(<a/>)` for an
 xs:boolean parameter is FORG0001.
- **List-typed nodes** (#305): a list of xs:QName keeps each item's namespace, and a general
 comparison compares each item.
- **A string is whitespace-collapsed before a cast to a non-string type** (#307).
- **An integer wider than 64 bits casts to xs:double and xs:float** (#319).
- **A double zero keeps its sign, and an xs:decimal zero has none** (#323). The optimizer folds
 `0 * x` and `x + 0` only where the result is exact.
- **Value equality compares xs:QName by namespace URI** (#338): `index-of`, `distinct-values` and
 the other value-equality functions.
- **`fn:deep-equal` treats a one-item sequence member as its item** (#333).
- **A validated tree's string values leave out the dropped whitespace** (#331).

### Error codes

- XPST0008 for a prolog variable in its own initializer; a cycle through a function stays XQDY0054
 (#325).
- XPST0081 for an undeclared prefix in a literal computed-constructor name (#312) and in a declared
 variable's name (#337).
- XQST0040 for two direct attributes with one expanded name (#322).
- XQST0022 only for a real enclosed expression in a namespace declaration; `{{` and `}}` are
 literal braces (#311).
- XQST0034 for a function named like an imported schema type's constructor (#335).
- XQST0052 for a cast to a type outside the in-scope schema types, reported at compile time (#341).
- XQTY0105 for a function item in element or document content (#326).
- XPST0003 for `validate` or an extension expression at the start of a postfix, path or simple map
 expression without parentheses (#344). **Queries that write `validate { … }/@a` need parentheses.**
- FOAR0001 for `INF idiv 0` (#313); FOAR0002 for a decimal sum out of range in `fn:avg` (#324);
 FOAY0002 for a negative `array:subarray` length, with the end check in `long` (#316).
- FODC0005 for a colon in the first segment of a relative `fn:doc` reference (#340).
- FOJS0005 for `json-to-xml` with `validate` and `duplicates="retain"`; `duplicates` defaults to
 `reject` with `validate` (#334).
- FOUT1190 for undecodable `fn:unparsed-text` content with no encoding named, and XPTY0004 for a
 non-string encoding (#308).
- XPTY0004 for an atomic lookup node in `fn:id` and `fn:element-with-id` (#309).

### Other behaviour

- **`fn:normalize-unicode` supports `FULLY-NORMALIZED`** (#339).
- **A document node without exactly one root element serializes as a fragment** (#318):
 `document{<a/>, <b/>}`.
- **`format-number` in a library function uses that module's decimal format first** (#336).
- **Each closing brace matches the brace it closes** (#320), for an enclosed expression nested in
 `document { }` in element content.
- MessagePack 2.5.307 in the language server, for GHSA-qhrr-8q5h-9q3h (#272).

### Also in this release

- **A string walked by a recursive function is linear** (#296, with tail calls): an argument
 `substring($s, 2)` to a declared function whose parameter is untyped or `xs:string` is passed
 as a slice of the string, not a copy of its rest. A 50,000-character walk: 909 ms and 2.5 GB
 -> 73 ms and 133 MB. Tail calls are also open to a host's own functions (the XSLT engine's
 stylesheet functions); a call never crosses from one kind to the other.

- **Tail calls of declared functions** (#296). A call to a declared function in tail position of
 a declared function's body (the body; the branches of `if`, `typeswitch` and `switch`; the
 return of a FLWOR of `let` clauses only; an arrow call there) runs as the caller's next
 iteration instead of nested, when the callee has the caller's declared return type. Tail
 recursion, direct or mutual, is no longer limited to a depth of 1,000 (FOER0000): a million
 iterations run in about 1.4 s. Recursion whose result is used after the call keeps the limit.
 An infinite tail recursion now runs until it is cancelled, as a loop does. QT3
 numberformat121, numberformat122.

- **The descendant axes allocate one stack per walk, not one iterator per node** (xquery#294).
 `iso_svrl_for_xslt2` over NY.EMS.v350: 2,355 -> 1,696 MB, time 0.85x; output unchanged.

- **A query context can be reset and used again** (internal, for hosts): `ResetForReuse`
 returns it to the state of a new one, and a context an inline function item keeps is marked
 and never reset. The XSLT engine uses it to evaluate its XPath expressions without a new
 context each (xquery#294). No public API changes.

- **`if`, arithmetic and value comparisons are evaluated synchronously** where their operands
 allow it, so an expression containing one no longer leaves the synchronous path (each
 evaluation allocated an async iterator per operator). W3C evaluate-010, which evaluates
 `if (position() mod 7 = 0) then $a else $b` 100,000 times: 518 ms -> 350 ms warm, 1,403 MB
 -> 685 MB. Results unchanged.

- **Character map keys are one character, and are checked as the source requires.** From a map, a
 key that is not a string (an `xs:QName`) is XPTY0004 and a longer string SEPM0016; from an
 `output:serialization-parameters` element, a `character` that is not one character makes the
 element invalid, SEPM0017. A supplementary character (a surrogate pair) is one key, and is
 mapped. QT3 serialize-xml-023, -023b, -139, -139b.

- **A constructor function item carries its signature** (XPath 3.1 §3.14.2): `xs:T#1` is
 `function(xs:anyAtomicType?) as xs:T?`, and for a list type returns its item type's sequence
 (`xs:NMTOKENS#1` returns `xs:NMTOKEN*`; a schema list type of integers, `xs:integer*`). It was
 `function(item()?) as item()`, so `let $f as function(xs:anyAtomicType) as xs:NMTOKEN* :=
 xs:NMTOKENS#1` was refused. A list type is cast to only from one `xs:string` or
 `xs:untypedAtomic`: `xs:ENTITIES(xs:anyURI("abcd"))` and a sequence of two strings are
 XPTY0004. QT3 CastAs-ListType-17..19, -26, -27.

- **An unprefixed type name in `element()` and `attribute()` tests is in the default namespace for
 elements and types** (XQuery 3.1 §3.1.1), which `import schema default element namespace` and
 `declare default element namespace` set. It was read as an xs type: `element(*, restrictedType)`
 raised XPST0008, or as a step matched nothing. A built-in type name keeps reading as one. QT3
 ForExprType026..041, step-expr-si-1..3.

- **`fn:substring` of a string held as a slice costs no copy** (#287). An internal string slice
 (`StringSlice`) shares the characters of the string it was taken from; `fn:substring`,
 `fn:string-length` and `fn:starts-with` read one without making its string, and every other
 reader of a variable gets the string. Slices are made only by the XSLT engine, for a template
 parameter bound to `substring(...)` (see the XSLT release notes); a query never sees one. No
 public API changes.

- **`count` and `subsequence` of a range are computed from its bounds.** `count(1 to N)`,
 `count(subsequence(a to b, ...))` and `subsequence(a to b, ...)` no longer generate the integers
 before the slice, so `count(subsequence(1 to 3000000000, 2147483648))` returns 852516353 instead
 of stopping at the materialization limit (FOER0000). QT3 cbcl-subsequence-010..014.

- **A year outside 1..9999 survives `fn:dateTime` and the component functions.** `fn:dateTime`
 dropped the year it holds apart from the .NET date, so `-0012-12-03` combined into
 `0004-12-03`; and `year-from-date` and the other twelve component functions parsed an untyped
 argument, or a node, as a .NET date, which refused `-0012-12-03`. An untyped argument is now cast
 by the XSD type's own parser, and a node atomized first.

- **`schema-element(N)` follows XSD 1.1 substitution groups**: an element in several groups is a
 member of each (with PhoenixmlDb.Core's change), an abstract element is a member of none, and a
 nilled element matches only where the declaration of its name is nillable (XPath 3.1 §2.5.5.3).
 `ISchemaProvider` gains `IsNillableElementDeclaration`, with a default that says true. QT3
 substitution-020..025.

- **A kind test as the node test of a step is schema-aware**: `document-node(element(N))` and
 `document-node(schema-element(N))` test the document element (the inner test was not applied, so
 the step selected every document), and `element(*, T)` / `attribute(*, T)` match a validated node
 of type T or one derived from it (only the untyped types were known). The prefix of the element
 test inside `document-node()` is resolved as a step's is. W3C XSLT validation-1401.

- **An untyped element does not match `schema-element(N)`, nor an untyped attribute
 `schema-attribute(N)`**, even where N is declared with xs:anyType or xs:anySimpleType, from which
 the untyped types derive (spec bug 10065). They matched such a declaration. W3C XSLT
 validation-1001, with `input-type-annotations="strip"`.

- **`fn:deep-equal` of two validated elements respects their types' content** (F&O 3.1 §14.2.1):
 elements whose types have different content varieties (simple, element-only, mixed, empty) are
 not equal, and two of simple content compare their typed values. Their text and children were
 compared, so `xs:integer` 2012 equalled `xs:gYear` 2012. QT3 K2-SeqDeepEqualFunc-36, 38, 39, 42.

- **A cast to a schema list type whose item type is a union with an xs:QName member casts each
 token with the query's prefixes**: `'a b xs:integer' cast as s:sensitiveList` gives two NCNames and
 a QName. The list was kept whole, and a union with such a list as a member refused the value.
 `SchemaSimpleType` gains `ListItemType`. QT3 CastAs-UnionType-28..32, CastAs-ListType-21.
- **`Q{uri}local` names a type of the schema for that namespace**, as a prefix bound to it does. It
 was looked up by its local name among the built-in types, so `Q{urn:s}u` was "not a recognized
 type" and `Q{urn:s}string` was taken for xs:string.

- **A member type derives from a union only when the union is pure** (XPath 3.1 §2.5.2, §2.5.6.1):
 not from a union of list types, nor one with facets. An element of one list type matched
 `element(*, U)` for U the union of two list types. W3C XSLT match-197.

- **A node passed for a parameter of type `xs:anyAtomicType` is atomized** (XPath 3.1 §3.1.5.2): an
 element validated as xs:integer gives an xs:integer, an untyped one an xs:untypedAtomic. The
 coercion of arguments skipped that type, so the node itself was bound. QT3 FunctionCall-005/007.
- **An xs:untypedAtomic value for the xs:QName parameter of `fn:local-name-from-QName`,
 `fn:namespace-uri-from-QName` and `fn:prefix-from-QName` is XPTY0117**: it is not cast to a
 namespace-sensitive type. It was XPTY0004. QT3 FunctionCall-018/041, LocalNameFromQNameFunc015a,
 NamespaceURIFromQNameFunc015a.

- **`document-node(element(...))` and `document-node(schema-element(N))` test the document element
 as an item**: its namespace, its type, and the substitution group of N. The inner test of
 `document-node(schema-element(N))` was not applied, so every document matched it, and
 `document-node(element(N, T))` compared the local name of the element only. W3C XSLT match-185,
 with the XSLT change.
- **A prefix that the host binds (`CompilationOptions.StaticNamespaces`) is known while the query is
 parsed.** A name the parser resolves itself, as in `schema-element(j:array)`, was looked for in no
 namespace when only the host bound `j` (XPST0008). A prolog declaration of the prefix still wins.

- **An element validated against a type has that type when the schema declares its name with
 another type.** Validation by type does not use the element's declaration, but the annotating
 parse assessed the element by it and refused the type, so the element stayed untyped (an XSLT
 `<e xsl:type="xs:string">` of a declared `e`). W3C XSLT match-220 and match-221, with the XSLT change.
- **Arithmetic on an xs:boolean, xs:anyURI, xs:hexBinary or xs:base64Binary is XPTY0004.** A boolean
 was taken as 1 or 0 (`1 + true()` was 2), and the other three escaped as a .NET `FormatException`
 or `InvalidCastException`. XPath 1.0 compatibility mode still converts its operands to numbers.
- **The grammar parses the operators from `or` to `cast as` with two left-recursive rules** (#401),
 not one rule per precedence level. Each operand went through seventeen nested rules, and the
 parse tree had a node for each. For a generated Schematron validator of 2.3 MB, an XSLT compile
 now allocates 316 MB, not 399 MB (-21%), and takes about 315 ms, not 375 ms. Precedence, and the
 XPST0003 for a second comparison, range, cast, castable, treat or instance of on one operand
 without parentheses, are as before.
- **A parse-tree node's child list starts with room for one child** (#401). The grammar's precedence
 chain gives most nodes one child, and each list made an array of four. For a generated Schematron
 validator of 2.3 MB, an XSLT compile now allocates 399 MB, not 419 MB.
- **Queries and XPath expressions are parsed in two stages** (#401): SLL prediction first, and full LL
 with error reporting only for input that SLL cannot parse. For a generated Schematron validator of
 2.3 MB, an XSLT compile now allocates 795 MB, not 1,095 MB (-27%). With the string-literal lexer
 change it is 435 MB (-60%), and the compile takes about 400 ms, not 590 ms. Syntax errors are
 reported as before.
- **String literals are lexed from the lexer's DFA cache** (#401). A predicate in the string-literal
 rule (`{AllowRawAmpersand}?`) kept every string literal out of the cache, so each character was
 matched by ATN simulation. For a generated Schematron validator of 2.3 MB, an XSLT compile now
 allocates 735 MB, not 1,095 MB (-33%), and takes 460 ms, not 590 ms. A bare `&` in an XQuery
 string literal is still XPST0003; the error now gives the position of the `&`.
- **The input document can be validated before a query runs** (#267): `QueryEngine.InputValidation`
 and `XQueryFacade.InputValidation` (strict or lax), and `--validation strict|lax` (also
 `--val:strict`, `--val:lax`) on the `xquery` command (package `xquery4`). The context item, when it is a document or an
 element, is validated as `validate { . }` would, against the schemas the query imports and the
 other schemas of the schema provider; the query and `$input` then see the typed node. An invalid
 input is XQDY0027; with strict validation, an element with no top-level declaration is XQDY0084.
 Without the setting the input stays untyped, as before.
- **A type derived from `xs:NOTATION` makes an `xs:NOTATION`.** Its constructor function made a
 value of the type that was not an `instance of xs:NOTATION`. A QName or a NOTATION now casts to a
 NOTATION type (XPath 3.1 §19.1); it was "Cannot cast xs:QName to Notation". QT3 notation-cast-3;
 W3C XSLT notation-0002, 0003, 0401-0404.
- **A partial application of a resource function runs as the module that made it**, as a named
 reference already did: `json-doc(?)` made in a library module and called from the main module
 resolved its location against the main module's base URI, and `ResourceRequest.ModuleUri`
 named the main module.
- **`fn:transform` turned off by the resource policy raises FOXT0004** ("the XSLT
 transformation has been disabled", F&O 3.1); it raised FOXT0001, which means no suitable
 XSLT processor.