JsonRazor 2.0.2

2.0.2:

Breaking changes:

• JsonToken.GetParent() has been removed. Its usage in internal code was scarce and for client code, it's not very usable anyway, since parent can be easily tracked, and most operations are in direction from parent to child, not conversely. Existance of need to keep track of parent has been main blocker to move away from JsonValue.
• JsonValue and ValueBase classes has been removed. This has a lot of consequences listed in next section.
• TokenType.Property enum option has been removed. String, number, bool or null values in both JsonArray and JsonObject are now set to use TokenType.Value.
• IToken.Index and ICommonToken.Index properties have been removed. Index was tracked by array as kind of workaround to JsonValue existance. Since now items in array have all original order, it doesn't seem justified to keep it.
• CachingDeserializer.UseInternalParsing has been removed. Over last couple of releases standard parsing has been improved to a point where its performance is as same as internal parsing's. Not much could be squeezed out of that either, so it's useless to keep it. I'd rather try different approach with deserialization.

Changes to existing code:

• JsonToken.Remove() has been changed to Remove(JsonToken) and now it's expected to be called on parent with passed child token to remove.
• Similarly, JsonToken.SwapFor(JsonToken) has been changed to Replace(JsonToken, JsonToken) and it's also expected to be called on parent. Both methods will expect that passed token is child of current parent, and in this case, target token should not have a parent. Aside from that, previous restriction on swapping only tokens of same type in JsonObject has been lifted. Target will be placed on the same index as removed token.
• JsonArray[int] indexer has been changed to return JsonToken instead. Since JsonToken replaces JsonValue to hold single value for array, said indexer can now return all tokens in array, including arrays and objects. It has also been moved to JsonToken class, so it's accessible to JsonObject as well. Although not very useful, it may prove handy when input is known or controlled.
• JsonArray default collection interface is now IEnumerable<JsonToken>, and implementation of IEnumerable<JsonValue> is removed, which means that again, array by default enumerates all children.
• JsonToken.Values() extension method return type is changed to IEnumerable<JsonToken> and it now handles both parent JsonArray and JsonObject, JsonToken.Properties() is removed. Behaviour stays similar, tho, as it will enumerate all tokens with Type = Value. Other methods, like ToList(this JsonToken), that collects child tokens in List<jsonToken>, will now also include values if JsonArray is passed.
• ToJsonArray(this IEnumerable<JsonValue>) has been removed. Possible replacement is ToJsonArray(this IEnumerable<JsonToken>). Similarly, all AddRange overloads taking JsonValue have been removed as well.
• JsonObject.FirstProperty is renamed to FirstValue and moved to JsonComposite class, so it's accessible to array as well.
• TokensWithNulls(this JsonArray) has been somewhat changed, it will return all tokens, but null in place of those, whose ValueType is Null. Not really useful tho. Considered to remove, but left to reduce bulk of hard changes.
• IToken.Convert<T>() marked as obsolete. Recommended replacement is IToken.Convert<T>(Deserialization).
• ValueBase.IntStyle, FloatStyle, True, False and Null constants, Value, ValueType properties, SetValue() overloads, ToLong(), ToDouble(), ToBool(), ToUtcDateTime(), ToDecimal(), Get() etc. have been all moved to JsonToken.
• JsonValue FromBoxedValue(), FromValue(), FromBool(), FromLong(), FromDouble() etc. overloads have been moved to JsonToken as well. It's now perfectly fine to create JsonToken without name, although JsonObject will complain if such token is added, without providing name.
• JsonObject.Append() overloads that had name argument optional are changed so name is mandatory. Needed change, since token.Name may be null, and such code could be forgotten :p. If token Name is not null, then passing token.Name is fine.
• JsonObject.Append(JsonObject, string) and Append(JsonArray, string) have been removed. Append(JsonToken, string) handles these cases just fine. Also ToJsonObject(this IEnumerable<JsonObject>), ToJsonObject(this IEnumerable<JsonArray), ToJsonArray(this IEnumerable<JsonObject>), ToJsonArray(this IEnumerable<JsonArray>) overloads have been removed. Existing code should automatically switch to ToJsonObject(this IEnumerable<JsonToken>) and ToJsonArray(this IEnumerable<JsonToken>) overloads.
• JsonArray.AsLazy(), FindToken() and GetFromPath() will now also return values as JsonToken, so usages may need attention.
• IToken.LoadTokens() and QueryTokens() will also return / call code on value tokens.
• JsonObject no longer groups tokens in any way, their source order will be maintained.
• JsonToken.ValueCount and IToken.ValueCount return number of tokens with Type == Value for both object and array.
• JsonComposite.IsEnumerated has been moved to JsonToken base class and renamed to IsParsed. New name better reflects meaning of flag, since from now on, enumerating object or array won't cause them to be parsed. This also applies to Tokens(), Values(), Objects() and Arrays() extension methods. AsLazy() still continues to return either loaded or newly parsed tokens.
• Cloning constructors - JsonArray(JsonArray) and JsonObject(JsonObject), all Equals() overloads, GetHashCode(), Write(StreamWriter) won't force parsing of composite or any child composite. Methods that continue to force full parsing of current composite are Add() and Append() overloads, Remove() and Replace(). Add() / Append() and Replace() also force parsing of added token. In case of these methods, certain changes to composite may render it unable to proceed, if parsing is then requested. Also ToJsonObject(), ToJsonArray(), AddRange() extension overloads also force parsing since they use same adding logic.
• JsonObject(string) and JsonArray(string) constructors have been removed. Alternative is JsonToken.From() method. JsonToken(string) constructor was also removed, because it's kinda ambiguous whether string refers to name or value.
• Query.ByIdentifyValue() has been renamed to ByValueInSet(string, IEnumerable<string>, bool), OrJoin(AbstractQuery, AbstractQuery) was renamed to Or(AbstractQuery, AbstractQuery), AndJoin(AbstractQuery, AbstractQuery, bool) was renamed to And(AbstractQuery, AbstractQuery) and it no longer supports creating query that targets single token, because it can be simply expressed in lambda.

New features:

• JsonObject.Add(string, JsonToken) has been added. It's similar to Add(JsonToken), difference is that it takes token name as argument. More important, thanks to this method it's now possible to initialize JsonObject like Dictionary<string, JsonToken>, e.g. new JsonObject { { "name", new JsonToken("value" } }
• Previous JsonObject token hashing has been moved to separate type - JsonObjectHashSet. Indexer[string], Equals(JsonObject) and AreNamesUnique() methods are available. Complementary JsonObject.Equals(JsonObjectHashSet) is added. Equals makes sense, since tokens are matched by name. Second method returns bool that indicates if token names in given object are unique, which may come handy, because this check is not performed automatically for obvious reasons. Important thing to note is that this type is struct and will not track changes made to original object, so it pretty much stays the snapshot of object's tokens at creation time.
• JsonToken[string] indexer was introduced to replace this[string, TokenType] when called with default type. Previous indexer's default type value has been removed. Calls that don't provide custom type should switch to new indexer automatically. Both contain less instructions so if they're devirtualized, both may now be inlined as well.
• JsonObject.GetToken(string) was added. Indexer and GetToken() allow to fine-tune performance of token lookup to use case.
• Query.ByFunc(Func<IToken, QueryResult>) has been added. It allows to e.g. query nested tokens when iterating over upper-lever hierarchy that is performed by QueryTokens().
• Query.ByValueInSet(IEnumerable<string>, bool) and ByValueInSet(IEnumerable<string>, int, int, bool). First overload checks if value of token passed to query is on the list of accepted values (or isn't, when invert is set to true). Since it doesn't check token name, it can be used to determine if array contains value. Query will either return Success for first encountered token with matching value or iterate over whole array and parse it. If searched value is said to be in some range (like first 10 elements), latter overload should be used. This query iterates only over skip + take tokens, so if value is not found, Fail will be returned, which in turn enabled parsing optimisations to be applied when parsing remainder of the array.
• AbstractQuery.Query(IToken) has been exposed to consumer, which allows to use query in delegate.
• JsonToken.ToString() when used on object's value will no longer print name of token ("Name":) before value, so the output is consistent with ToString() output of JsonArray / JsonObject also being children of object.

Fixed:

• JsonObject.AsLazy(TokenType) enumerator wouldn't distinguish Object and Array.
• Enumerators returned by Objects(), Values() or Arrays() extension methods weren't able to restart properly. Well, as long as someone doesn't grab enumerable in two variables and get enumerator from both (same enumerator instance will be returned), it'll work.
• JsonStream could mess up IToken content in certain cases with nested arrays.
• JsonStream.StreamFile(StreamReader, bool, bool) could incorrectly optimize loading. Issue could also affect DrawTokens(StreamReader) depending on usage.
• Cloning constructor of JsonArray(JsonArray) could return broken array when source array is empty.
• In some cases JsonToken.Add(JsonToken) or Append() methods could break internal token list.
• AbstractQuery returned from Query.AndJoin() could in some cases return wrong result.
• AbstractString char enumerator used to omit first char.

Performance:

• String parsing in two tested cases improved around 7-13% on .net 5, 7-8% on .netcoreapp 3.1, 7-10% on .net 4.5. In all cases memory usage decreased around 2.5-4.5%, so that should reduce pressure on GC a little bit.
• Deserialization from string could be faster up to 8%.
• Stream parsing and parsing-on-demand on tokens from JsonToken.From() should also be faster and allocate less.
• Some operations (like calling Add / Append) on tokens won't cause allocations from now on.
• IToken parsing using JsonStream should be marginally faster.
• JsonToken.Remove() and Replace() methods should now also be faster.
• JsonStream.StreamConverted<T>() should apply optimisation on top-level hierarchy, if T is object with properties. In example scenario with 100 elements deserialization was around 28% faster. At this point size of array is unknown, so it's possible that applying optimisation will actually decrease performance (array with element number < 30 or something, depends), so it's advised to just deserialize using CachingDeserializer. For small input ITokens would still be slower anyway.
• JsonObject[string, TokenType] indexer will now perform adaptive hashing by default. Only tokens that were tried to be matched during lookup will be hashed. It also doesn't incur any additional allocations, it's not cancelled when any token is added, replaced or removed from object. A lot depends on the input, but it should perform nicely when names are similar. Lookup of 10 tokens could be improved up to ~50%, but on the other hand, might as well be slowed down in certain name cases. In my example linear lookup of 20 tokens performed similarly to Dictionary<string, JsonToken> (except for .net 5, where dictionary seemed like 20% faster), and hashed lookup was practically twice as fast. Hashing will not apply to other methods (GetToken(), GetFromPath(), Find() etc). I guess using GetToken(string) is better idea when there's going to be like one or few searches (let's say up to 6-9, in this vicinity indexer seems to perform closely to linear search, or the loss is negligible.) Indexer hashing is also not applied the first time token is read as part of searching.
• JsonObjectHashSet uses improved algorithm than JsonObject used previously, and can outperform linear search (tested on GetToken()) at around 25-30 tokens. It seems to be faster than Dictionary<string, JsonToken> for number of tokens up to 100 (checked results on 2k and it was noticeably slower, but dunno how it behaves inbetween - on .net framework 4.5) but dictionary catches up around ~40 on .net core 3.1, and is usually faster on .net 5 around ~20 tokens.
• JsonToken.Write(StreamWriter) has been improved and depending on case it's ~2-7% faster.