vei 1.0.0.5

Ven — Vector Enrichment Library

Part of the RAG Document Toolkit for .NET, Version 1.0 — Sub Systems, Inc.

Convert DOCX, RTF, and HTML into structure-rich, RAG-ready chunks — with vector enrichment built in.

Vector search returns the chunks that resemble the question. It does not return the other half of the table, the rest of the tracked revision, or the comment that answers the question but happens to share no words with it. Ven closes that gap. Given the chunks your retrieval step found, Ven uses the document structure recorded by Dan to add the chunks that belong with them — and tells you the token cost as it goes.

Ven also builds the per-document index entries used for AI-assisted document pre-selection, so a routing model can pick the few relevant files in a collection before any retrieval happens.

What Ven does

Completions finish a structure that retrieval only partly captured:

• VesCompleteTables — brings in the remaining chunks of any table that has a fragment in the retrieved set; optionally adjacent tables or all tables.
• VesCompleteRevisions — brings in the other fragments of tracked revisions already present.
• VesCompleteComments — brings in the rest of reviewer comments already present.

Additions widen the context around the hits:

• VesAddPageChunks — the rest of the page, neighboring pages, and the document's first and last pages, within a token budget.
• VesAddSectPages — pages of the enclosing section (or the whole section).
• VesAddRevisionChunks / VesAddCommentChunks — every revision or comment chunk in the document, for one author or for all.

All-in-one — VesAddRelatedChunks runs the completions in a single call, with a token budget governing the all-tables option.

Document queries — VesHasTables, VesHasRevisions, VesHasComments, VesGetRevisionAuthors, VesGetCommentAuthors, VesGetMdoSectPages, VesGetMdoSectChunks, VesGetTokenCount, VesGetChunkTokenCount.

Document index — VesCreateDocumentIndex plus the static VesGetFilterSystemPrompt and VesGetIndexSystemPrompt.

How it fits in your pipeline

Ven does not touch your vector database and does not call any AI service. It works purely on sequence numbers and metadata:

1. Retrieve chunks however you like.
2. Group the retrieved chunks by ssDocId.
3. For each document, load that document's complete set of Dan metadata records from your store and construct a Ven object.
4. Pass in the retrieved sequence numbers (ssSeq), run the expansion methods you want, and get back the expanded set.
5. Fetch the chunk text for the expanded sequence numbers and send it to the model in document order.

Chunks from other providers in a mixed pool simply bypass Ven.

Installation

dotnet add package vei

Quick start

using SubSystems.RagDocumentToolkit.Ven;

// metaRecs: ALL metadata records of one document, as produced by Dan
// (Dictionary<string, object>[]). Records that have been through a JSON
// or database round trip are accepted.
var ven = new Ven(metaRecs);

// Sequence numbers (ssSeq) of the chunks your vector search returned
// for this document.
var retrievedSeq = new SortedSet<int>();
foreach (var meta in retrievedMeta)
    retrievedSeq.Add(Convert.ToInt32(meta["ssSeq"]));

var exp = ven.VesBeginExpansion(retrievedSeq);

// Completions in one call.
ven.VesAddRelatedChunks(exp,
    false,      // revisions
    false,      // comments
    true,       // AllTables (subject to TokenBudget)
    true,       // AdjacentTables
    4000);      // TokenBudget - a literal limit; pass a very large value for no limit

// Optional additions, each returning the cumulative enrichment tokens added.
ven.VesAddPageChunks(exp, true /*FullPage*/, 0 /*NumPages*/, 2000 /*TokenBudget*/);

int addedTokens = ven.VesGetEnrichmentTokenCount(exp);
SortedSet<int> expandedSeq = ven.VesGetExpandedSeq(exp);
ven.VesEndExpansion(exp);

// Replace this document's retrieved chunks with the chunks in expandedSeq.

Design notes

• The Ven object is immutable. All mutable state lives in the expansion object returned by VesBeginExpansion, so one Ven object can serve many queries, concurrently.
• Expansion is anchored. Every method expands relative to the set you passed to VesBeginExpansion, not the accumulated result, so chained calls do not snowball. Call VesAnchorExpandedSeq when you do want the next round to expand from the expanded set.
• Token budgets are literal. VesGetTokenCount(exp) and VesGetEnrichmentTokenCount(exp) report cost at any stage at negligible expense, so you can stop, or start over with fewer options, to stay within a context budget.
• Gate on capability. Check VesHasRevisions() / VesHasComments() before spending a model call to classify whether a question concerns revisions or comments.
• Errors. The constructor and VesBeginExpansion throw ArgumentException for null, incomplete, or duplicate-sequence input. The LogMsg event reports diagnostics.

AI-assisted document pre-selection

For collections of many files, retrieval alone must give every file a chance, which spends tokens on irrelevant documents. Ven supports a cheaper first step:

// Once per file, at ingestion time. pct is the share of the file's tokens
// allowed for its index entry (capped at 50).
string entry = ven.VesCreateDocumentIndex(10, out int entryTokens);

Join the entries of all files into one collection index. At query time, send the index and the user's question to an inexpensive model using Ven.VesGetFilterSystemPrompt(); the model scores each document's relevance, and your application keeps those above a threshold. If the selected files are small enough, send them whole and skip retrieval and enrichment entirely. With Ven.VesGetIndexSystemPrompt(), questions about the collection itself can be answered from the index alone.

In the toolkit's demo, this step resolves more than half of all queries by sending only the few selected documents in full — better context at a fraction of the tokens — with an index of roughly 5% of the collection's size.

Documentation and demo

The complete Ven reference is in the toolkit help file, rag_document_toolkit_help.htm. The toolkit's multi-file C# demo shows Ven in a full pipeline: pre-selection, LOCAL/GLOBAL retrieval, per-document enrichment under a token budget, and cited answers.

Licensing

Ven is licensed together with Dan; no separate Ven license is required. One Dan.DasSetLicenseInfo call at program start activates both libraries. Licensing is per developer, in Desktop, Server, Unlimited Server, and Enterprise editions. The full license agreement is in the toolkit help file.

Support

Sub Systems, Inc. 3200 Maysilee Street, Austin, TX 78728 512-733-2525 https://www.subsystems.com

Copyright © 2026 Sub Systems, Inc. All rights reserved.