Transformations.Text 2.0.5.2

Transformations.Text

A comprehensive, first-class text extraction and transformation library for .NET — purpose-built for RAG pipelines, vector-store ingestion, and AI data workflows.

Features

• 18 built-in extractors — PDF, DOCX, PPTX, Excel, HTML, Markdown, CSV, EML, MSG, ODT/ODP, RTF, EPUB, JSON, XML, YAML, log files, source code, and ZIP archives
• AI-ingestion safety — detects hidden text, invisible Unicode, obfuscated and AI-targeted instructions; withholds suspicious parts or documents; GetModelInput returns provenance-labelled segments for RAG and agents. Read Using with AI models before feeding untrusted documents to a model
• DI-ready — IDocumentTextExtractor facade interface; register with one call via AddTextExtractor(); fluent TextExtractorBuilder for fine-grained pipeline control
• Async & streaming — GetTextAsync / GetTextWithMetadataAsync accept Stream directly; GetTextBatchAsync parallelises multi-document pipelines
• Extraction options — per-call ExtractionOptions: MaxCharacters, MaxPages, StartPage, EndPage, IncludePageMarkers, TableMode, MaxDecompressedBytes, MaxContainerEntries, MaxInputBytes, Timeout, CancellationToken, and WarningSink
• Table rendering modes — TableMode.KeyValue (default), Markdown (pipe tables), Csv, or Omit; applies to DOCX, HTML, Excel, and ODT tables
• Content-type sniffing — magic-byte fallback when the extension is wrong or missing; detects PDF, DOCX/XLSX/PPTX, MSG, HTML, RTF, EML, EPUB, and ZIP families
• RAG chunking — TextChunker splits text by characters (word-boundary-aware), sentences, paragraphs, Markdown sections, or token count with configurable overlap; chunks carry PageNumber? and HeadingPath? provenance metadata
• Extensible — implement ITextExtractor and plug in a custom format; or use TextExtractorBuilder to add, replace, or disable built-in extractors
• Encoding-aware — all extractors honour BOM and detect encoding from stream headers
• ILogger integration — structured log events (extractor used, char count, duration, failures) via ILogger<TextExtractor>; zero-config when logging is wired via DI
• Rich result metadata — ExtractionResult.ExtractorName, .Duration, and .Warnings on every call; format-specific document properties (PDF title/author/page count, DOCX title/author/revision, MSG subject/sender/date, and more)
• Zero new dependencies for RTF and EPUB — both formats parsed with BCL only (self-contained RTF state machine; EPUB via System.IO.Compression + built-in HTML extractor)
• JSON Schema utilities — validate, list errors, normalize AI-emitted JSON (strips code fences), compare payloads
• Markdown structure — build heading maps, extract sections, normalize, compare
• CSV data profiling — CsvDataProfiler infers column types, builds distributions, and detects data issues

Installation

dotnet add package Transformations.Text

Supported .NET Versions

• .NET 10.0
• .NET 9.0
• .NET 8.0

---

Quick Start

using Transformations.Text;

var extractor = new TextExtractor();

byte[] bytes = File.ReadAllBytes("report.docx");
var result = extractor.GetText("report.docx", bytes);

if (result.IsSuccess)
    Console.WriteLine(result.Text);
else
    Console.WriteLine(result.ErrorMessage);

With extraction options

var options = new ExtractionOptions
{
    MaxPages          = 10,           // first 10 pages / slides / chapters
    TableMode         = TableMode.Markdown,  // emit tables as pipe tables
    IncludePageMarkers = true,        // insert [Page N] / [Slide N] markers
};

var result = extractor.GetText("slides.pptx", bytes, options);

Format detection (no file name available)

string? extension = TextExtractor.DetectFormat(unknownBytes);
// ".pdf", ".docx", ".rtf", ".epub", ... or null

---

Using with AI models (RAG, agents)

Documents from outside your organisation (uploads, email, web pages, shared drives) can carry text written to manipulate a model that reads them: "ignore your instructions", "send the customer list to…", often hidden from the human who looks at the document. This library helps you detect, withhold and label such content. It does not make text safe, and it cannot stop a model from following text it is given. Correct use needs the steps below in your application as well.

What the library does and does not do

It does	It does not
Find known signals: hidden text (HTML, DOCX, PDF, PPTX, XLSX), invisible Unicode, bidi tricks, lookalike or encoded instructions, AI-targeted wording, OCR text	Prove that text without findings is harmless. SafetyVerdict.NoKnownSignals means no known signal, nothing more
Keep hidden text and withheld parts out of the text you get back	Understand meaning. A visible instruction in plain words is only flagged (Low), because security articles and quoted examples look the same
Say where each piece of text came from (file, archive entry, attachment, page, OCR)	Decide what your model or agent may do. Permissions are yours to enforce
Bound parsing work (sizes, entries, pages, time)	Fully isolate hostile files. Some parser calls cannot be interrupted

1. Extract with the AI profile

var extractor = new TextExtractor();
ModelInput input = extractor.GetModelInput(fileName, bytes);          // uses ExtractionOptions.ForAiIngestion()

ForAiIngestion() turns on quarantine at High, review at Medium, drops hidden text from the output, and sets finite limits that nested archives and emails share. To read text in images and scanned PDFs, add an OCR engine (with { OcrEngine = new TesseractOcrEngine(tessdata) }, namespace Transformations.Text.Ocr, Windows only; see OCR). Without OCR, image content is not read at all, so it is neither indexed nor checked: an image file fails with ErrorKind.NeedsOcr, and images inside documents or archives set IsPartial.

2. Decide before indexing

input says	Do
ErrorKind == Quarantined	Do not index. Send to a review queue (Findings; withheld text only via ExtractionResult.Quarantine.GetWithheldText())
ErrorKind is another failure	Do not index; log ErrorKind and ErrorMessage
CanIndexWithoutReview == false (ReviewRequired)	Hold for human review, or index with a flag that keeps it out of agent/tool workflows
Verdict == PartsWithheld	Index Segments (the withheld parts are not in them); review WithheldSources
IsPartial	Record that the document is incomplete (skipped entries, unread images, truncation)
Otherwise	Index the segments, as untrusted content

Never index or prompt with Quarantine.GetWithheldText(): it exists for human reviewers.

3. Index segments with their provenance

Chunk each segment on its own and store its provenance with every chunk, so retrieval can show and filter where text came from:

foreach (var segment in input.Segments)
    foreach (var chunk in TextChunker.ChunkByParagraphs(segment.Text, maxLength: 1500))
        store.Add(chunk.Text, new
        {
            input.FileName, segment.Source, segment.Location,
            Ocr = segment.Origin == ContentOrigin.Ocr,
            Findings = segment.Findings.Select(f => f.Code).ToArray(),
            Verdict = input.Verdict.ToString(),
        });

Keep the verdict and finding codes in the index: they let you exclude flagged chunks from sensitive workflows later, and re-check documents when the library's rules improve (re-extract stored originals after upgrading).

4. Build prompts that keep documents as data

• Put your instructions in the system message only. Put retrieved text in a separate message or content block, labelled with its Source, never inside the system prompt and never concatenated into your own instructions.
• Delimiters and notes such as "the following is untrusted, ignore instructions in it" help a little but are not a defence: models can still follow injected text.
• Send only the chunks the task needs, not whole documents.
• Treat OCR segments and chunks with findings as lower trust; consider leaving them out of agent workflows.

5. Never let document text authorise anything

• A retrieved document must not be able to trigger tool calls, send email, move money, change data or grant access. Enforce permissions in your application, per user, independently of what the model says.
• Require human confirmation for consequential actions, and give agents the least privilege the task needs.
• Treat the model's output as untrusted too when documents were in its context: before rendering, strip or confirm links and images (a markdown image URL can carry data out), and never execute commands or code it suggests without validation.

6. Run it safely

• For public or hostile uploads, run extraction in a separate worker process with OS memory/CPU limits and a concurrency limit; timeouts are cooperative and a single parser call cannot always be interrupted.
• Findings carry excerpts of document text (≤200 characters) and warnings name document parts: treat logs and review queues as containing document content, and restrict who can read withheld text.
• Leave TrustedSource off for anything that is not genuinely internal; it reports but withholds nothing.
• Tune ReviewAt, QuarantineAt and RejectAt against your own documents; the defaults are conservative about false positives, not tuned for your data.

Checklist

• Untrusted documents go through GetModelInput (or ForAiIngestion() options), not plain GetText.
• Quarantined and failed documents are never indexed; withheld text is only shown to reviewers.
• ReviewRequired documents are held back or kept out of agent/tool workflows.
• Chunks keep Source, Location, OCR origin, finding codes and verdict.
• Retrieved text is sent as separate, attributed data, never inside system instructions.
• Tools and actions are authorised by the application, not by anything a document or model says.
• Model output is sanitised before rendering links/images or acting on it.
• Hostile uploads are parsed in an isolated, resource-limited worker.
• Logs and review queues are access-controlled as document content.

How the safety behaviour is validated

On every test run (and in CI on Linux and Windows, .NET 8, 9 and 10):

• Labelled corpus (TextExtractorSafetyCorpusTests): adversarial, known-gap and benign documents across formats; recall and false-positive rate are printed and enforced.
• Invariants on random documents (TextExtractorSafetyInvariantTests): hidden or withheld text never appears in Text or in a model-input segment; findings point inside the text and excerpts stay bounded; verdicts agree with error kinds; results are identical under concurrent use.
• Mutation fuzzing of every built-in format (MutatedInput_NeverThrows) and limit/cancellation tests.
• Mutation check of the tests themselves: disabling HTML hidden-text detection or per-part withholding makes the corpus and invariant tests fail.

Not covered by these tests, and worth doing before relying on the library for high-risk ingestion: measuring on your own real documents and on public prompt-injection datasets, an end-to-end test with your actual model (plant canary instructions in documents and check whether they reach or influence the model), coverage-guided fuzzing of the parsers, and comparing extracted text with what a browser or Office actually renders.

Reference details: Safety gates and quarantine, AI ingestion (settings, escalation rules, per-format coverage, measured detection) and Partial success contract.

---

Dependency Injection

// Minimal registration — all built-in extractors
services.AddTextExtractor();

// Fluent builder — add, replace, or disable extractors
services.AddTextExtractor(b => b
    .Replace<PdfExtractor, MyPdfExtractor>()
    .Add<MyCustomExtractor>()
    .Disable<LogExtractor>());

// Legacy lambda — mutate the list directly (still supported)
services.AddTextExtractor(list =>
{
    list.Add(new MyCustomExtractor());
    list.RemoveAll(e => e is CsvExtractor);
});

Inject IDocumentTextExtractor wherever you need it:

public class DocumentProcessor(IDocumentTextExtractor extractor)
{
    public async Task ProcessAsync(string name, Stream stream, CancellationToken ct)
    {
        var result = await extractor.GetTextAsync(name, stream, ct);
        if (result.IsSuccess) ...
    }
}

---

API Reference

IDocumentTextExtractor / TextExtractor

Method	Description
GetText(fileName, byte[])	Extract from byte array
GetText(fileName, byte[], options)	Extract with per-call options
GetText(fileName, Stream)	Extract from stream
GetText(fileName, Stream, options)	Stream variant with options
GetTextAsync(fileName, Stream, ct)	Async stream extraction
GetTextAsync(fileName, Stream, options, ct)	Async with options
GetTextWithMetadata(fileName, byte[])	Extract + rich metadata
GetTextWithMetadata(fileName, byte[], options)	Metadata variant with options
GetTextWithMetadata(fileName, Stream)	Stream variant
GetTextWithMetadataAsync(fileName, Stream, ct)	Async variant
GetTextBatchAsync(docs, options?, maxDegreeOfParallelism?, ct)	Async multi-document batch
GetModelInput(fileName, byte[], options?)	AI use: provenance-labelled segments, verdict, findings, withheld part names (TextExtractor only; options default to ForAiIngestion())
GetSupportedExtensions()	All extensions the current pipeline handles
IsSupported(fileName)	Quick check for a specific file name
static DetectFormat(byte[])	Sniff format from magic bytes; returns extension or null
static CreateDefaultExtractors()	Build the default extractor pipeline

ExtractionResult — bool IsSuccess, string Text, ExtractionErrorKind ErrorKind, string? ErrorMessage, string? ExtractorName, TimeSpan Duration, IReadOnlyList<ExtractionWarning> Warnings, IReadOnlyList<SafetyFinding> Findings, QuarantinedContent? Quarantine, SafetyVerdict SafetyVerdict (the metadata result has the same safety members)
ExtractionMetadataResult — adds Dictionary<string, string> Metadata, string? ExtractorName, TimeSpan Duration, IReadOnlyList<ExtractionWarning> Warnings

Common metadata keys: file.name, file.extension, file.bytes, text.length, text.lineCount, text.wordCount, extraction.extractor, extraction.durationMs. If non-fatal container issues occur, metadata also includes extraction.warningCount and extraction.warnings.

Format-specific metadata:

• Markdown — markdown.headingCount, markdown.sectionCount, markdown.headings
• JSON — json.rootKind, json.topLevelPropertyCount, json.topLevelProperties, json.topLevelArrayLength
• EML — email.subject, email.fromCount, email.toCount, email.date, email.attachment.count, email.attachment.names
• MSG — msg.subject, msg.from, msg.date
• PDF — pdf.title, pdf.author, pdf.subject, pdf.keywords, pdf.creator, pdf.pageCount
• DOCX — docx.title, docx.author, docx.subject, docx.created, docx.modified, docx.revision
• PPTX — pptx.title, pptx.author, pptx.slideCount
• XLSX — xlsx.title, xlsx.author

---

ExtractionOptions

var options = new ExtractionOptions
{
    MaxCharacters       = 50_000,          // hard cap on output length
    MaxPages            = 5,               // max pages / slides / chapters to extract
    StartPage           = 2,               // 1-based; skip earlier pages
    EndPage             = 8,               // inclusive; stop after this page
    IncludePageMarkers  = true,            // insert [Page N], [Slide N], [Chapter N]
    TableMode           = TableMode.Csv,   // KeyValue | Markdown | Csv | Omit
    TableHeaders        = TableHeaderMode.Auto, // Auto (format's header marker) | FirstRow | None
    MaxDecompressedBytes = 256 * 1024 * 1024, // ZIP bomb guard (default 512 MB)
    MaxContainerEntries = 5_000,           // archive/attachment guard (default 10,000)
    MaxInputBytes       = 50 * 1024 * 1024, // reject larger inputs before parsing (default 256 MB; null = no limit)
    Timeout             = TimeSpan.FromSeconds(30), // stop a slow parse; result is ErrorKind.TimedOut
    CancellationToken   = requestAborted,  // stop the parse itself; throws OperationCanceledException
    RetryWithDetectedFormat = true,        // retry once with the sniffed format if the extension's parser fails
    WarningSink         = warning => logger.LogWarning(
        "{Code} while extracting {Source}: {Message}",
        warning.Code,
        warning.Source,
        warning.Message),
};

TableMode applies to any extractor that produces tables: DOCX, HTML, Excel (.xlsx/.xls), and ODT.

ZIP, EPUB, and EML attachment extraction can partially succeed. Unsupported entries, skipped chapters, unreadable attachments, and decompression/entry limits are reported as ExtractionWarning records instead of being silently swallowed.

Partial success contract

Extraction has three outcomes:

Outcome	Result shape	Meaning
Full success	IsSuccess == true, Warnings.Count == 0	The selected extractor completed without known skipped content.
Partial success	IsSuccess == true, Warnings.Count > 0	The top-level document was readable, but some container content was skipped, missing, unsupported, or over a configured limit.
Success with withheld parts	IsSuccess == true, Quarantine != null	ZIP entries, EML attachments or the EML body were withheld by the safety gate; stub lines mark them in Text.
Quarantined	IsSuccess == false, ErrorKind == Quarantined	The whole document was withheld by the safety gate. Text is empty; the text is only in Quarantine.
Failure	IsSuccess == false, ErrorKind != None	The top-level document could not be extracted. Text is empty and ErrorMessage explains the failure.

Findings can be non-empty in every outcome: findings below the threshold are reported without withholding anything. A pipeline that feeds an AI model can handle all outcomes like this:

var result = extractor.GetText(name, bytes);

if (result.ErrorKind == ExtractionErrorKind.Quarantined)
    reviewQueue.Add(name, result.Findings, result.Quarantine!);   // never send it to the model
else if (!result.IsSuccess)
    log.Warn($"{name}: {result.ErrorKind}: {result.ErrorMessage}");
else
{
    if (result.Quarantine != null)
        reviewQueue.Add(name, result.Findings, result.Quarantine); // parts were withheld; Text has stubs instead
    if (result.Findings.Any(f => f.Severity >= SafetySeverity.Medium))
        log.Warn($"{name}: {result.Findings.Count} safety findings below the quarantine threshold");
    model.Ingest(result.Text);                                     // Text never contains withheld content
}

Failure kinds (ExtractionErrorKind):

Kind	Meaning
Empty	The input had zero bytes.
Unsupported	No extractor is registered for the file type.
Binary	The file is binary (image, executable, unknown archive) and only the plain-text fallback would have accepted it. Earlier versions returned an empty success here.
TooLarge	The input is bigger than MaxInputBytes. It was not parsed; stream input stops being read once it passes the limit.
LimitExceeded	A DOCX, XLSX/XLSM, PPTX, ODT or ODP file has more parts than MaxContainerEntries or expands past MaxDecompressedBytes.
TimedOut	Parsing ran longer than Timeout and was stopped.
NeedsOcr	The file is an image and no OCR engine is available (OcrEngine).
Quarantined	The safety gate withheld the whole document (see Safety gates and quarantine). Text is empty; the reasons are in Findings.
PasswordProtected, Corrupted, ExtractionFailed	The parser itself failed.

Cancelling ExtractionOptions.CancellationToken (or the cancellationToken argument of the async methods) throws OperationCanceledException instead of returning a result. The built-in extractors check the token between pages, slides, rows, paragraphs, HTML nodes and archive entries, so a cancelled or timed-out parse actually stops instead of running on in the background. A single library call that cannot be interrupted (for example, laying out one PDF page or loading one XML part) finishes before the check runs.

Warnings are recoverable issues. They do not mean the returned text is invalid; they mean it may be incomplete. For ingestion systems, a common policy is to index full successes automatically, index partial successes with an audit flag, and reject or quarantine failures.

Stable warning codes are exposed as ExtractionWarningCodes constants:

Code	Constant	Typical cause
container.manifestMissing	ContainerManifestMissing	EPUB META-INF/container.xml is missing or unreadable.
container.spineEmpty	ContainerSpineEmpty	EPUB OPF/spine exists but no readable ordered content could be found.
container.entryMissing	ContainerEntryMissing	EPUB metadata references a chapter that is not present in the archive.
container.entryLimit	ContainerEntryLimit	ZIP, EPUB, or EML had more entries than MaxContainerEntries.
container.byteLimit	ContainerByteLimit	ZIP, EPUB, or EML attachment extraction would exceed MaxDecompressedBytes.
container.unsupportedEntry	ContainerUnsupportedEntry	ZIP entry or EML attachment has no safe registered extractor.
container.entryFailed	ContainerEntryFailed	An individual entry was selected for extraction but failed independently of the top-level document.
content.typeMismatch	ContentTypeMismatch	The extension contradicts the content's binary signature (a DOCX named invoice.pdf, a PDF named notes.txt). Raised for the document itself, ZIP entries and EML attachments, on success and on failure. Disguised files are a common malware-delivery trick: quarantine or scan rather than index.
content.needsOcr	NeedsOcr	An image part or scanned PDF page has no text layer and no OCR engine ran.
content.ocr	OcrText	Text came from OCR. Treat it as untrusted.
safety.finding	SafetyFinding	The safety gate found something; the full record is in Findings.
safety.withheld	ContentWithheld	The safety gate withheld a part (replaced by a stub line) or the whole document.

Safety gates and quarantine

Every extraction runs a safety gate over the text it produced, aimed at content that tries to manipulate an AI reading it (hidden prompts). Each signal becomes a SafetyFinding (Code, Severity, Source, Reason, Excerpt of at most 200 characters, Offset, Withheld) on result.Findings, through ExtractionOptions.FindingSink, and as a safety.finding warning.

Code (SafetyFindingCodes)	Signal	Severity alone	Raised to High when
safety.tagCharacters	Unicode tag characters U+E0000–E007F (invisible text models still read; the excerpt decodes it)	High	—
safety.bidiControl	Bidi overrides/isolates in a line with no right-to-left script	Medium	instruction wording on the same line
safety.zeroWidth	Zero-width characters, except ZWJ in emoji, ZWNJ/ZWJ next to Indic, Arabic-script, Myanmar or Khmer letters, and a leading BOM	Low	they split instruction wording (ig​nore previous instructions)
safety.hiddenHtml	Inline display:none, visibility:hidden, font-size:0, opacity:0, zero size with overflow:hidden, off-screen position, text colour equal to background, hidden attribute	Low	the hidden text contains instruction wording
safety.hiddenDocx	Runs with w:vanish, white text without shading or highlight, font size of 2 pt or less	Low	as above
safety.hiddenPdf	Letters in render mode 3 on a page with no image, below 1 pt, or outside the crop box	Low	as above
safety.instruction	Instruction wording aimed at a model: "ignore previous instructions", "do not tell the user", "if you are an AI", chat role tags (<\|im_start\|>, [INST], <<SYS>>), plus SafetyPolicy.AdditionalInstructionPhrases	Low	—
safety.ocrInstruction	Instruction wording in a part that produced OCR text	Medium	—

Invisible characters are removed before instruction wording is matched, so they cannot be used to split a phrase.

What happens at the threshold. ExtractionOptions.Safety is a SafetyPolicy; the default quarantines at High. When a part's worst finding reaches QuarantineAt:

• In a ZIP entry or EML attachment, or the EML message body, that part is replaced by a stub line ([bad.txt] withheld: High: ..., [Attachment: x] withheld: ..., [Message body] withheld: ...) and the rest is kept. The result is still a success. Nested containers are gated at the innermost part.
• In any other document, the whole result is IsSuccess == false with ErrorKind.Quarantined.

Withheld text is never in Text. It is reachable only through result.Quarantine.GetWithheldText() (or per part, Quarantine.Parts[i].GetWithheldText()). That is a deliberate escape hatch, for a human review queue, not for an AI model.

The gate scans the normalised text before MaxCharacters truncation, so a payload past the cut is still found.

var options = new ExtractionOptions
{
    Safety = new SafetyPolicy
    {
        QuarantineAt = SafetySeverity.High,     // null = report only
        TrustedSource = false,                  // true = report, never withhold (internal mail, say)
        AdditionalInstructionPhrases = ["approve this vendor"],
    },
    HiddenText = HiddenTextMode.Mark,           // Mark (default): "[hidden] ..." line | Drop | Keep
    FindingSink = f => logger.LogWarning("{Code} {Severity} in {Source}: {Reason}", f.Code, f.Severity, f.Source, f.Reason),
};
// SafetyPolicy.Off disables every detector.

HiddenText decides what hidden HTML, DOCX and PDF text looks like in Text. Behaviour change in 2.0.5: HTML hidden by inline style used to be extracted as if it were visible; it is now on its own [hidden] line.

Limits of the detectors:

• HTML: inline styles and the hidden attribute only. There is no CSS engine, so classes and stylesheets are not evaluated. Hidden text inside table cells is not separated.
• DOCX: direct run formatting only; text hidden through a paragraph or character style is not detected.
• PDF: invisible (render mode 3) text whose glyph centre lies inside an image's placement rectangle looks like the text layer of a searchable scan. With an OCR engine it is compared with what OCR reads in the image: if more than half of its words are not in the picture, it is hidden text (an instruction in it is then High). Without OCR it cannot be checked: it counts as visible and the page is reported as unverified (content.needsOcr, so IsPartial). Invisible text anywhere else on the page is always hidden. Hidden letters are regrouped into words and lines by position, so the text of such a page may be laid out differently from the rest.
• Flag emoji: the England, Scotland and Wales flags (the only standard emoji tag sequences) are ignored; other well-formed flag tag sequences are Low unless their tag letters spell instruction keywords.
• OCR: a document counts as OCR text if any part of it was OCR'd, so instruction wording on a normal page of a partly scanned PDF is scored Medium.
• Custom extractors get only the text-level detectors (tag, bidi, zero-width, instruction).

AI ingestion

For untrusted documents whose text goes to an AI model (RAG, agents), start from the profile:

var options = ExtractionOptions.ForAiIngestion();   // change anything with `with { ... }`
var result = extractor.GetText(name, bytes, options);

switch (result.SafetyVerdict)
{
    case SafetyVerdict.Quarantined:    reviewQueue.Add(name, result); break;              // nothing to index
    case SafetyVerdict.PartsWithheld:  reviewQueue.Add(name, result); Index(result.Text); break;
    case SafetyVerdict.ReviewRequired: reviewQueue.Add(name, result); break;              // index after review
    default:                           Index(result.Text); break;                         // still untrusted text
}

The profile sets SafetyPolicy.AiIngestion (quarantine at High, review at Medium), HiddenText = Drop (hidden text is scanned, then kept out of Text), and finite limits: 64 MB input, 256 MB decompressed and 1,000 entries shared across nested containers (ShareContainerBudget), 5,000,000 characters, 2,000 pages, 60 s.

Setting	Meaning
SafetyPolicy.QuarantineAt	Withhold the part (ZIP entry, EML attachment/body) or document whose worst finding reaches it
SafetyPolicy.RejectAt	Withhold the whole document when any finding, in any part, reaches it
SafetyPolicy.ReviewAt	SafetyVerdict.ReviewRequired when a finding reaches it; withholds nothing (also for TrustedSource)
SafetyPolicy.RetainHiddenText	With HiddenText = Drop, keep dropped hidden text in Quarantine for review. Off by default: findings then hold only a ≤200-character excerpt

SafetyVerdict: NoKnownSignals → Flagged (findings below ReviewAt) → ReviewRequired → PartsWithheld → Quarantined. Each finding also has a Category (Override, Role, Disclosure, ToolAction, Concealment, RoleMarker, Obfuscation, HiddenContent, InvisibleCharacters) and a rule-assigned Confidence (not a calibrated probability).

Escalation is deterministic: a visible instruction alone is Low; in OCR text Medium; obfuscated (lookalike letters, fullwidth/compatibility forms, letter spacing like i g n o r e, base64) Medium; concealment within 300 characters of an override, disclosure or tool-action instruction Medium (safety.correlatedInstructions); hidden + instruction, obfuscated + concealment, zero-width splitting, stray tag characters and bidi around instructions High.

Fraud and social engineering. Besides content aimed at the model, the gate looks for scams aimed at the human reader. Single cue words ("urgent", "invoice", "password") never fire; a scenario needs its cues within about 600 characters of each other in the same part (email body, attachment, page):

Code (safety.fraud.*)	Scenario
paymentRedirection	payment/invoice/supplier wording + new or changed bank details, IBAN or account number
credentialRequest	a request to confirm or enter a password or login details
accountThreat	account suspended/locked/closed + verify, click or sign in
codeRequest	"send me / share your" verification, one-time or access code (not "do not share this code")
giftCards	gift cards + send the codes or a photo of the card
pressurePayment	urgency + secrecy + a payment or purchase
extortion	a threat (recorded you, files encrypted) + cryptocurrency payment
macroLure	"enable editing / content / macros"

They are Medium (review under the AI profile; nothing is quarantined on fraud alone) with category Fraud. Urgency, secrecy or an authority figure (CEO, CFO, director) near a scenario raises its confidence to High. Every result also has a RiskScore (0–100) summing all findings by severity and confidence, for ranking and thresholds; it is a heuristic, not a probability. Genuine notices (a real bank-detail change, a real account lock) look the same as the scams and will be routed to review, which is usually what a finance or support team wants anyway.

What each format is checked for (anything not listed is not inspected, which is not the same as clean):

Format	Hidden content detected	Not inspected
HTML	inline hiding styles, simple <style> rules (tag, .class, #id, tag.class), hidden, <template>, same text/background colour, hidden table cells/rows	CSS cascade, combinators, @media, external stylesheets, layout
DOCX	w:vanish directly or via character/paragraph style, white and ≤2 pt text, tracked deletions	comments, footnotes, headers/footers (not extracted)
PDF	invisible text not over an image, invisible text over an image that OCR shows does not match it, text under 1 pt, text outside the crop box, images placed off the page	text colour, overlapping text; invisible text over an image when no OCR engine is set (reported as unverified)
PPTX	hidden slides, hidden shapes and groups	alt text, charts, SmartArt (not extracted); speaker notes are extracted as [Notes]
XLSX	hidden and very hidden sheets, zero-height rows, zero-width columns	cell comments (not extracted)
EML / ZIP / EPUB	each part gated separately (EML, ZIP); everything above applies to the parts	EPUB is gated as one document
Images (OCR)	OCR text is scanned like any text and its findings are scored higher	text the OCR engine does not see

Structured model input. GetModelInput returns segments with provenance instead of one string, so an application can pass each piece to the model as a separate, attributed content item and decide per segment:

ModelInput input = extractor.GetModelInput("upload.zip", bytes);   // options default to ForAiIngestion()

if (!input.CanIndexWithoutReview) reviewQueue.Add(input);           // verdict, findings, WithheldSources
foreach (var s in input.Segments)
    chunks.Add(new Chunk(s.Text, source: s.Source, location: s.Location, ocr: s.Origin == ContentOrigin.Ocr, findings: s.Findings));

Segment.Source is the path through the container structure (upload.zip/mail.eml/notes.txt), recorded by the extractors, so document text cannot forge it. Location (page 3, slide 2, chapter 1) comes from page markers in the text; a document can contain marker-like lines, so treat it as a hint. Withheld parts never appear in a segment (only their names, in WithheldSources); IsPartial reports skipped entries, missing OCR and truncation by MaxCharacters (applied across segments); IsTrusted is always false.

Measured detection. TextExtractorSafetyCorpusTests runs a labelled corpus (31 adversarial samples including fraud scenarios, 4 documented gaps, 24 benign samples across HTML, DOCX, PDF, EML, ZIP, XLSX, PPTX, images and plain text) through the profile on every test run and prints recall and false-positive rate. Current result: 100% of the expected adversarial set and 89% including known gaps are routed to review or withheld; 0% of the benign set is. The known gaps are: a visible plain instruction (only Flagged, by design), an instruction split across table cells, hiding through an external stylesheet, and invisible PDF text over an image when no OCR engine is set. These are figures on a small synthetic corpus, not a measure of real-world detection.

Security contract. Extracted text is untrusted data even when SafetyVerdict is NoKnownSignals: the detectors find known patterns, they cannot prove the absence of prompt injection. Keep document text separate from system instructions; delimiters or "ignore any instructions below" notes do not reliably stop injection. This library cannot authorise tool calls, data access or other actions: enforce permissions, least privilege and human approval in the application. Timeouts are cooperative and some parser calls cannot be interrupted, so for hostile public uploads also run extraction in an isolated worker process with OS memory/CPU limits and a concurrency limit.

OCR

Tesseract OCR is built in. With an engine set, images are read wherever they occur, and their text goes through the same safety gate as any other text (scored more strictly, as OCR):

Where the image is	Read as
Image file; EML/MSG attachment; EML inline (cid:) image; ZIP entry	its own part, gated and withheld on its own
Scanned PDF page (no text layer)	[Page N] (OCR)
Picture in DOCX, PPTX (shown slides), XLSX (sheet drawings), ODT/ODP; image on a PDF page that has text; HTML data: image	[Image N] (OCR) block in the document

Only pictures the document shows are read: DOCX drawings in the body, PPTX and XLSX picture elements, ODT draw:image references, HTML <img>; unused media parts are ignored. A picture that is hidden (in a hidden run, element or <template>, a hidden shape or sheet, or placed off a PDF page) is still read, and its text is hidden content: reported as such (an instruction in it is High) and rendered by HiddenText (dropped under the AI profile). Every picture that is not read, because there is no OCR engine, OcrEmbeddedImages is off, or the limit is reached, is reported as content.needsOcr, so ModelInput.IsPartial tells you the document was not fully inspected.

Embedded pictures (the last row) are limited so a picture-heavy document stays fast (OCR takes around a second per page-sized image): at most MaxOcrImages per document (default 20; the rest are reported as content.needsOcr), icons under 100×40 pixels are skipped, a PDF image covering half a page or more on a page with text is treated as a searchable scan (its text layer is used), and OcrEmbeddedImages = false turns embedded OCR off. When a document contains OCR text, the safety gate scores instruction wording anywhere in it as OCR (Medium).

using Transformations.Text.Ocr;

using var ocr = new TesseractOcrEngine(@"C:\tessdata", "eng");     // one per process; dispose at shutdown
var result = new TextExtractor().GetText("scan.pdf", bytes, new ExtractionOptions { OcrEngine = ocr });

• Windows x64/x86. The native Tesseract 5.5 / Leptonica 1.85 binaries come with the TesseractOCR package and need the Microsoft Visual C++ 2022 runtime. Elsewhere IsAvailable is false, no native code is loaded, and images are reported as needing OCR (content.needsOcr / ExtractionErrorKind.NeedsOcr).
• Language data is not bundled. Download eng.traineddata (and others) from https://github.com/tesseract-ocr/tessdata_fast. A missing file makes IsAvailable false; UnavailableReason says why.
• OCR text is untrusted. Every OCR result adds a content.ocr warning and goes through the safety gate with stricter scoring.
• Limits. Image size is read from the header (or, in PDFs, from the image dictionary) before decoding and checked against ExtractionOptions.MaxImagePixels (default 64 MP). Tesseract's recognition call cannot be interrupted; cancellation and timeouts are checked before each image.
• Engines are pooled: one TesseractOcrEngine can be shared across threads.

Extension vs content

The extension picks the extractor. If that extractor fails and the magic bytes identify a different format, the document is retried once with that format's extractor (RetryWithDetectedFormat, default on). The retry never targets ZIP, EML or EPUB (a disguised archive is reported, not unpacked), never follows a safety-limit failure, and shares the original Timeout. Independently, any mismatch between extension and a strong binary signature (PDF, OOXML, OpenDocument/ZIP, EPUB, OLE2) is reported as content.typeMismatch, and GetTextWithMetadata adds file.detectedFormat. Sibling extensions (.xlsm/.xlsx, .docm/.docx) and text heuristics (a .txt starting with Date:) are not mismatches. Format-specific metadata follows the extractor that ran, so a PDF named upload.txt still gets pdf.* keys.

Security limits are intentionally conservative and apply before routing content deeper into the pipeline:

• MaxInputBytes rejects oversized input before any parser runs (default 256 MB).
• MaxDecompressedBytes caps total decompressed ZIP entries, EPUB chapter and package-document bytes, and decoded EML attachment bytes. DOCX, XLSX/XLSM, PPTX, ODT and ODP files are ZIPs too: before parsing, every part is decompressed into a byte counter, so a file whose headers under-report its size is still caught.
• MaxContainerEntries caps ZIP entries, EPUB spine items, EML attachments, and the number of parts in an Office/OpenDocument file.
• Limits apply per document. An Office file inside a ZIP or an email gets its own MaxDecompressedBytes budget; if it goes over, that file is skipped with a warning and the rest of the container is still extracted.
• Nested ZIPs are not recursively extracted by the default pipeline; they are reported as unsupported entries.
• Binary-looking unknown files are not extracted via the text fallback inside ZIP/EML containers.

TableMode	Output
KeyValue (default)	Header: Value \| Header2: Value2 per row
Markdown	GitHub-flavored pipe tables
Csv	CSV rows
Omit	Tables suppressed entirely

---

TextExtractorBuilder

var extractors = TextExtractor.CreateDefaultExtractors().ToList();
var builder    = new TextExtractorBuilder(extractors);

builder.Add(new WordPerfectExtractor())  // inserted before catch-all TxtExtractor
       .Replace<PdfExtractor, MyPdfExtractor>()
       .Disable<LogExtractor>();

var pipeline = new TextExtractor(builder.Build());

Method	Description
Add(extractor)	Adds instance before TxtExtractor catch-all
Add<T>()	Adds parameterless instance
Replace<TOld>(replacement)	Replaces first extractor of type TOld in-place
Replace<TOld, TNew>()	Replaces with new TNew()
Disable<T>()	Removes all extractors of type T
Build()	Returns the configured pipeline

---

TextChunker — RAG chunking

// Fixed character windows with overlap — snaps to nearest word boundary
var chunks = TextChunker.ChunkByCharacters(text, maxLength: 500, overlap: 50);

// Sentence-boundary aware
var chunks = TextChunker.ChunkBySentences(text, maxLength: 800, overlap: 100);

// Paragraph-boundary aware
var chunks = TextChunker.ChunkByParagraphs(text, maxLength: 1000);

// One chunk per Markdown section — carries HeadingPath breadcrumb
var sections = MarkdownStructureExtractor.ExtractSections(markdownText);
var chunks   = TextChunker.ChunkBySections(sections, maxLength: 1500);
// chunks[i].HeadingPath → ["Chapter 1", "Section 2", ...]

// Token-budget aware — pass your model's tokenizer; no tokenizer bundled
var chunks = TextChunker.ChunkByTokens(text, maxTokens: 512,
    tokenCounter: t => myTokenizer.CountTokens(t),
    overlapTokens: 32);

TextChunk record:

Property	Type	Description
Index	int	Zero-based position in the sequence
Text	string	Chunk content
StartOffset	int	Character offset in the original text
EndOffset	int	Exclusive end offset
PageNumber	int?	1-based source page, when available
HeadingPath	IReadOnlyList<string>?	Breadcrumb from ChunkBySections; null otherwise

---

JsonSchemaValidator

bool isValid  = JsonSchemaValidator.ValidateJson(rawJson, schemaJson);
var  errors   = JsonSchemaValidator.ListSchemaErrors(rawJson, schemaJson);
string clean  = JsonSchemaValidator.NormalizeJson(rawJson);  // strips ```json fences
bool changed  = JsonSchemaValidator.HasChanged("{\"a\":1}", "{\"a\":2}");

---

MarkdownStructureExtractor

string normalized = MarkdownStructureExtractor.NormalizeMarkdown(markdown);
var    headings   = MarkdownStructureExtractor.BuildHeadingMap(markdown);
var    sections   = MarkdownStructureExtractor.ExtractSections(markdown);
bool   changed    = MarkdownStructureExtractor.HasChanged(oldMarkdown, newMarkdown);

---

DocumentContent — unified normalize/compare

string normalized = DocumentContent.Normalize(content, DocumentFormat.Markdown);
bool   changed    = DocumentContent.HasChanged(oldContent, newContent, DocumentFormat.Json);

Supported DocumentFormat values: PlainText, Markdown, Json

---

CsvDataProfiler

var profile = CsvDataProfiler.ProfileCsv(csvText);
// profile.RowCount, .ColumnCount, .Headers, .ColumnTypes, .Distributions, .Issues

var types  = CsvDataProfiler.InferColumnTypes(csvText);
var issues = CsvDataProfiler.DetectDataIssues(csvText);

CsvColumnType values: Empty, Boolean, Integer, Decimal, DateTime, Guid, String, Mixed

---

Custom extractors

Implement ITextExtractor to add support for a new format:

public class WordPerfectExtractor : ITextExtractor
{
    public bool CanHandle(string extension) =>
        extension.Equals(".wpd", StringComparison.OrdinalIgnoreCase);

    public string ExtractText(byte[] data) { ... }

    // Optional: override for per-call options (backward-compatible default provided)
    public string ExtractText(byte[] data, ExtractionOptions? options) { ... }
}

// Register via builder:
services.AddTextExtractor(b => b.Add(new WordPerfectExtractor()));

// Or ad-hoc:
var pipeline  = TextExtractor.CreateDefaultExtractors().ToList();
var extractor = new TextExtractor(new[] { new WordPerfectExtractor() }.Concat(pipeline));

---

Supported Formats

Format	Extensions	Notes
PDF	.pdf	Page text + document properties; MaxPages, StartPage, EndPage, IncludePageMarkers
Word	.docx	Paragraphs + tables; TableMode, IncludePageMarkers
PowerPoint	.pptx	Slide titles, body text, speaker notes; MaxPages, StartPage, EndPage
Excel	.xlsx, .xls, .xlsm	TableMode
HTML	.html, .htm	TableMode
OpenDocument Text/Presentation	.odt, .odp	TableMode
RTF	.rtf	Self-contained parser; no extra dependency
EPUB	.epub	ZIP + OPF spine + HTML; MaxPages, StartPage, EndPage, IncludePageMarkers, MaxDecompressedBytes, MaxContainerEntries
Markdown	.md, .markdown
CSV	.csv
Outlook email	.msg	Subject, sender, date
Internet email	.eml	Full metadata + attachment text; unsupported/failed attachments produce warnings
JSON	.json
XML / Config	.xml, .config
YAML	.yaml, .yml
ZIP archive	.zip	Routes known entries through the pipeline; MaxDecompressedBytes and MaxContainerEntries guards
Log / Output	.log, .out
Source code	.cs, .py, .js, .ts, .cpp, .h, .java, .ps1, .sh, .go, .rs, .sql, .rb, .php, .swift, .kt, .scala, .fs, .vb, .lua, .r, .m, .dart
Plain text (fallback)	any

---

Dependencies

• CsvHelper — CSV parsing
• DocumentFormat.OpenXml — DOCX / XLSX support
• ExcelDataReader — legacy Excel reading
• HtmlAgilityPack — HTML parsing
• JsonSchema.Net — JSON schema validation
• Markdig — Markdown processing
• Microsoft.Extensions.DependencyInjection.Abstractions — DI registration extension
• Microsoft.Extensions.Logging.Abstractions — structured logging support
• MimeKit — EML parsing
• MSGReader — Outlook MSG parsing
• PdfPig — PDF text extraction
• YamlDotNet — YAML parsing

RTF and EPUB parsing use BCL only — no additional dependencies.

---

Changelog

2.0.5 (current)

Fraud signals: combination-based social-engineering scenarios (safety.fraud.*: payment redirection, credential and code requests, account threats, gift cards, pressure payments, extortion, macro lures) and a 0–100 RiskScore on results and ModelInput.

Images everywhere: with an OCR engine, EML inline images, MSG attachments (now extracted, each gated on its own) and pictures embedded in DOCX/PPTX/XLSX/ODT, PDF text pages and HTML data: URIs are OCR'd (OcrEmbeddedImages, MaxOcrImages). content.ocr is reported only when OCR produced text.

Packaging: Transformations.Text.Ocr is merged into this package (same Transformations.Text.Ocr namespace, so code using TesseractOcrEngine compiles unchanged). Reference Transformations.Text only; drop any Transformations.Text.Ocr package reference.

Safety gates and quarantine (2.0.5.1):

• New safety gate: ExtractionOptions.Safety (SafetyPolicy), HiddenText (HiddenTextMode), FindingSink; Findings and Quarantine on both result types; ExtractionErrorKind.Quarantined; warning codes safety.finding and safety.withheld. See Safety gates and quarantine.
• Behaviour change: quarantine at High is on by default. Documents with Unicode tag characters, or hidden HTML/DOCX/PDF text containing instruction wording, now fail with Quarantined; in ZIP and EML only the offending part is withheld.
• Behaviour change: HTML hidden by inline style, DOCX hidden runs and invisible PDF text are now on their own [hidden] ... line (HiddenText = Keep restores the old output).

Review fixes (2.0.5.1):

• Behaviour change: table header rows follow the format's marker (ExtractionOptions.TableHeaders, default Auto): HTML <th>/<thead>, DOCX "repeat header row" or the table style's header-row flag, ODT table:table-header-rows. An unmarked first row is now data instead of labels; TableHeaders = FirstRow restores the old behaviour. ODT header rows, previously dropped, are now read. Markdown output gives a header-less table an empty header row.
• HTML tables nested in a cell no longer add their rows to the outer table.
• ZIP and EPUB count every decompressed byte against MaxDecompressedBytes, including entries that fail, and stop once it is spent. EPUB package documents count too.
• EPUB manifest hrefs are percent-decoded, lose #fragment/?query, and resolve ./...
• Scanned PDF pages check the image size the PDF declares against MaxImagePixels before converting it.
• ExtractionOptions.CancellationToken cancels stream reads (sync and async). HtmlExtractor.ExtractFromStream, the stream overloads of DocxExtractor, PptxExtractor, OdtExtractor and the default ITextExtractor.ExtractText(Stream) enforce MaxInputBytes and, for Office/OpenDocument, the package limits.
• JSON, YAML, XML, Markdown and MSG check cancellation, and the facade checks again after any extractor returns, so a parse that overran Timeout is TimedOut, not a success.
• Text is detected as email only with at least two well-known headers followed by a blank line; Date: tomorrow in a .txt stays plain text. An Outlook MSG named .doc (or a .msg that is not one) is a content.typeMismatch.
• MarkdownStructureExtractor ignores headings inside fenced code blocks and makes repeated anchors unique (setup, setup-1, ...).
• GetSupportedExtensions() lists image formats; ODP honours StartPage/EndPage/MaxPages; a DOCX page break inside a paragraph puts the [Page N] marker where the break is.
• EML with a whitespace-only plain-text part uses the HTML part. MSG HTML bodies get the safety and hidden-text checks.
• CSV with repeated header names keeps every column. RTF \'hh escapes are decoded with \ansicpg and the font's \fcharset (including Shift-JIS, GBK, Big5, Korean), so \'80 in Windows-1252 is €.
• TextChunker sentence, paragraph and token chunks are exact slices of the input: text[StartOffset..EndOffset] == Text, with the original separators.
• Nesting limits: YAML deeper than 64 levels, XML deeper than 256 elements and HTML deeper than 512 elements fail with LimitExceeded instead of overflowing the stack; YAML aliases that expand past 1,000,000 values ("billion laughs") fail the same way.
• Safety gate: hidden HTML table cells, rows (<tr>, <tbody>) and hidden elements inside cells are detected and follow HiddenText. Bidi controls are judged locally: an override over opposite-direction letters or digits is suspicious whatever else is on the line, and any control whose span covers instruction wording is High.
• EML attachments that fail part-way count against MaxDecompressedBytes; once it is spent, no further attachment is decoded.
• Source code keeps its indentation and spacing; all text keeps leading indentation (only runs of 3+ spaces inside a line are collapsed).
• CsvDataProfiler: an all-text table needs evidence before its first row counts as a header (alice,red / bob,blue is data); new ProfileCsv(csv, hasHeader) overload; generated header names never collide (Name,Name,Name_2).
• OpenDocument text and presentations are recognised by their mimetype entry, so an ODT without an extension is read as ODT.
• JsonSchemaValidator: empty input no longer validates against an object schema, and HasChanged("", "{}") is true.
• TesseractOcrEngine.Dispose can no longer race an in-flight call into leaking a native engine.
• Flag emoji (England, Scotland, Wales) no longer quarantine a document; stray tag characters still do.
• CodeExtractor strips copyright /* */ comments only from the leading file header, never from string literals or later code.
• WarningSink/FindingSink never report a first attempt that failed and was retried with the detected format.
• PDF invisible text is exempt as a scan's text layer only where it lies over an image.
• Transformations.Tests runs on .NET 8, 9 and 10, and includes mutation fuzzing of every built-in format.
• AI ingestion: ExtractionOptions.ForAiIngestion(), SafetyPolicy.AiIngestion, ReviewAt, RejectAt, RetainHiddenText, ShareContainerBudget, SafetyVerdict on results, SafetyFinding.Category/Confidence; detection of obfuscated (lookalike, fullwidth, letter-spaced, base64) and correlated instructions; tool-action, disclosure, role and concealment rules; hidden content from HTML <style> rules and <template>, DOCX styles and tracked deletions (which no longer leak into Text), PPTX hidden slides/shapes, XLSX hidden sheets/rows/columns. See AI ingestion.
• Dependencies: HtmlAgilityPack 1.13.1, JsonSchema.Net 9.4.1, Markdig 1.4.2, Microsoft.Extensions.Logging.Abstractions 8.0.3, MimeKit 4.18.2, MSGReader 6.2.0, PdfPig 0.1.16, YamlDotNet 17.1.0.

Safety limits and cancellation:

• ExtractionOptions.CancellationToken and ExtractionOptions.Timeout. The built-in extractors check the token while they work, so a cancelled parse stops. GetTextBatchAsync and the async methods pass their cancellationToken through to the parser as well.
• ExtractionOptions.MaxInputBytes (default 256 MB) and ExtractionErrorKind.TooLarge.
• DOCX, XLSX/XLSM, PPTX, ODT and ODP are checked against MaxDecompressedBytes / MaxContainerEntries before parsing: new ExtractionErrorKind.LimitExceeded and ExtractionLimitExceededException.
• EPUB package documents and content sniffing no longer read an archive entry of any size.
• ExtractionErrorKind.Binary. Behaviour change: binary files that previously returned IsSuccess == true with empty text now return a failure with this kind.
• CsvExtractor and RtfExtractor now have the ExtractText(byte[], ExtractionOptions?) overload.

Extension/content handling:

• Retry with the content-detected format when the extension's extractor fails (RetryWithDetectedFormat); new content.typeMismatch warning and file.detectedFormat metadata. Failures now carry Warnings too.
• OLE2 files are identified by stream name (.msg, .doc, .xls, .ppt) instead of always .msg; extensionless legacy .xls is now extracted.
• Invalid ExtractionOptions values (negative limits, EndPage < StartPage, zero Timeout) throw ArgumentOutOfRangeException.
• Wrapped exceptions (PDF) are classified by their cause, so a corrupt PDF reports Corrupted.

Extraction fixes:

• TextChunker.ChunkByCharacters no longer skips text after a chunk snaps back to a word boundary, and no longer emits a redundant tail chunk; ChunkBySentences no longer splits after Mr., Dr., vs. and similar.
• CodeExtractor copyright stripping can no longer delete code between two comments.
• XmlExtractor keeps text in mixed-content elements (<p>Hello <b>world</b></p>).
• RtfExtractor handles surrogate pairs (emoji) instead of failing.
• HtmlExtractor turns <br> into a line break and honours <meta charset> / XML encoding declarations.
• Single-row tables (DOCX, HTML, ODT), single-row sheets (Excel) and header-only CSVs are no longer dropped.
• PptxExtractor reads text inside grouped shapes and tables.
• GetTextBatchAsync streams results as they become ready (in input order, bounded read-ahead), and cancels and awaits outstanding work when the caller stops enumerating.

Behaviour changes:

• PPTX and ODP slide markers ([Slide N]) now follow IncludePageMarkers like PDF/DOCX/EPUB; previously they were always emitted. Set IncludePageMarkers = true to keep them.

2.0.4

New formats (zero new dependencies):

• RtfExtractor — Rich Text Format .rtf; self-contained state machine parser handles control words, hex/Unicode escapes, and skips metadata groups (\fonttbl, \info, etc.)
• EpubExtractor — EPUB .epub; reads OPF spine from ZIP, extracts XHTML chapters via the HTML extractor; supports MaxPages, StartPage, EndPage, IncludePageMarkers

ExtractionOptions expanded:

• MaxPages, StartPage, EndPage — page/slide/chapter windowing for PDF, PPTX, EPUB
• IncludePageMarkers — inserts [Page N], [Slide N], [Chapter N] markers in DOCX, PDF, PPTX, EPUB output
• MaxDecompressedBytes — ZIP bomb guard (default 512 MB); honoured by ZipExtractor, EpubExtractor, and EML attachment extraction
• MaxContainerEntries — archive/attachment entry guard (default 10,000); honoured by ZIP, EPUB, and EML attachment extraction
• WarningSink and result .Warnings — report non-fatal partial extraction issues such as unsupported entries, missing EPUB chapters, failed attachments, and container limits

TableMode enum — controls table rendering across all table-capable extractors:

• KeyValue (default) — Header: Value | ... key-value rows
• Markdown — GitHub-style pipe tables
• Csv — CSV rows
• Omit — suppress tables entirely

ITextExtractor interface — new default method ExtractText(byte[], ExtractionOptions?) so existing custom extractors remain source-compatible without changes

TextChunk provenance fields:

• PageNumber? — 1-based source page, set by page-aware extractors
• HeadingPath? — breadcrumb of heading titles from ChunkBySections; ChunkBySections now maintains a heading stack and stamps every chunk with its full ancestor path

TextExtractor.DetectFormat(byte[]) static method — sniffs content type from magic bytes and returns the file extension (.pdf, .docx, .rtf, .epub, .eml, etc.) or null

TextExtractorBuilder class — fluent, type-safe pipeline builder available for both DI and direct construction; exposes Add, Add<T>, Replace<TOld>, Replace<TOld, TNew>, Disable<T>, and Build

Content-type sniffing extended — now detects RTF ({\rtf), EML (RFC 2822 headers), and EPUB (mimetype entry in ZIP)

---

4.0.0

New formats:

• PptxExtractor — slide titles, body text, and speaker notes from .pptx files
• ZipExtractor — routes each ZIP entry through the full extraction pipeline; handles nested text, DOCX, XLSX, PPTX etc.

Extraction result enrichment:

• ExtractionResult.ExtractorName — which extractor handled the file
• ExtractionResult.Duration — wall-clock time for the extraction
• ExtractionMetadataResult gains the same two properties plus extraction.extractor and extraction.durationMs keys in Metadata

Format-specific document properties (via GetTextWithMetadata):

• PDF: pdf.title, pdf.author, pdf.subject, pdf.keywords, pdf.creator, pdf.pageCount
• DOCX: docx.title, docx.author, docx.subject, docx.created, docx.modified, docx.revision
• MSG: msg.subject, msg.from, msg.date

DOCX table extraction — tables are now emitted as Header: Value | Header2: Value2 rows rather than a flat text stream

ILogger integration — TextExtractor(ILogger<TextExtractor>) constructor; DI injects the logger automatically; structured log events at Debug and Warning levels

Content-type sniffing — magic-byte fallback; detects PDF, DOCX/XLSX/PPTX (ZIP + [Content_Types].xml), MSG (OLE2 magic), HTML

TextChunker.ChunkByCharacters — chunk boundaries now snap to word boundaries

TextChunker.ChunkByTokens — splits at paragraph boundaries using a caller-supplied token counter; supports token overlap

---

3.0.0

Breaking changes:

• Compare on DocumentContent, MarkdownStructureExtractor, and JsonSchemaValidator replaced by HasChanged (same semantics, clearer name)
• CSV rows now emitted in document order (previously reversed)
• Binary files produce IsSuccess=true, Text="" rather than a [Skipped...] placeholder
• All extractor classes are now public

New:

• IDocumentTextExtractor facade interface
• TextExtractor(IEnumerable<ITextExtractor>) constructor
• TextExtractor.CreateDefaultExtractors() static method
• ServiceCollectionExtensions.AddTextExtractor()
• Stream overloads and async API
• MsgExtractor — Outlook .msg support
• TextChunker — ChunkByCharacters, ChunkBySentences, ChunkByParagraphs, ChunkBySections
• CodeExtractor extended to 22 language extensions

---

License

MIT — see repository for details.