WebFlux 0.1.3

WebFlux

A .NET SDK for preprocessing web content for RAG (Retrieval-Augmented Generation) systems.

Overview

WebFlux processes web content into chunks optimized for RAG systems. It handles web crawling, content extraction, and intelligent chunking with support for multiple content formats.

Installation

dotnet add package WebFlux

Quick Start

using WebFlux;
using Microsoft.Extensions.DependencyInjection;

var services = new ServiceCollection();

// Register your AI service implementations
services.AddScoped<ITextEmbeddingService, YourEmbeddingService>();
services.AddScoped<ITextCompletionService, YourLLMService>(); // Optional

// Add WebFlux
services.AddWebFlux();

var provider = services.BuildServiceProvider();
var processor = provider.GetRequiredService<IWebContentProcessor>();

// Process a website
await foreach (var result in processor.ProcessWithProgressAsync("https://example.com"))
{
    if (result.IsSuccess && result.Result != null)
    {
        foreach (var chunk in result.Result)
        {
            Console.WriteLine($"Chunk {chunk.ChunkIndex}: {chunk.Content}");
        }
    }
}

Features

• Interface-Based Design: Bring your own AI services (OpenAI, Anthropic, Azure, local models)
• Multiple Chunking Strategies: Auto, Smart, Semantic, Intelligent, MemoryOptimized, Paragraph, FixedSize
• Content Formats: HTML, Markdown, JSON, XML, PDF
• Web Standards: robots.txt, sitemap.xml, ai.txt, llms.txt, manifest.json
• Streaming: Process large websites with AsyncEnumerable
• Parallel Processing: Concurrent crawling and processing

Chunking Strategies

Core Interfaces

You provide implementations for AI services:

public interface ITextEmbeddingService
{
    Task<double[]> GetEmbeddingAsync(string text, CancellationToken cancellationToken = default);
}

public interface ITextCompletionService // Optional, for content reconstruction
{
    Task<string> CompleteAsync(string prompt, CancellationToken cancellationToken = default);
}

WebFlux handles the rest: crawling, extraction, analysis, and chunking.

Configuration

var options = new CrawlOptions
{
    MaxDepth = 3,
    MaxPages = 100,
    RespectRobotsTxt = true,
    UserAgent = "MyBot/1.0"
};

var chunkOptions = new ChunkingOptions
{
    Strategy = "Auto",
    MaxChunkSize = 512,
    OverlapSize = 64
};

await foreach (var result in processor.ProcessWithProgressAsync(url, options, chunkOptions))
{
    // Handle results
}

Documentation

• Tutorial - Step-by-step guide with practical examples
• Architecture - System design and pipeline
• Interfaces - API contracts and implementation guide
• Chunking Strategies - Detailed strategy guide

License

MIT License - see LICENSE file for details.

Support

• Issues: GitHub Issues
• Package: NuGet