OKF4net.Catalog 0.5.0

OKF4net.Catalog

A catalog of local Open Knowledge Format (OKF) v0.2 knowledge bundles for .NET: a hot-reloadable catalog.json manifest naming one or more bundles as sources, and a resolver that searches every enabled source and returns results grouped by source.

OKF is the Open Knowledge Format — a directory tree of markdown files with YAML frontmatter (see OKF4net for the format core). catalog.json is not an OKF concept and is not part of the OKF spec: it is an OKF4net manifest, a small piece of catalog configuration that points at OKF bundles from the outside. Nothing about the manifest format is portable to another OKF implementation.

This package references only OKF4net (the format core) — no third-party runtime dependencies.

What it does

• Loads and validates catalog.json. FileKnowledgeCatalog parses the manifest, validates every enabled source's path against the catalog root (rejecting absolute paths, paths that escape the root, missing directories, and reparse points in between), and publishes the result as an immutable, versioned KnowledgeCatalogSnapshot. An invalid initial manifest throws CatalogException (fail-fast at startup).

• Hot-reloads, best-effort. A debounced FileSystemWatcher on the manifest file triggers automatic reloads. This is best-effort only — OS/ filesystem/container layers can miss or duplicate watcher events. Call IKnowledgeCatalog.ReloadAsync() directly whenever you need a reliable, synchronous guarantee that an edit has been picked up. A reload is errors-as-data: a malformed or invalid replacement manifest leaves the current snapshot untouched and records the reject reasons in LastReloadDiagnostics instead of throwing.

• Searches every enabled source, with a selectable ranking strategy. IKnowledgeResolver fans a query out across every enabled KnowledgeCatalogSource (using the same ConceptSearch scorer the OKF4net.Agents tools use). How the results come back is your choice:

  • GroupedBySource (default) — each source's own ranked results concatenated, source by source, in priority order. No fusion or dedup.
  • Merged — one cross-source ranking by descending score, with priority as a tie-break only.
  • PriorityWeighted — one cross-source ranking by source priority first, score only within a priority tier.

  Both merged strategies also collapse two manifest entries that resolve to the same directory (searching that bundle once, not twice) and accept an optional fairness quota that interleaves sources so one prolific source cannot crowd out the rest of a budget-truncated result.

Minimal catalog.json

{
  "version": 1,
  "sources": [
    { "id": "products", "path": "./bundles/products", "priority": 10, "enabled": true },
    { "id": "support", "path": "./bundles/support", "priority": 0, "enabled": true }
  ]
}

• id — unique within the manifest, a single OKF concept-id segment.
• path — relative to the manifest's own directory; must resolve inside the catalog root.
• priority (optional, default 0) — higher priority sources are searched first and their passages appear first in a grouped result.
• enabled (optional, default true).
• role (optional, default "knowledge") — "knowledge" (read-only, searched by the resolver) or "memory" (writable, scoped by tier — see below); any other string is rejected.
• tier — required when role is "memory", one of "session", "user", or "tenant"; not allowed otherwise.

Quick start

using OKF4net.Catalog;

var options = new KnowledgeCatalogOptions
{
    CatalogFilePath = "./config/catalog.json",
    CatalogRoot = "./config",
};

using var catalog = new FileKnowledgeCatalog(options);
IKnowledgeResolver resolver = new GroupedKnowledgeResolver(catalog);

KnowledgeContext result = await resolver.SearchAsync(new KnowledgeQuery("refund policy"));

foreach (var passage in result.Passages)
{
    Console.WriteLine($"[{passage.SourceId}] {passage.ConceptId} ({passage.Score}): {passage.Excerpt}");
}

foreach (var diagnostic in result.Diagnostics)
{
    Console.WriteLine($"[{diagnostic.Code}] {diagnostic.Message}");
}

CatalogRoot is a strict containment boundary. When constructing KnowledgeCatalogOptions directly, its shared path components with CatalogFilePath must have identical spelling, including case. A case-variant root is rejected as OutsideRoot even on a case-insensitive filesystem. AddKnowledge(o => o.AddCatalogFile(...)) derives both values from one resolved path and therefore satisfies this requirement automatically.

KnowledgeContext is deliberately never a bare string: Passages (grouped by source, in source-priority then per-source descending-score order) and Diagnostics (e.g. NoEnabledSources, SourceUnavailable, NoMatches) let a caller distinguish "no results" from "a source failed" from "no source is enabled" without parsing text.

Scoped memory (role: "memory")

A role: "memory" source is written by capture (e.g. OkfContextProviderOptions.CaptureTier in OKF4net.Agents), not searched by IKnowledgeResolver — it feeds an IMemoryStore instead. Configure one source per tier you need:

{
  "version": 1,
  "sources": [
    { "id": "kb", "path": "./bundles/products", "role": "knowledge" },
    { "id": "mem-user", "path": "./memory/user", "role": "memory", "tier": "user" },
    { "id": "mem-tenant", "path": "./memory/tenant", "role": "memory", "tier": "tenant" },
    { "id": "mem-session", "path": "./memory/session", "role": "memory", "tier": "session" }
  ]
}

using OKF4net.Catalog;
using OKF4net.Catalog.Hosting;

services.AddKnowledge(o => o.AddCatalogFile("./config/catalog.json"));
services.AddMemory();

// Elsewhere:
IMemoryStore memory = provider.GetRequiredService<IMemoryStore>();
await memory.DeleteScopeAsync(scope, MemoryTier.Session); // e.g. when a conversation ends

There is no code-level distinction between ephemeral and persistent tiers — every role:"memory" source's path, like every other source's, must be relative to the manifest directory and resolve inside the catalog root (CatalogPathResolver.TryResolve rejects absolute paths, paths that escape the root, and reparse points anywhere along the way), so a source cannot point directly at an OS temp directory or a symlink into one. "Ephemeral" therefore isn't a per-source path trick; it's one of two real choices:

• Run the whole catalog root on ephemeral storage (e.g. a container's tmpfs mount or ephemeral volume) — every source under it, including a session-tier source like mem-session above at its own ordinary relative path, is then ephemeral by construction. The catalog root itself is exempt from the reparse-point walk, so this is the one place a mount point is fine.
• Treat any tier's subtree as revocable at will via IMemoryStore.DeleteScopeAsync — nothing purges automatically, but nothing stops a host from calling it the moment a conversation ends.

V1 limitation: OKF4net.Catalog.Hosting's AddMemory() resolves the set of role:memory sources once, at first IMemoryStore resolution from the container, and does not pick up a source added/removed/edited afterward (including via IKnowledgeCatalog.ReloadAsync()) — see AddMemory's own XML doc for the full explanation. Per-scope path resolution (the tenant/user/session segments) stays fully live on every call; only the fixed set of configured tiers is frozen.

Choosing a ranking strategy

Set a default for the whole host, and override it per query where needed:

using OKF4net.Catalog;
using OKF4net.Catalog.Hosting;

services.AddKnowledge(o =>
{
    o.AddCatalogFile("./config/catalog.json");
    o.DefaultResolverStrategy = KnowledgeResolverStrategy.Merged;
    o.DefaultFairnessQuota = 2;   // optional; null (the default) disables it
});

// Per-query override, through the same injected IKnowledgeResolver:
var context = await resolver.SearchAsync(new KnowledgeQuery("refund policy")
{
    ResolverStrategy = KnowledgeResolverStrategy.PriorityWeighted,
});

DefaultResolverStrategy defaults to GroupedBySource, so upgrading changes no existing deployment's result ordering until you opt in.

A fairness quota caps how many consecutive passages one source may contribute before another source's next-best passage is pulled ahead. It reorders and never drops, so it matters only to consumers that truncate early — an agent context provider spending a token budget top-down, for instance, which would otherwise let one source's whole run consume the budget.

Choosing source visibility

Restrict which sources a caller may see, per host default or per query:

services.AddKnowledge(o =>
{
    o.AddCatalogFile("./config/catalog.json");
    o.DefaultSourceVisibilityPolicy = (scope, source) =>
        // Fails CLOSED, not open: a caller with no TenantId (KnowledgeAccessScope.Local,
        // the default when a host resolves no scope at all) sees NOTHING here -- an empty
        // "" ?? fallback would make StartsWith("") true for every source, silently exposing
        // every tenant's catalog to an unauthenticated/unscoped caller. Matches a source
        // named EXACTLY as the tenant, or "tenantId-" followed by anything -- the "-" is an
        // explicit segment separator, not just a prefix, so a short tenant id (e.g. "acme")
        // can never accidentally match an unrelated one (e.g. "acmeland-kb"). Do not drop
        // the "-" to make a bare-named source match "more easily": that reopens exactly
        // that collision.
        scope.TenantId is { Length: > 0 } tenantId
        && (source.Id == tenantId || source.Id.StartsWith(tenantId + "-", StringComparison.Ordinal));
});

// Per-query override, through the same injected IKnowledgeResolver:
var context = await resolver.SearchAsync(new KnowledgeQuery("refund policy")
{
    Scope = new KnowledgeAccessScope(tenantId: "acme"),
    PermittedSourceIds = new HashSet<string> { "acme-support", "acme-billing" },
});

Two mutually exclusive mechanisms — setting both on the same query throws:

• PermittedSourceIds — a host-precomputed set of source IDs, the recommended default. A host does whatever lookup it needs (tenant, application, or both) and hands the resulting set to the query; OKF4net.Catalog never needs to know how it was computed. Always wins over any host-level default policy for that one call.
• SourceVisibilityPolicy — a function evaluated per source, for rules a flat ID list can't express conveniently. Configurable once per host (DefaultSourceVisibilityPolicy) and overridable per query, mirroring DefaultResolverStrategy.

Neither has any effect on a query that sets neither field and a host that configures no default: every enabled source stays visible to every caller, exactly as before this feature existed.

KnowledgeAccessScope has no value-equality override (reference equality only) -- a SourceVisibilityPolicy function should compare TenantId/ UserId/SessionId individually, not compare two KnowledgeAccessScope instances with ==/Equals.

V1 limits

• Local filesystem bundles only — no remote/HTTP sources, no external connectors.
• One shared catalog per FileKnowledgeCatalog instance.
• No semantic/fuzzy deduplication — two concepts with similar content in genuinely different bundles are both returned. Only under the two merged strategies are two manifest entries resolving to the same directory collapsed; the default GroupedBySource strategy does not dedup.
• No tenant-aware authorization of any kind.

See the project README for the full documentation, and NOTICE/LICENSE.Apache-2.0 for the attribution chain of the underlying OKF implementation.

Licensed LGPL-3.0-or-later.