60 packages returned for Tags:"tokenizer"

Tokenizer extracts structured information from blocks of text and reflects them onto .NET objects

Fast and memory-efficient WordPiece tokenizer as it is used by BERT and others. Tokenizes text for further processing using NLP/language models.

C# Expression parser and evaluator, inspired from jokenizer project.

OpenAI GPT utils, e.g. GPT3 Tokenizer

Anthropic Claude BPE Tokenizer unofficial implementation

This package contains tokenizers for following models: · BERT Base · BERT Large · BERT German · BERT Multilingual · BERT Base Uncased · BERT Large Uncased

Provide tokenizers to allow counting content tokens for text and embeddings

The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech... More information

A shared package used by Loretta. Do not install this package manually, it will be added as a prerequisite by other packages that require it.

A GLua/Lua lexer, parser, code analysis, transformation and generation library.

A .NET class library that makes it easier to parse text. The library tracks the current position within the text, ensures your code never attempts to access a character at an invalid index, and includes many... More information

Tokenizador (Xamarin) para Conekta. Necesitas tener alguna libreria de servidor para usar el token.

.NET wrapper of HuggingFace Tokenizers library

NLTK python library wrapper for .NET

Experimental code that might become part of Loretta.CodeAnalysis.Lua.

Open AI Chat Completion Models (GPT 3.5/GPT 4) BPE Tokenizer unofficial implementation

VBF.Compilers.Scanners is a scanner builder. It contains a regular expression to DFA engine, can generate high performance scanners for unicode source text.

Package Description

Trl.PegParser contains a tokenizer and a parser. The tokenizer uses regular expressions to define tokens, and exposes both matched and unmatched character ranges. The PEG Parser uses parsing expression grammers... More information

The Stringe is a wrapper for the .NET String object that tracks line, column, offset, and other metadata for substrings.