Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
36 packages returned for Tags:"tokenizer"
Sort by
Relevance
Downloads
Recently updated
Tokenizer
by:
flipbit
228,750 total downloads
last updated
1/4/2021
Latest version:
2.2.2
tokenizer
tokens
pattern
matching
Tokenizer extracts structured information from blocks of text and reflects them onto .NET objects
Jokenizer.
Net
by:
umutozel
60,969 total downloads
last updated
2/10/2020
Latest version:
1.1.1
csharp
expression
parser
tokenizer
C# Expression parser and evaluator, inspired from jokenizer project.
OpenNLP.
NET
by:
sergey_tihon
50,045 total downloads
last updated
8/23/2022
Latest version:
1.9.4.1
NLP
OpenNLP
Tokenizer
Categorizer
Chunker
Parser
POS
NER
IKVM
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech...
More information
SoftCircuits.
Parsing.
Helper
by:
SoftCircuits
17,021 total downloads
last updated
10/2/2022
Latest version:
4.2.0
text
parse
parser
parsing
text-parser
text-parsing
parsing-helper
.net
csharp
tokenizer
A .NET class library that makes it easier to parse text. The library tracks the current position within the text, ensures your code never attempts to access a character at an invalid index, and includes many...
More information
Conekta.
Xamarin
by:
bscheiman
8,825 total downloads
last updated
3/1/2015
Latest version:
1.0.6
boris
scheiman
bscheiman
conekta
xamarin
tokenizer
Tokenizador (Xamarin) para Conekta. Necesitas tener alguna libreria de servidor para usar el token.
NltkNet
by:
skeleton
9,761 total downloads
last updated
1/1/2021
Latest version:
1.0.14
nltk
nlp
parser
tokenizer
categorizer
POS
NER
chunker
IronPython
NLTK python library wrapper for .NET
OpenNLP.
NET.
Signed
by:
tastybeefalo
8,200 total downloads
last updated
2/26/2020
Latest version:
1.9.1
NLP
OpenNLP
Tokenizer
Categorizer
Chunker
Parser
POS
NER
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech...
More information
VBF.
Compilers.
Scanners
by:
Ninputer
5,389 total downloads
last updated
10/24/2015
Latest version:
1.0.5
VBF
Compiler
scanner
lexer
lex
tokenizer
VBF.Compilers.Scanners is a scanner builder. It contains a regular expression to DFA engine, can generate high performance scanners for unicode source text.
BERTTokenizers
by:
NMZivkovic
2,380 total downloads
last updated
9/9/2022
Latest version:
1.2.0
BERT
Tokenizer
charp
dotnet
This package contains tokenizers for following models: · BERT Base · BERT Large · BERT German · BERT Multilingual · BERT Base Uncased · BERT Large Uncased
Loretta.
CodeAnalysis.
Common
by:
gggkiller
3,894 total downloads
last updated
11/15/2022
Latest version:
0.2.11-nightly.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
A shared package used by Loretta. Do not install this package manually, it will be added as a prerequisite by other packages that require it.
AI.
Dev.
OpenAI.
GPT
by:
devis
3,087 total downloads
last updated
10/20/2022
Latest version:
1.0.2
OpenAI
GPT
GPT3
Tokenizer
Tokens
AI
OpenAI GPT utils, e.g. GPT3 Tokenizer
Trl.
PegParser
by:
wcoetser
3,835 total downloads
last updated
11/26/2020
Latest version:
1.3.0
Tokenizer
PEG
Grammer
Regular_Expressions
Regex
Parser
Parser_Generator
Trl.PegParser contains a tokenizer and a parser. The tokenizer uses regular expressions to define tokens, and exposes both matched and unmatched character ranges. The PEG Parser uses parsing expression grammers...
More information
Stringes
by:
Berkin
13,863 total downloads
last updated
11/8/2015
Latest version:
1.4.8
stringes
tokenizer
token
lexer
string
substring
parser
ast
syntax
grammar
More tags
The Stringe is a wrapper for the .NET String object that tracks line, column, offset, and other metadata for substrings.
owl
by:
SplittyDev
5,370 total downloads
last updated
10/21/2014
Latest version:
1.0.5407.37498
html
preprocessor
compiler
lexer
parser
scanner
tokenizer
owl
A html preprocessor which allows you to write html code using a beautiful syntax
Loretta.
CodeAnalysis.
Lua
by:
gggkiller
3,484 total downloads
last updated
11/15/2022
Latest version:
0.2.11-nightly.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
A GLua/Lua lexer, parser, code analysis, transformation and generation library.
Tortoise
by:
m-wantia
1,299 total downloads
last updated
6/17/2022
Latest version:
0.1.1
lexer
lexical
tokenizer
Package Description
Loretta.
CodeAnalysis.
Lua.
Experimental
by:
gggkiller
2,717 total downloads
last updated
11/15/2022
Latest version:
0.2.11-nightly.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
Experimental code that might become part of Loretta.CodeAnalysis.Lua.
CEM.
Utilities.
Tokenizer
by:
dominic.proulx
807 total downloads
last updated
7/29/2022
Latest version:
1.0.1
CEM
Utilities
Tokenizer
Classe qui gère le découpage d'une chaine en paires de clés/valeurs et permet de la reconstruire avec des valeurs de remplacement ou des valeurs additionnelles
TokenCs
by:
zeplar_exe
1,318 total downloads
last updated
8/10/2022
Latest version:
1.0.5
token
tokenizer
lexer
parse
parser
A simple library to abstract the gritty process of lexical parsing.
GParse
by:
gggkiller
1,863 total downloads
last updated
6/15/2021
Latest version:
5.0.0-alpha.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
Parsing and lexing utilities to create your own parser and lexer
Previous
Next