Meta tags:
description= GitHub is where people build software. More than 225 million people use GitHub to discover, fork, and contribute to over 800 million projects.;
Headings (most frequently used words):
tokenizer, navigation, footer, chevrotain, natasha, lindera, open, korean, text, menu, here, are, 365, public, repositories, matching, this, topic, related, topics, rasbt, llms, from, scratch, theseer, marcelroed, gigatoken, andialbrecht, sqlparse, dqbd, tiktokenizer, roshan, research, hazm, uhop, stream, json, lovit, soynlp, ikawaha, kagome, no, context, moo, wangfenjin, simple, niieani, gpt, blkserene, wordless, risesoft, y9, data, labeling, mathewsanders, mustard, cbaziotis, ekphrasis,
Text of the page (most frequently used words):
#tokenizer (29), code (25), updated (22), star (21), pull (20), requests (20), issues (20), for (17), and (17), 2026 (15), github (15), parser (14), text (13), language (12), python (12), gpt (12), processing (11), nlp (11), korean (10), library (8), nov (8), discussions (8), corpus (8), the (8), with (7), javascript (7), analysis (6), word (6), json (6), open (5), segmentation (5), from (5), support (5), typescript (5), sponsor (5), lexer (5), all (5), you (4), security (4), topics (4), source (4), 2025 (4), tokenization (4), data (4), morphological (4), openai (4), features (4), fts5 (4), sql (4), your (4), copilot (4), wikipedia (3), natural (3), normalization (3), ekphrasis (3), swift (3), java (3), chinese (3), rust (3), multilingual (3), oct (3), learning (3), sqlite3 (3), japanese (3), memory (3), filter (3), stream (3), syntax (3), natasha (3), persian (3), llm (3), php (3), scratch (3), most (3), stars (3), grammar (3), tokens (3), another (3), tab (3), window (3), refresh (3), session (3), reload (3), sign (3), developers (3), changelog (3), customer (3), enterprise (3), grade (3), build (3), this (2), community (2), status (2), footer (2), navigation (2), parsing (2), followers (2), processor (2), jun (2), spell (2), correction (2), tool (2), twitter (2), performs (2), splitting (2), statistics (2), english (2), may (2), mustard (2), tools (2), analyzer (2), lindera (2), search (2), linguistics (2), dependency (2), literature (2), translation (2), bpe (2), decoder (2), encoder (2), machine (2), models (2), tiktoken (2), pinyin (2), sqlite (2), feb (2), moo (2), context (2), pos (2), tagging (2), web (2), streams (2), jsonc (2), streaming (2), large (2), sax (2), building (2), api (2), russian (2), embeddings (2), tasks (2), projects (2), apr (2), toolkit (2), chatgpt (2), chevrotain (2), formatter (2), model (2), xml (2), into (2), jupyter (2), notebook (2), pytorch (2), like (2), step (2), recently (2), fewest (2), forks (2), sort (2), 188 (2), 365 (2), sponsors (2), events (2), trending (2), explore (2), signed (2), universe (2), training (2), integrations (2), stories (2), center (2), documentation (2), learn (2), business (2), fixes (2), products (2), secret (2), quality (2), can, perform, that, action, time, not, share, personal, information, manage, cookies, contact, docs, privacy, terms, inc, yacc, symbol, table, semantic, lookahead, generated, antlr, related, com, website, load, more, scala, mar, 2024, 668, corrector, spelling, semeval, geared, towards, social, networks, such, facebook, hashtags, using, big, corpora, 330mil, tweets, 672, cbaziotis, 2018, substrings, tokenizing, strings, when, whitespace, doesn, cut, 683, mathewsanders, annotation, nacos, vue3, springboot2, annotations, elasticsearch, docker, 数据标注是一款专门对文本数据进行处理和标注的工具, 通过简化快捷的文本标注流程和动态的算法反馈, 支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间, 数据标注的过程先由人工标注构建基础, 再由自动标注反哺人工标注, 最后由人工标注进行纠偏, 从而大幅度提高标注的精准度和高效性, 数据标注需要依赖开源的数字底座进行人员岗位管控, 702, labeling, risesoft, 708, stopword, lemmatizer, tagger, integrated, study, 765, wordless, blkserene, fastest, etc, port, additional, 862, niieani, fts, cpp14, 支持中文和拼音的, 全文搜索扩展, which, supports, 875, simple, wangfenjin, regexp, optimised, generator, uses, performance, 886, hacktoberfest, self, contained, written, pure, 988, kagome, ikawaha, extraction, postagging, 한국어, 자연어처리를, 파이썬, 라이브러리입니다, 토크나이저, 품사판별, 전처리의, 기능을, 제공합니다, 996, soynlp, lovit, files, efficient, pipeline, nodejs, micro, components, custom, pipelines, minimal, footprint, parse, transform, far, larger, than, available, inspired, token, node, uhop, sentence, ner, morphology, visualization, solves, basic, lower, level, lemmatization, farsi, hazm, roshan, research, stack, nextjs, online, playground, openapi, tokenizers, tiktokenizer, dqbd, grammars, non, validating, module, sqlparse, andialbrecht, gigatoken, marcelroed, small, converting, tokenized, potentially, other, formats, theseer, instruction, tuning, generative, pretraining, finetuning, attention, mechanism, artificial, intelligence, transformers, deep, implement, 109k, llms, rasbt, least, options, 104, 110, 134, 174, 287, 711, here, are, public, repositories, matching, topic, describes, programming, might, defined, backus, naur, form, bnf, lexical, turning, takes, builds, structure, abstract, tree, ast, concerned, does, sequence, fit, compiler, combined, built, specific, collections, message, dismiss, alert, switched, accounts, out, appearance, settings, pricing, event, talks, workshops, what, next, partners, about, company, forum, webinars, blog, resource, connect, resources, premium, advanced, add, ons, trust, insights, why, evaluate, see, industries, government, manufacturing, financial, services, healthcare, industry, nonprofits, startups, team, enterprises, stage, businesses, latest, improvements, across, one, place, developer, articles, repos, customize, certifications, scanning, leak, prevention, protection, vulnerability, detection, remediation, secure, cloud, development, environments, codespaces, enforcement, automation, actions, ship, coding, agent, automated, work, repository, hosting, review, collaboration, platform, menu, skip, content,
Text of the page (random words):
tokenizer github topics github skip to content navigation menu sign in products platform github repository hosting code review and collaboration copilot ai coding agent for tasks projects and automated work build ship actions ci cd and automation code quality quality analysis fixes and enforcement codespaces cloud development environments secure code security vulnerability detection and remediation secret protection secret scanning and leak prevention all features developers learn documentation training certifications build build with copilot customize copilot integrations explore trending repos github sponsors developer articles changelog changelog your source for the latest features improvements and fixes across github products all in one place businesses by stage enterprises team startups nonprofits by industry healthcare financial services manufacturing government see all industries evaluate github why github customer stories business insights trust center add ons github advanced security enterprise grade code security copilot for business enterprise grade ai features premium support enterprise grade 24 7 support resources learn connect documentation resource center blog events webinars community forum customer stories integrations support customer support training github status company about partners changelog github universe github universe the event for developers by developers talks workshops and what s next for github pricing search sign in sign up appearance settings you signed in with another tab or window reload to refresh your session you signed out in another tab or window reload to refresh your session you switched accounts on another tab or window reload to refresh your session dismiss alert message explore topics trending collections events github sponsors tokenizer star a grammar describes the syntax of a programming language and might be defined in backus naur form bnf a lexer performs lexical analysis turning text into tokens a parser takes tokens and builds a data structure like an abstract syntax tree ast the parser is concerned with context does the sequence of tokens fit the grammar a compiler is a combined lexer and parser built for a specific grammar here are 2 365 public repositories matching this topic language all filter by language all 2 365 python 711 jupyter notebook 287 rust 188 typescript 188 javascript 174 c 134 java 110 go 104 c 98 c 58 sort most stars sort options most stars fewest stars most forks fewest forks recently updated least recently updated rasbt llms from scratch star 109k code issues pull requests discussions implement a chatgpt like llm in pytorch from scratch step by step python machine learning natural language processing ai deep learning tokenizer transformers pytorch artificial intelligence gpt language model attention mechanism from scratch finetuning pretraining large language models llm generative ai instruction tuning updated nov 9 2026 jupyter notebook theseer tokenizer sponsor star 5 2k code issues pull requests a small library for converting tokenized php source code into xml and potentially other formats php xml tokenizer updated feb 3 2026 php marcelroed gigatoken sponsor star 4 2k code issues pull requests language model tokenization at gb s nlp tokenizer tokenization llm updated nov 2 2026 rust andialbrecht sqlparse star 4k code issues pull requests discussions a non validating sql parser module for python python formatter sql tokenizer sql parser sql formatter updated nov 1 2026 python chevrotain chevrotain star 2 8k code issues pull requests discussions parser building toolkit for javascript javascript open source grammars typescript parsing parser library tokenizer lexer updated nov 14 2026 typescript dqbd tiktokenizer star 1 7k code issues pull requests online playground for openapi tokenizers nextjs tokenizer openai t3 stack chatgpt tiktoken updated apr 24 2025 typescript roshan research hazm star 1 4k code issues pull requests persian nlp toolkit python nlp natural language processing tokenizer embeddings persian text processing dependency parser farsi pos tagging persian nlp normalization lemmatization updated apr 1 2026 python natasha natasha star 1 4k code issues pull requests solves basic russian nlp tasks api for lower level natasha projects visualization python nlp syntax morphology tokenizer embeddings russian ner sentence segmentation updated oct 14 2026 python uhop stream json sponsor star 1 2k code issues pull requests discussions a micro library of stream components for building custom json and jsonc processing pipelines with a minimal memory footprint parse filter and transform json far larger than available memory with a sax inspired token api on node js or web streams nodejs parser json streaming stream pipeline filter tokenizer sax json parser memory efficient large files streaming json jsonc web streams updated nov 9 2026 javascript lovit soynlp star 996 code issues pull requests 한국어 자연어처리를 위한 파이썬 라이브러리입니다 단어 추출 토크나이저 품사판별 전처리의 기능을 제공합니다 nlp tokenizer postagging korean text processing korean nlp word extraction updated oct 4 2026 python ikawaha kagome sponsor star 988 code issues pull requests self contained japanese morphological analyzer written in pure go japanese tokenizer segmentation korean japanese language nlp library hacktoberfest pos tagging morphological analysis updated nov 11 2026 go no context moo star 886 code issues pull requests optimised tokenizer lexer generator uses y for performance moo javascript regexp tokenizer lexer updated feb 27 2026 javascript wangfenjin simple star 875 code issues pull requests discussions 支持中文和拼音的 sqlite fts5 全文搜索扩展 a sqlite3 fts5 tokenizer which supports chinese and pinyin sqlite tokenizer cpp14 pinyin chinese sqlite3 fts fts5 sqlite3 fts5 updated may 17 2026 c niieani gpt tokenizer sponsor star 862 code issues pull requests discussions the fastest javascript bpe tokenizer encoder decoder for openai s gpt models gpt 5 gpt o gpt 4o etc port of openai s tiktoken with additional features machine learning encoder decoder tokenizer openai bpe gpt 2 gpt 3 gpt 4 gpt 5 gpt 4o gpt o1 updated nov 10 2026 typescript blkserene wordless star 765 code issues pull requests discussions an integrated corpus tool with multilingual support for the study of language literature and translation translation tokenizer corpus linguistics tagger literature dependency parser corpus linguistics lemmatizer corpus tools corpus processing corpus search corpus statistics stopword corpus analysis updated oct 30 2025 python lindera lindera star 708 code issues pull requests discussions a multilingual morphological analysis library multilingual library tokenizer analyzer morphological updated nov 17 2026 rust risesoft y9 data labeling star 702 code issues pull requests 数据标注是一款专门对文本数据进行处理和标注的工具 通过简化快捷的文本标注流程和动态的算法反馈 支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间 数据标注的过程先由人工标注构建基础 再由自动标注反哺人工标注 最后由人工标注进行纠偏 从而大幅度提高标注的精准度和高效性 数据标注需要依赖开源的数字底座进行人员岗位管控 java docker elasticsearch tokenizer chinese data annotations springboot2 vue3 nacos tokenizer parser data annotation tools updated jun 23 2025 java mathewsanders mustard star 683 code issues pull requests mustard is a swift library for tokenizing strings when splitting by whitespace doesn t cut it swift tokenizer substrings updated may 14 2018 swift cbaziotis ekphrasis star 672 code issues pull requests ekphrasis is a text processing tool geared towards text from social networks such as twitter or facebook ekphrasis performs tokenization word normalization word segmentation for splitting hashtags and spell correction using word statistics from 2 big corpora english wikipedia twitter 330mil english tweets nlp tokenizer text processing semeval nlp library word segmentation spelling correction tokenization text segmentation spell corrector word normalization updated jun 2 2025 python open korean text open korean text star 668 code issues pull requests open korean text processor an open source korean text processor natural language processing tokenizer korean text processing korean text processing korean tokenizer updated mar 12 2024 scala load more followers 10 9k followers website github com topics parsing wikipedia wikipedia related topics antlr generated parser language ll lookahead lr parser semantic analysis symbol table yacc footer 2026 github inc footer navigation terms privacy security status community docs contact manage cookies do not share my personal information you can t perform that action at this time
|