Meta tags:
description= :book: A curated list of resources dedicated to Natural Language Processing (NLP) - keon/awesome-nlp;
Headings (most frequently used words):
and, nlp, in, models, libraries, embeddings, datasets, language, data, uh, oh, navigation, text, saved, searches, awesome, files, research, tutorials, multilingual, license, footer, reading, information, extraction, languages, corpora, search, code, repositories, users, issues, pull, requests, provide, feedback, keon, menu, use, to, filter, your, results, more, quickly, folders, latest, commit, history, repository, scope, contents, summaries, trends, prominent, labs, tasks, methods, frameworks, for, per, see, also, citation, about, releases, packages, contributors, historical, highlights, content, videos, online, courses, books, services, annotation, tools, tokenization, morphology, segmentation, pos, tagging, dependency, parsing, named, entity, recognition, coreference, resolution, classification, sentiment, analysis, topic, modeling, summarization, machine, translation, question, answering, comprehension, beyond, ner, retrieval, speech, pretraining, adaptation, cross, lingual, evaluation, benchmarks, reasoning, test, time, compute, long, context, alternative, architectures, factuality, hallucination, calibration, probing, interpretability, efficient, small, instruction, tuning, preference, optimization, bias, fairness, safety, arabic, chinese, anthology, danish, dutch, german, hungarian, indic, treebanks, word, tooling, indonesian, korean, blogs, persian, polish, portuguese, spanish, thai, ukrainian, urdu, uzbek, vietnamese, other, topics, resources, contributing, stars, watchers, forks, that, need, login, access, can, be, gained, via, email,
Text of the page (most frequently used words):
and (360), for (250), nlp (233), language (124), with (108), the (103), #models (100), open (78), languages (71), natural (65), 2024 (64), text (63), top (63), processing (61), 2025 (61), back (54), based (47), library (44), multilingual (43), model (42), data (40), python (37), from (36), embeddings (35), across (33), ner (33), tasks (33), datasets (32), word (32), learning (30), llms (29), reasoning (28), retrieval (27), toolkit (27), pos (26), annotation (26), korean (25), evaluation (25), benchmark (25), level (25), corpus (24), free (24), long (24), libraries (23), tagging (23), persian (23), bert (22), context (22), neural (22), llm (21), speech (21), machine (20), chinese (20), instruction (20), tokenization (20), lms (20), extraction (20), research (20), arabic (20), resources (19), deep (19), parsing (19), classification (19), analysis (19), code (19), generation (18), large (18), vietnamese (18), dependency (18), translation (18), multi (18), training (18), topic (18), github (17), awesome (17), tools (17), sentiment (17), hindi (17), pretraining (17), english (16), treebank (16), 2023 (16), spanish (16), encoder (16), source (16), 2026 (15), list (15), summarization (15), transformer (15), modeling (15), that (14), time (14), corpora (14), covering (14), sentence (14), strong (14), trained (14), embedding (14), coreference (14), document (14), tool (14), curated (13), rag (13), see (13), dataset (13), package (13), efficient (13), platform (13), small (13), tuning (13), transformers (13), attention (13), modern (13), information (12), also (12), linguistic (12), indonesian (12), tokens (12), scale (12), fast (12), recognition (12), entity (12), cross (12), fine (12), search (12), token (12), test (12), all (12), your (12), end (12), this (11), including (11), task (11), benchmarks (11), indic (11), quality (11), parser (11), named (11), inference (11), built (11), you (10), can (10), security (10), production (10), collection (10), thai (10), implementation (10), other (10), family (10), pretrained (10), segmentation (10), foundation (10), word2vec (10), course (10), high (10), spacy (10), low (10), state (10), features (10), view (10), foundational (10), such (10), systems (9), tokenizer (9), bilingual (9), sentences (9), universal (9), tuned (9), java (9), methods (9), morphological (9), pytorch (9), dutch (9), support (9), over (9), resource (9), framework (9), bias (9), more (9), art (9), self (9), decoder (9), sparse (9), hallucination (9), like (9), labeling (9), license (8), general (8), current (8), using (8), standard (8), first (8), format (8), llama (8), university (8), via (8), common (8), human (8), shot (8), mteb (8), google (8), web (8), are (8), performance (8), architecture (8), 200 (8), lingual (8), api (8), error (7), please (7), reload (7), asian (7), words (7), semantic (7), sequence (7), urdu (7), portuguese (7), used (7), analyzer (7), scala (7), nltk (7), preference (7), feedback (7), huggingface (7), aware (7), dense (7), unified (7), survey (7), factuality (7), calibration (7), detection (7), question (7), answering (7), compute (7), stanford (7), community (6), contributing (6), topics (6), prompt (6), generative (6), here (6), 100 (6), news (6), southeast (6), relation (6), available (6), part (6), fully (6), dependencies (6), suite (6), samples (6), pipeline (6), danish (6), moe (6), mmlu (6), domain (6), alignment (6), safety (6), matches (6), widely (6), written (6), without (6), interpretable (6), representations (6), probing (6), gpt (6), linear (6), sota (6), system (6), following (6), nlu (6), seq2seq (6), baseline (6), reading (6), how (6), tokenizers (6), building (6), teams (6), various (6), clojure (6), node (6), enterprise (6), navigation (5), there (5), was (5), while (5), repository (5), keon (5), serving (5), prompting (5), augmented (5), papers (5), classical (5), japanese (5), parallel (5), core (5), 400 (5), fasttext (5), covers (5), roberta (5), only (5), wikipedia (5), wiki (5), advanced (5), blog (5), tutorials (5), browser (5), alternative (5), scratch (5), elmo (5), iit (5), german (5), lemmatization (5), deepseek (5), book (5), per (5), many (5), resolution (5), sft (5), reference (5), optimization (5), parameter (5), structured (5), few (5), settings (5), memory (5), larger (5), distillation (5), autoencoders (5), interpretability (5), scaling (5), dynamic (5), pipelines (5), hybrid (5), meta (5), beyond (5), simple (5), loaders (5), statistical (5), traditional (5), ready (5), bpe (5), subword (5), folia (5), spark (5), projects (5), use (5), online (5), actions (5), content (5), include (5), network (5), creating (5), kotlin (5), group (5), notebooks (5), issues (5), summaries (5), illustrated (5), loading (4), page (4), readme (4), mining (4), dedicated (4), find (4), tooling (4), purpose (4), lists (4), out (4), emnlp (4), tts (4), ukrainian (4), best (4), polish (4), annotated (4), most (4), tags (4), nmt (4), technology (4), style (4), provides (4), polarity (4), facebook (4), review (4), stopwords (4), hungarian (4), focus (4), qwen (4), correction (4), morphology (4), anthropic (4), fairness (4), results (4), zero (4), weight (4), adaptation (4), layer (4), cache (4), quantization (4), bge (4), paragraph (4), accuracy (4), non (4), gemma (4), reproducible (4), than (4), clustering (4), architectures (4), feature (4), transcoders (4), outputs (4), hop (4), knowledge (4), factual (4), form (4), set (4), grained (4), extension (4), matching (4), process (4), step (4), surpasses (4), into (4), questions (4), refresh (4), translate (4), behind (4), work (4), size (4), new (4), functionality (4), files (4), grade (4), asr (4), includes (4), scoring (4), series (4), then (4), pre (4), comprehension (4), networks (4), vectors (4), canonical (4), approaches (4), byte (4), rust (4), wide (4), supports (4), costs (4), powered (4), interface (4), services (4), julia (4), ruby (4), regular (4), metrics (4), visual (4), notable (4), acl (4), pull (4), action (3), terms (3), report (3), stars (3), cc0 (3), about (3), 2018 (3), useful (3), citation (3), techniques (3), template (3), engineering (3), hebrew (3), texts (3), elasticsearch (3), russian (3), good (3), tagger (3), books (3), 250 (3), big (3), constituency (3), sailor (3), mistral (3), chat (3), mit (3), uzbek (3), focused (3), etc (3), multiple (3), center (3), both (3), annotations (3), tagged (3), stemmer (3), squad (3), archive (3), movie (3), institute (3), bahasa (3), collections (3), indobert (3), telugu (3), patna (3), access (3), labelled (3), sense (3), twitter (3), domains (3), aspect (3), bangla (3), classes (3), treebanks (3), developed (3), industrial (3), binding (3), frog (3), alibaba (3), trainable (3), javascript (3), faking (3), policy (3), value (3), reveals (3), jailbreak (3), integrating (3), turn (3), strategies (3), pairs (3), grpo (3), way (3), ai2 (3), post (3), recipe (3), tülu (3), adapting (3), class (3), loss (3), qwen3 (3), 128k (3), competitive (3), much (3), encoders (3), specific (3), sae (3), shows (3), introduces (3), graphs (3), tracing (3), methodology (3), recall (3), learns (3), claim (3), uncertainty (3), before (3), through (3), faithfulness (3), under (3), comprehensive (3), rope (3), extends (3), recurrent (3), tradeoffs (3), patterns (3), minimal (3), mamba (3), reward (3), produce (3), chain (3), thought (3), adaptive (3), math (3), cot (3), short (3), supervised (3), needle (3), haystack (3), but (3), together (3), pro (3), gap (3), between (3), gemini (3), seamlessm4t (3), 101 (3), transfer (3), default (3), optimized (3), deberta (3), still (3), application (3), endangered (3), disambiguation (3), frameworks (3), linguistically (3), consistent (3), fineweb (3), gensim (3), fields (3), late (3), interaction (3), colbert (3), improves (3), supporting (3), dpr (3), analyzing (3), read (3), real (3), practical (3), bleu (3), chrf (3), learned (3), moses (3), contextual (3), latent (3), box (3), classifier (3), higher (3), cost (3), types (3), flair (3), stanza (3), vocabulary (3), input (3), unicode (3), implementations (3), unigram (3), agnostic (3), well (3), glove (3), where (3), environment (3), xml (3), shoonya (3), cloud (3), intent (3), syntax (3), working (3), variety (3), engine (3), archived (3), snips (3), software (3), opennlp (3), exploring (3), developing (3), applications (3), expression (3), forms (3), jupyter (3), lectures (3), recent (3), labs (3), penn (3), trends (3), repositories (3), commit (3), name (3), insights (3), requests (3), signed (3), another (3), tab (3), window (3), session (3), sign (3), saved (3), documentation (3), copilot (3), solutions (3), explore (3), perform (2), not (2), manage (2), status (2), privacy (2), footer (2), releases (2), forks (2), adjacent (2), scope (2), ancient (2), 10k (2), device (2), vieneu (2), grounded (2), speeches (2), prime (2), minister (2), tensorflow (2), character (2), coverage (2), national (2), reliable (2), uppsala (2), translated (2), 250k (2), suitable (2), french (2), petition (2), chatbot (2), naver (2), blogs (2), klue (2), native (2), mecab (2), multimodal (2), hub (2), providing (2), standardized (2), wordnet (2), sea (2), lion (2), cendol (2), indolem (2), indonlu (2), algorithm (2), sarvam (2), sanskrit (2), bengali (2), whole (2), ulmfit (2), marathi (2), need (2), email (2), labels (2), absa (2), reviews (2), smaller (2), strength (2), kaldi (2), anthology (2), jieba (2), released (2), alongside (2), arabert (2), organized (2), cases (2), when (2), vector (2), mitigation (2), refusal (2), attacks (2), failures (2), user (2), measurement (2), gender (2), masked (2), instruct (2), rlhf (2), less (2), adopted (2), direct (2), instructions (2), follow (2), flan (2), lora (2), peft (2), rank (2), quantized (2), throughput (2), icml (2), precision (2), relevant (2), fastfit (2), prompts (2), setfit (2), apple (2), intelligence (2), thinking (2), technical (2), local (2), global (2), yarn (2), microsoft (2), ones (2), phi (2), distilbert (2), identifies (2), circuits (2), highly (2), inputs (2), skip (2), applies (2), attribution (2), case (2), enables (2), circuit (2), causal (2), computational (2), examples (2), behavior (2), browsing (2), extracting (2), monosemanticity (2), structural (2), what (2), taxonomy (2), choice (2), pattern (2), sampling (2), ssm (2), combining (2), degradation (2), contexts (2), term (2), memorize (2), historical (2), lost (2), middle (2), windows (2), position (2), rotary (2), length (2), aime (2), baselines (2), verification (2), discriminative (2), clipping (2), four (2), improvements (2), mcts (2), systematic (2), study (2), openai (2), pure (2), tree (2), consistency (2), improve (2), direction (2), explicit (2), nested (2), score (2), longbench (2), synthetic (2), exposing (2), diverse (2), evidence (2), critique (2), beir (2), flores (2), xnli (2), xtreme (2), african (2), speaker (2), madlad (2), nllb (2), 500 (2), cohere (2), massively (2), aya (2), weights (2), base (2), substrate (2), denoising (2), bart (2), modernbert (2), nli (2), since (2), around (2), agents (2), kits (2), uralic (2), some (2), uralicnlp (2), offers (2), server (2), conll (2), udpipe (2), generate (2), edu (2), redpajama (2), index (2), pointer (2), mmteb (2), query (2), trace (2), retrieve (2), one (2), gritlm (2), nomic (2), grammatical (2), srl (2), rebel (2), triviaqa (2), era (2), atlas (2), extractive (2), metric (2), ter (2), sacrebleu (2), facto (2), comet (2), fairseq (2), understanding (2), abstractive (2), generator (2), hierarchical (2), regularized (2), bigartm (2), lda (2), dirichlet (2), allocation (2), white (2), pyss3 (2), sst (2), category (2), coref (2), span (2), handles (2), crf (2), parsers (2), reducing (2), complex (2), theory (2), tokenized (2), sentencepiece (2), unsupervised (2), lemma (2), leaderboards (2), contextualized (2), grams (2), explainer (2), agent (2), control (2), label (2), argilla (2), flat (2), users (2), lab (2), management (2), easy (2), hosted (2), active (2), freemium (2), collaborative (2), discourse (2), offering (2), doccano (2), brat (2), rapid (2), custom (2), cognitive (2), service (2), amazon (2), related (2), vscode (2), clojurescript (2), visualizations (2), mallet (2), apache (2), epic (2), alike (2), ucto (2), bllip (2), crfsuite (2), conditional (2), random (2), crfs (2), sequential (2), evaluate (2), dsl (2), allows (2), which (2), regex (2), industry (2), build (2), very (2), extensible (2), modelling (2), jptdp (2), interactive (2), better (2), plain (2), adversarial (2), world (2), beginner (2), friendly (2), hands (2), hugging (2), face (2), authors (2), program (2), rnns (2), taking (2), introduction (2), carnegie (2), mellon (2), oxford (2), videos (2), courses (2), paper (2), guides (2), explanations (2), guide (2), sebastian (2), ruder (2), newsletter (2), playbook (2), engineer (2), their (2), contributions (2), development (2), create (2), project (2), prominent (2), rolling (2), venues (2), they (2), add (2), pull_request_template (2), assets (2), 622 (2), commits (2), last (2), message (2), menu (2), appearance (2), cancel (2), searches (2), business (2), developer (2), customer (2), devops (2), app (2), share, personal, cookies, contact, docs, inc, contributors, packages, published, watching, 568, watchers, activity, note, https, com, url, year, kim, woo, author, title, misc, consider, citing, mlops, modalities, nlph_resources, doing, cltk, lao, icu, pymorphy2, 20m, law, articles, film, subtitles, evb, sql, 2020, findings, vitext2sql, 75m, vntqcorpus, txt, hours, recorded, hcmus, ailab, vivos, ud_vietnamese, bktreebank, vistral, vinai, phogpt, bartpho, phobert, voice, cloning, pyvi, vncorenlp, vitk, underthesea, culturally, rows, soas, urduhack, ukrainianlt, thailand, inter, openthaigpt, scb, 10x, typhoon, wangchanberta, synthai, cutkum, cluster, jtcc, pythainlp, sent2vec, rigochat, bsc, barcelona, supercomputing, salamandra, basque, latxa, bne, beto, compilation, unannotated, billion, copenhagen, columbian, political, detect, censor, clean, profanity, hate, bullying, speaking, countries, spanlp, portulan, albertina, maritaca, sabiá, brazilian, bertimbau, clef, 2008, 2009, hamshahri, syntactically, updt, 982, verbs, valency, lexicon, syntactic, semeval, 2010, perlex, 25m, farsiyar, persianner, 682, iob, armanpersonercorpus, lscp, 120m, 27m, casual, tweets, colloquial, freely, upc, farsi, manually, bijankhan, dorna, persianmind, parsbert, cleaning, virastar, parsianalyzer, partial, perstem, keyphrase, perke, parsivar, hazm, html, korquad, expired, blue, house, site, petitions, major, south, newspaper, chosun, ilbo, korea, science, kaist, kangwon, dsindex, hyperclova, exaone, polyglot, skt, kobert, running, client, side, webassembly, 1mb, offline, garu, kiwi, splitter, kss, konlp, koalanlp, konlpy, seacrowd, dictionary, indosum, idn, 39k, 900k, panl10n, kompas, tempo, ilps, singapore, regional, nusacrowd, indonesia, sastrawi, stemming, pysastrawi, bharatgpt, krutrim, airavata, continuation, openhathi, indictrans2, 2022, indicbert, ai4bharat, indicnlp, fastai, inltk, kannada, sivareddy, python3, port, transliteration, helpers, oscar, albert, bunch, crawl, languge, hindi2vec, tdil, aggregates, lot, otherwise, gated, sentiwordnet, bombay, cfilt, tamil, sail, 2015, login, gained, bbc, 60k, peter, graham, isi, fire, above, mentioned, representational, layered, off, shelf, does, alpino, surface, realiser, simplenlg, simplenlg_nl, danlp, funnlp, baichuan, tsinghua, chatglm3, glm, improved, mlm, macbert, masking, wwm, hit, ltp, hanlp, fudannlp, snownlp, arabicmmlu, aggregated, labr, largest, multidomain, sdaia, allam, jais, araelectra, msa, dialectal, camelbert, qcri, farasa, segmenter, coptic, rftokenizer, pyarabic, jsastem, goarabic, dialect, camel, click, section, expand, occurs, conflicts, internalized, values, steering, reduces, vlaf, conflict, agreement, scripts, indicsafe, modular, defenses, risk, categories, teleai, 4000, dialogues, scenarios, safedialbench, finetuning, narrow, insecure, unexpectedly, produces, broad, unrelated, emergent, misalignment, moderation, wildguard, strategically, complying, during, tailoring, answers, beliefs, sycophancy, toxicity, realtoxicityprompts, demographic, axes, holisticbias, winobias, social, crows, measuring, stereotypical, stereoset, synthesizes, response, aligned, nothing, filtered, subset, official, magpie, dpo, trl, goes, lima, among, simpler, generated, against, constitution, constitutional, 1600, super, naturalinstructions, bootstrapping, instructgpt, finetuned, learners, bundling, prefix, ia3, others, decomposed, dora, adapters, modest, hardware, qlora, tgi, sglang, pagedattention, vllm, portable, gguf, cpp, sensitivity, wise, mixed, improvement, uniform, kv8, kvtuner, activation, awq, gptq, deploying, compact, near, stella, gte, siamese, sharing, bit, qat, reduction, 235b, modes, 30b, a3b, activating, parameters, 27b, ratio, keep, tractable, nope, smollm3, smollm2, distilled, minilm, rather, mechanism, finding, explanation, reconstructing, yield, saes, beat, claude, haiku, rhyme, planning, studies, biology, construct, replacement, interactions, revealing, identifying, driving, influence, functions, neuronpedia, towards, toy, superposition, pyramid, probes, locating, editing, associations, rome, limitations, alternatives, classifiers, belinkov, primer, bertology, trains, reason, generating, gains, biography, factbench, auroc, cure, think, hard, required, responses, rates, persist, even, halluhard, logits, principled, quantification, fact, checking, formally, separates, franq, substantially, worse, calibrated, extended, single, claims, atomic, extrinsic, intrinsic, regeneration, resist, leakage, hallulens, effects, lookback, lens, ragas, selfcheckgpt, evaluator, longfact, safe, factscore, truthfulness, truthfulqa, speed, 220k, ssms, faster, hybrids, balance, efficiency, characterizing, undertraining, frequency, dimensions, evolutionary, rescaling, llama3, 80x, fewer, longrope2, coarse, compression, selection, speedups, 64k, nsa, 456b, lightning, softmax, minimax, module, scales, outperforms, titans, extending, interpolation, jamba, rnn, counts, rwkv, selective, space, 1000, controlled, experiments, recipes, closed, openthoughts, prms, supervision, thinkprm, gae, stable, vapo, key, decoupled, entropy, bonus, reproduces, dapo, paired, rollouts, bootstrap, distilling, rstar, prm, reaching, kimi, budget, forcing, optimally, replicated, let, verify, reasoners, reflexion, refine, trees, thoughts, majority, vote, sampled, chains, result, intermediate, steps, trend, defining, traces, benefit, extra, configurations, mitigates, degrades, niah, 503, expert, crafted, spanning, humans, pressure, ruler, probe, 824, requiring, frames, conversational, simultaneous, tested, frontier, below, multichallenge, typologically, prox, harness, contamination, resistant, monthly, livebench, elo, leaderboard, arena, lmsys, verifiable, ifeval, scientific, overclaim, missing, refute, graduate, proof, gpqa, harder, successor, multitask, subjects, capabilities, bench, holistic, helm, heterogeneous, massive, xglue, superglue, glue, xiaomi, scaled, commercial, milmmt, specialized, translategemma, mcgill, 14b, continued, 26b, empirical, mixing, afriquellm, princeton, mila, adapted, wura, irokobench, afriqa, lugha, 83b, population, speakers, comparably, sized, xcopa, mgsm, babel, targeting, seallm, seamless, left, glot500, expanse, 176b, bloom, mt5, commoncrawl, xlm, objective, generalization, mixtral, mid, reproducibility, olmo, especially, often, framing, 250m, depth, width, identical, neobert, modernized, flashattention, replaced, sample, electra, disentangled, robustly, bidirectional, workhorse, them, scoped, phenomena, mostly, sami, mordvin, mari, komi, supported, finnish, swedish, splitting, dynet, standalone, cli, bindings, rest, cube, tokenizing, lemmatizing, primarily, solution, tiny, own, copies, tiny_qa_benchmark_pp, mixture, 167, culturax, 15t, cleaned, filters, educational, documented, filtering, dolma, reproductions, 30t, signals, 825, gib, pile, central, versioned, streamable, hubs, coqui, wav2vec, 170, realtime, gpu, vad, punctuation, diarization, emotion, autoregressive, sensevoice, fun, nano, funasr, nvidia, canary, whisper, borders, expansion, marco, lotte, att, intensive, remixer, synthesis, redapter, record, ndcg, bright, reasonembed, reranking, ood, rank1, surpassing, prior, proprietary, derived, xor, original, variable, dimensionality, matryoshka, representation, embed, families, colbertv2, dual, passage, increasingly, novel, rate, schedule, bea, edit, gec, surpass, role, docred, privee, automatically, policies, templates, schema, openie, assistant, gaia, fid, drqa, distantly, hotpotqa, have, restructured, bridging, divide, sub, 10b, turbo, comparison, professional, translators, chatgpt, translator, similarity, bertscore, marian, reset, field, fluency, hurts, grounding, longer, budgets, harm, element, summarizers, benchmarking, scrolls, booksum, pegasus, graph, textrank, anchor, corex, jointly, top2vec, bertopic, lsi, hdp, blei, caveats, annotators, imdb, lead, closing, shared, dethrone, motivated, lingmess, maverick, order, hoi, spanbert, lee, eight, replace, guideline, gollie, generalist, arbitrary, gliner, string, bilstm, lample, 2003, attentive, kitaev, klein, light, trankit, biaffine, hoc, additions, coalesce, sequences, retraining, premiums, quantifies, fertility, predicts, penalties, morphologically, tax, iclr, formal, stochastic, map, establishes, conditions, foundations, decouples, output, vocabularies, log, relationship, independently, superword, downstream, superbpe, patching, revives, blt, operating, characters, canine, byt5, wordpiece, units, pair, encoding, sennrich, morfessor, two, dominant, schemes, infersent, cove, doc2vec, sense2vec, handle, oov, static, problem, each, subsection, linking, mace, checks, assisted, 300, example, potato, modal, studio, collecting, curating, rich, assertion, unlimited, documents, foss
Text of the page (random words):
cing inference cost without retraining pos tagging and dependency parsing back to top universal dependencies cross linguistically consistent treebanks 100 languages spacy and stanza production parsers across many languages deep biaffine attention for neural dependency parsing foundational neural parsing architecture trankit light weight transformer based multilingual nlp toolkit self attentive constituency parsing kitaev klein strong neural constituency parser named entity recognition and information extraction back to top foundational and neural conll 2003 ner canonical english ner benchmark neural architectures for ner lample et al bilstm crf the long time go to ner architecture flair contextual string embeddings strong ner across languages spacy ner production ready open and instruction following ie universal ner instruction tuned lm for open set ner across languages gliner 2023 small generalist ner model that handles arbitrary entity types at inference gollie guideline following information extraction with lms rebel end to end relation extraction as seq2seq llm based gpt ner llms for named entity recognition can llms replace sentence level ner 2024 cost quality tradeoffs generative ner in the era of llms 2026 eight open llms across four ner benchmarks peft with structured outputs matches encoder based ner coreference resolution back to top end to end neural coreference lee et al foundation for modern neural coreference spanbert span based pretraining strong coreference baseline coref hoi higher order inference coreference maverick coref 2024 efficient coreference matching the best larger systems lingmess linguistically motivated category based coreference scoring llm based llms for coreference resolution prompting and fine tuning for coreference multilingual coreference shared task can llms dethrone traditional approaches 2025 9 systems across 4 llm based and 5 traditional approaches traditional methods still lead but llms are closing the gap text classification and sentiment analysis back to top fasttext classifier strong fast linear baseline sentiment treebank sst canonical fine grained sentiment dataset setfit few shot text classification without prompts fastfit fast few shot for many class settings sst imdb ag news with deberta v3 current encoder fine tuning baseline pyss3 white box interpretable text classifier llms as annotators using llms for text classification labeling with caveats topic modeling back to top latent dirichlet allocation blei et al foundational topic model gensim lda lsi hdp in python bigartm fast regularized topic modeling bertopic clustering based topic modeling on top of contextual embeddings common modern default top2vec jointly learns topic and document vectors corex topic hierarchical topic modeling with anchor words summarization back to top textrank extractive graph based summarization pointer generator networks see et al foundational neural abstractive summarization pegasus gap sentences pretraining for summarization bart widely used denoising seq2seq baseline booksum and scrolls long document summarization benchmarks llm based benchmarking llms for news summarization llms vs fine tuned summarizers element aware summarization with llms structured prompting for summarization understanding llm reasoning for abstractive summarization 2025 explicit reasoning improves fluency but hurts factual grounding longer reasoning budgets can harm faithfulness machine translation back to top statistical and foundational neural moses reference statistical mt system attention is all you need transformer reset the field marian nmt efficient c nmt framework fairseq pytorch sequence modeling toolkit massively multilingual nllb 200 mt for 200 languages madlad 400 400 language mt seamlessm4t speech and text mt 100 languages evaluation comet learned mt metric current de facto standard alongside chrf sacrebleu reproducible bleu chrf ter scoring bertscore similarity based generation metric llm based is chatgpt a good translator llms as machine translation systems adapting llms for document level mt 2024 llms for context aware translation gpt 4 vs human translators quality comparison on professional mt multilingual mt with open llms at practical scale 2025 benchmarks sub 10b open llms on 28 language mt matches gpt 4 turbo and google translate bridging the linguistic divide survey on llms for mt 2025 survey of how instruction following in context learning and preference alignment have restructured mt methodology question answering and reading comprehension back to top datasets and foundational systems squad squad 2 0 extractive reading comprehension natural questions real user questions over wikipedia hotpotqa multi hop reasoning triviaqa distantly supervised qa drqa open domain qa over wikipedia document qa multi paragraph reading comprehension modern open domain qa dpr and fid retrieve then read the standard pre llm open domain qa pipeline atlas retrieval augmented lm for few shot qa see also retrieval for nlp llm era gpt 4 with retrieval on triviaqa nq self rag 2023 retrieval generation and self critique gaia general ai assistant benchmark including multi step qa information extraction beyond ner back to top openie 6 schema free open information extraction template based information extraction without the templates privee an architecture for automatically analyzing web privacy policies rebel end to end relation extraction docred document level relation extraction benchmark llms for semantic role labeling 2025 generative llms with rag and self correction surpass encoder decoder bert style models on srl in english and chinese adapting llms for minimal edit gec 2025 decoder only llms with a novel error rate adaptation schedule set new sota on bea test grammatical error correction retrieval and embeddings back to top dense and late interaction retrieval increasingly the substrate for qa and ir dpr dense passage retrieval dual encoder retrieval baseline colbert and colbertv2 late interaction retrieval strong on out of domain e5 and e5 mistral widely used dense embedding families bge and bge m3 2024 multilingual multi functionality embeddings top of mteb across languages nomic embed 2024 fully open reproducible embedding model matryoshka representation learning nested embeddings supporting variable dimensionality at inference gritlm 2024 unified generation and embedding from one model rag retrieval augmented generation the original retrieval augmented framework foundation for modern qa pipelines gemini embedding 2025 gemini derived dense embeddings sota on mmteb across 250 languages and on cross lingual retrieval xor retrieve xtreme up qwen3 embedding 2025 decoder based embedding series 0 6b 8b built on qwen3 1 on mteb multilingual and mteb code surpassing prior proprietary models rank1 2025 first reranking model trained with test time compute via deepseek r1 reasoning trace distillation sota on instruction following and ood retrieval reasonembed 2025 embedding model for reasoning intensive retrieval with remixer data synthesis and redapter adaptive training record ndcg 10 of 38 1 on bright colbert att 2026 extends late interaction retrieval by integrating query and document attention weights into colbert scoring improves recall on ms marco beir and lotte embedding and retrieval benchmarks mmteb 2025 community expansion of mteb to 500 tasks across 250 languages speech and text back to top a short pointer set since this borders adjacent fields whisper multilingual asr the modern open default seamlessm4t unified speech and text translation canary nvidia 2024 top open multilingual asr model funasr industrial grade asr toolkit 170 realtime on gpu 50 languages built in vad punctuation speaker diarization and emotion detection includes non autoregressive sensevoice and llm based fun asr nano models wav2vec 2 0 foundational self supervised speech pretraining coqui tts and vieneu tts open tts datasets back to top dataset hubs and lists huggingface datasets hub the central index for modern nlp datasets with versioned streamable loaders nlp datasets large collection of nlp datasets gensim data data repository for pretrained nlp models and nlp corpora pretraining scale corpora open the pile 825 gib diverse text corpus redpajama redpajama v2 2023 2024 reproductions of llama pretraining data v2 is 30t tokens with quality signals dolma ai2 2023 2024 3t token open pretraining corpus with documented filtering pipeline fineweb fineweb edu 2024 15t token cleaned web corpus fineweb edu filters for educational quality culturax 6 3t tokens across 167 languages common corpus 2024 2t token open license multilingual corpus task and instruction datasets universal dependencies cross linguistically consistent treebank annotation 100 languages tülu 3 sft mixture 2024 open instruction tuning data behind tülu 3 tiny_qa_benchmark_pp tiny nlp multi lingual qa datasets and library to generate your own synthetic copies multilingual nlp frameworks back to top udpipe is a trainable pipeline for tokenizing tagging lemmatizing and parsing universal treebanks and other conll u files primarily written in c offers a fast and reliable solution for multilingual nlp processing nlp cube natural language processing pipeline sentence splitting tokenization lemmatization part of speech tagging and dependency parsing new platform written in python with dynet 2 0 offers standalone cli python bindings and server functionality rest api uralicnlp is an nlp library mostly for many endangered uralic languages such as sami languages mordvin languages mari languages komi languages and so on also some non endangered languages are supported such as finnish together with non uralic languages such as swedish and arabic uralicnlp can do morphological analysis generation lemmatization and disambiguation language models for nlp back to top pretrained language models and the research around them scoped to nlp tasks and linguistic phenomena for general purpose llm tooling agents or rag application kits see see also pretraining and adaptation encoders still the workhorse for classical nlp tasks bert bidirectional transformer pretraining foundation for most encoder based nlp work since 2018 roberta robustly optimized bert pretraining common encoder baseline deberta deberta v3 disentangled attention strong on classification ner nli electra replaced token detection pretraining sample efficient modernbert 2024 modernized encoder with rotary embeddings flashattention 8k context current go to encoder for classification ner retrieval neobert 2025 250m parameter encoder integrating modern architecture improvements rope 4k context optimized depth to width state of the art on mteb surpasses modernbert and roberta large under identical fine tuning encoder decoder and seq2seq t5 and flan t5 text to text framing for nlp tasks strong instruction tuned encoder decoder baselines bart denoising seq2seq pretraining widely used for summarization and generation open decoder only lms used as substrate for nlp tasks llama 3 3 1 3 3 meta 2024 2025 widely adopted open weight family default base for fine tuning across nlp tasks qwen 2 5 qwen 3 alibaba 2024 2025 strong multilingual coverage especially chinese often top open model on multilingual benchmarks deepseek v3 2024 efficient moe pretraining competitive open base model olmo 2 ai2 2025 fully open weights training data code reproducibility benchmark gemma 2 gemma 3 google 2024 2025 open small mid size models with strong nlp task performance mistral mixtral efficient dense and sparse moe open models what language model architecture and pretraining objective work best for zero shot generalization encoder vs decoder vs encoder decoder for nlp transfer multilingual and cross lingual models xlm r cross lingual masked lm trained on commoncrawl 100 languages mt5 multilingual t5 covering 101 languages bloom 176b parameter open multilingual lm 46 natural languages aya 23 aya expanse cohere for ai 2024 massively multilingual instruction tuned models covering 23 101 languages glot500 encoder for 500 languages focus on low resource nllb 200 no language left behind mt for 200 languages madlad 400 400 language mt model and 3t token multilingual corpus seamlessm4t seamless meta 2023 2024 multilingual and multimodal speech text translation 100 languages sea lion seallm 2024 2025 lms targeting southeast asian languages babel 2025 open multilingual llms 9b and 83b covering the top 25 languages by speaker population 90 of global speakers surpasses comparably sized open multilingual models on xcopa xnli mgsm flores 200 lugha llama princeton mila 2025 llama 3 1 8b adapted for low resource african languages via the curated wura corpus sota open source results on irokobench and afriqa afriquellm mcgill 2026 suite of open llms 4b 14b continued pretrained on 26b tokens across 20 african languages with a comprehensive empirical study of data mixing translategemma google 2026 open translation specialized models built on gemma 3 covering 55 language pairs via sft and rl with quality reward models milmmt 46 xiaomi 2026 open multilingual mt scaled across 46 languages matching commercial systems like google translate and gemini 3 pro evaluation and benchmarks nlu and cross lingual glue and superglue english nlu benchmarks xtreme and xglue cross lingual nlu xnli cross lingual natural language inference 15 languages flores 200 mt evaluation across 200 languages mteb massive text embedding benchmark standard for sentence document encoders beir heterogeneous ir benchmark for retrieval models modern lm evaluation 2023 2026 helm holistic evaluation across nlp tasks accuracy and beyond big bench 200 tasks probing language model capabilities mmlu multitask knowledge evaluation across 57 subjects mmlu pro 2024 harder more discriminative successor to mmlu gpqa graduate level q a google proof reasoning evaluation refute 2026 scientific reasoning benchmark for evidence grounded critique overclaim detection missing evidence refusal and calibration ifeval verifiable instruction following evaluation chatbot arena lmsys human preference elo leaderboard for chat models livebench 2024 contamination resistant benchmark with monthly refresh lm evaluation harness unified framework for lm benchmark evaluation mmlu prox 2025 multilingual extension of mmlu pro to 29 typologically diverse languages reveals up to 24 3 performance gap between high and low resource languages multichallenge 2025 multi turn conversational benchmark exposing simultaneous instruction following and in context reasoning failures all tested frontier models score below 50 frames 2025 unified rag evaluation 824 multi hop questions requiring factuality retrieval accuracy and cross document reasoning together long context evaluation needle in a haystack retrieval probe for long context windows ruler 2024 synthetic long context tasks beyond simple retrieval longbench bilingual long context benchmark across nlp tasks longbench v...
|
