Meta tags:
description=A Library to parse natural language in pure Clojure and ClojureScript - turbopape/postagga;

Headings (most frequently used words):

postagga, of, uh, oh, code, and, navigation, saved, searches, the, contributing, conduct, license, to, files, tagger, contributors, footer, search, repositories, users, issues, pull, requests, provide, feedback, turbopape, getting, how, does, it, work, workflow, complete, list, features, todo, credits, menu, use, filter, your, results, more, quickly, folders, latest, commit, history, repository, training, pos, using, parse, free, speech, backers, sponsors, about, releases, packages, languages, patching, viterbi, output, meaning, tags, topics, resources, stars, watchers, forks,

Text of the page (most frequently used words):
the (116), you (60), #postagga (38), and (35), can (34), this (33), for (28), your (27), that (21), with (21), all (19), model (18), parser (17), code (16), rules (16), tags (16), step (15), github (14), parse (13), state (13), models (13), clojure (12), tagger (12), project (12), french (12), from (12), use (11), get (11), security (10), pos (10), language (10), viterbi (10), our (10), sample (10), data (10), have (10), value (10), corpus (10), these (10), ponct (10), sentence (9), what (9), structure (9), one (9), how (9), two (9), was (8), please (8), like (8), refer (8), some (8), qui (8), heureux (8), are (8), example (8), adj (8), edn (8), reload (7), license (7), will (7), see (7), name (7), tag (7), into (7), rule (7), vector (7), say (7), first (7), cls (7), word (7), noun (7), english (7), annotated (7), view (7), action (6), not (6), information (6), error (6), contributing (6), clojurescript (6), natural (6), features (6), issues (6), patch (6), result (6), which (6), words (6), def (6), input (6), also (6), multiple (6), npp (6), require (6), verb (6), more (6), out (6), train (6), enterprise (6), navigation (5), there (5), while (5), conduct (5), about (5), support (5), tokenizer (5), test (5), them (5), work (5), text (5), call (5), map (5), output (5), want (5), another (5), namespace (5), sentences (5), parsers (5), search (5), community (4), terms (4), loading (4), page (4), readme (4), algorithm (4), logo (4), great (4), its (4), tab (4), free (4), version (4), changelog (4), mood (4), dictionary (4), function (4), above (4), any (4), tell (4), using (4), sets (4), such (4), det (4), find (4), possible (4), extract (4), look (4), now (4), last (4), subject (4), speech (4), used (4), generated (4), corpora (4), tools (4), namespaces (4), take (4), phrase (4), turbopape (4), actions (4), contributors (3), repository (3), stars (3), mit (3), resources (3), topics (3), under (3), rafik (3), make (3), etc (3), workflow (3), case (3), enough (3), integrate (3), patching (3), errors (3), suis (3), set (3), jvm (3), need (3), just (3), specify (3), keyword (3), steps (3), following (3), states (3), entry (3), grab (3), having (3), finally (3), instance (3), then (3), meaning (3), provide (3), indochine (3), actually (3), does (3), source (3), clj (3), files (3), commit (3), open (3), insights (3), pull (3), requests (3), signed (3), settings (3), window (3), refresh (3), session (3), sign (3), saved (3), documentation (3), feedback (3), grade (3), copilot (3), platform (3), solutions (3), explore (3), time (2), manage (2), footer (2), packages (2), releases (2), forks (2), 162 (2), bots (2), processing (2), library (2), pure (2), sponsor (2), here (2), become (2), sponsors (2), backers (2), thanks (2), who (2), contribute (2), note (2), track (2), feel (2), included (2), other (2), than (2), training (2), names (2), nicolas (2), force (2), entries (2), close (2), parsing (2), each (2), large (2), repl (2), being (2), nil (2), down (2), defined (2), full (2), fledged (2), specific (2), start (2), alternatives (2), different (2), might (2), captured (2), through (2), product (2), tells (2), optional (2), grand (2), beau (2), multi (2), why (2), key (2), where (2), structured (2), yielded (2), let (2), put (2), assign (2), basically (2), machine (2), before (2), tagged (2), trained (2), adjective (2), provided (2), dictionaries (2), but (2), should (2), tries (2), cljc (2), proper (2), nouns (2), when (2), know (2), type (2), every (2), must (2), small (2), fr_tb_v_model (2), suite (2), process (2), without (2), sequoia (2), processed (2), trainer (2), vars (2), issue (2), guerre (2), computed (2), create (2), sure (2), qualify (2), parts (2), self (2), contained (2), easily (2), apples (2), object (2), loves (2), part (2), semantic (2), able (2), external (2), clear (2), samples (2), hand (2), write (2), understanding (2), file (2), size (2), git (2), clone (2), add (2), thought (2), png (2), code_of_conduct (2), gitignore (2), src (2), commits (2), message (2), menu (2), quality (2), projects (2), appearance (2), cancel (2), available (2), searches (2), repositories (2), business (2), advanced (2), developer (2), customer (2), services (2), devops (2), app (2), perform, share, personal, cookies, contact, docs, status, privacy, 2026, inc, 100, languages, published, report, watching, watchers, activity, distributed, created, talented, friend, chakib, daoud, happily, brought, fekr, copyright, 2017, naccache, credits, becoming, show, link, website, thank, backer, exists, people, released, participating, agree, abide, contributor, enhancements, bugs, send, prs, todo, per, tests, complete, list, trie, est, core, afterwards, looks, takes, computes, entity, patched, wraps, discussed, proceed, follows, something, reported, collection, mapping, failed, quite, careful, spit, contents, directly, drilled, detected, detailed, curried, partial, back, holding, string, split, hey, works, only, break, line, waiting, develop, couple, naive, splits, strings, space, characters, capturing, talk, person, attribute, specified, via, stanza, raise, belonging, present, parait, feed, tokenized, encountered, repeatedly, certain, had, flag, corresponding, referring, upstream, reflected, associated, related, somewhere, putting, led, representing, confusing, isn, goes, encompassing, refers, matched, relate, preferred, wants, assigned, either, deserves, explanation, carry, alternate, atep, drill, understand, currently, done, detecting, token, doing, detect, reference, technically, speed, lookup, may, evolve, during, considered, mere, details, implementation, phase, hereafter, en_tr_names, fr_tr_names, encounters, doesn, generate, arbitrarily, assigns, less, randomly, picked, somehow, enhance, detection, known, accordingly, given, hmm, forcing, happening, fed, form, tokens, fr_sequoia_pos_v_model, en_fn_v_model, licensing, extent, derived, corpuscule, en_fn_model, fr_tb_model, exposed, embed, functionality, chose, lightest, ones, limit, possibility, cause, network, resource, tree, bank, inria, framenet, beware, avoid, realizing, printing, var, unsurprisingly, named, république, quatrième, française, catégories, piastres, des, affaire, résumé, dernier, fraudes, les, sur, colloque, constitute, relies, hidden, markov, makes, matrices, likely, transition, eventually, compiled, single, third, party, dependency, run, servers, browser, really, trying, naturally, handle, much, complex, lets, express, literally, label, packaging, comes, offers, needs, particular, depicts, carrying, represented, coming, after, undergo, has, enable, taggers, relying, libs, care, however, consistent, annotating, reliable, yourself, favor, maintaining, analysis, tagging, implies, fidelity, mechanism, phrases, easier, interpret, common, way, extracting, grammatical, meant, study, simple, next, magic, extracts, compare, many, finds, match, shall, meaningful, making, compatible, across, readers, conditional, easy, ship, cool, bot, shipped, light, javascript, artifacts, concern, requiring, load, cloned, folder, https, com, walk, around, lib, clojars, dependencies, getting, assist, generating, efficient, processors, capable, welcome, george, orwell, 1984, corrupts, corrupt, items, history, latest, date, folders, branches, master, additional, options, star, fork, change, notification, notifications, public, dismiss, alert, switched, accounts, resetting, focus, qualifiers, query, filter, results, quickly, submit, include, email, address, contacted, read, piece, very, seriously, syntax, tips, users, jump, pricing, premium, ons, powered, collections, trending, archive, program, accelerator, maintainer, lab, programs, fund, developers, partners, trust, center, forum, skills, ebooks, reports, events, webinars, stories, software, development, topic, industries, government, manufacturing, financial, healthcare, industry, cases, devsecops, modernization, nonprofits, startups, medium, teams, enterprises, company, marketplace, blog, stop, leaks, they, secret, protection, secure, build, fix, vulnerabilities, application, changes, review, plan, instant, dev, environments, codespaces, automate, workflows, mcp, registry, new, direct, agents, merge, better, creation, toggle, skip, content,

Text of the page (random words):
ojure and clojurescript github skip to content navigation menu toggle navigation sign in appearance settings platform ai code creation github copilot write better code with ai github copilot app direct agents from issue to merge mcp registry new integrate external tools developer workflows actions automate any workflow codespaces instant dev environments issues plan and track work code review manage code changes application security github advanced security find and fix vulnerabilities code security secure your code as you build secret protection stop leaks before they start explore why github documentation blog changelog marketplace view all features solutions by company size enterprises small and medium teams startups nonprofits by use case app modernization devsecops devops ci cd view all use cases by industry healthcare financial services manufacturing government view all industries view all solutions resources explore by topic ai software development devops security view all topics explore by type customer stories events webinars ebooks reports business insights github skills support services documentation customer support community forum trust center partners view all resources open source community github sponsors fund open source developers programs security lab maintainer community accelerator github stars archive program repositories topics trending collections enterprise enterprise solutions enterprise platform ai powered developer platform available add ons github advanced security enterprise grade security features copilot for business enterprise grade ai features premium support enterprise grade 24 7 support pricing search or jump to search code repositories users issues pull requests search clear search syntax tips provide feedback we read every piece of feedback and take your input very seriously include my email address so i can be contacted cancel submit feedback saved searches use saved searches to filter your results more quickly name query to see all available qualifiers see our documentation cancel create saved search sign in sign up appearance settings resetting focus you signed in with another tab or window reload to refresh your session you signed out in another tab or window reload to refresh your session you switched accounts on another tab or window reload to refresh your session dismiss alert message turbopape postagga public notifications you must be signed in to change notification settings fork 15 star 162 code issues 11 pull requests 1 actions projects security and quality 0 insights additional navigation options code issues pull requests actions projects security and quality insights turbopape postagga master branches tags go to file code open more actions menu folders and files name name last commit message last commit date latest commit history 81 commits 81 commits github github models models src postagga src postagga test postagga test postagga gitignore gitignore changelog md changelog md code_of_conduct md code_of_conduct md contributing md contributing md license license readme md readme md logo png logo png project clj project clj view all files repository files navigation readme code of conduct contributing mit license more items postagga but if thought corrupts language language can also corrupt thought george orwell 1984 postagga is a suite of tools to assist you in generating efficient and self contained natural language processors you can use postagga to process annotated text samples into full fledged parsers capable of understanding free speech input as structured data ah and you ll be able to do this easily you re welcome getting postagga you can refer postagga as a lib in your clojure project grab it from clojars in your dependencies in project clj just add you can also clone the project and walk around the source and models git clone https github com turbopape postagga git the models are included under the models folder in jvm clojure provided you have cloned the repository def fr model load edn models fr_tb_v_model edn for french for instance we also shipped two light models as vars defined in namespaces one for french and one for english as for javascript the artifacts size are a concern you can use these models by requiring the two namespaces ns your cool bot require postagga en fn v model refer en model for english postagga fr tb v model refer fr model for french these namespaces make it easy for you to ship parsers for clojurescript you can see an example on how to work with this model all while making sure your code is compatible across clojure and clojurescript thanks to readers conditional in the test file how does it work to do its magic postagga extracts the phrase structure of your input and tries to find how does this structure compare to its many semantic rules and if it finds a match where in this structure shall it extract meaningful information let s study a simple example look at the next sentence rafik loves apples that is our natural language input first step in understanding this sentence is to extract some structure from it so it is easier to interpret one common way to do this is extracting its grammatical phrase structure which is close enough to what function words are actually meant to provide noun verb noun that was the phrase structure analysis or as we call it pos part of speech tagging these tags qualify parts of the sentence as the name implies and will be used as a hi fidelity mechanism to write rules for parsers of such phrases postagga has tools that enable you to train pos taggers for any language you want without relying on external libs actually it does not care about the meaning of the tags at all however you should be consistent and clear enough when annotating your input data samples with tags on one hand your parser will be more reliable on the other hand you ll do yourself a great favor maintaining your parser now comes the parser part actually postagga offers a parser that needs semantic rules to be able to map a particular phrase structure into data in our example we know that the first noun depicts a subject carrying out some action this action is represented by the verb following it finally the noun coming after the verb will undergo this action postagga parsers lets you express such rules so it can extract the data for you you literally tell it to take the first noun call it subject take the verb label it action take the last noun call it the object finally packaging it all into the following data structure subject rafik action loves object apples naturally postagga can handle much more complex sentences postagga parsers are eventually compiled into self contained packages with no single third party dependency from there it can easily run on servers clojure version and on the browser clojurescript now your bots can really get what you re trying to tell them the postagga workflow training a pos tagger first of all you need to train a pos tagger that can qualify parts of your natural text postagga relies on hidden markov models computed with the viterbi algorithm this algorithm makes use of a set of matrices like what states the pos tags we have how likely we transition from one pos tag to another etc all of these constitute a model these are computed out of what we call an annotated text corpus the postagga trainer namespace is used create models out of such annotated text corpus to train a model make sure you have an annotated corpus like so a vector of sentences like this one ponct guerre nc d p indochine npp ponct colloque nc sur p les det fraudes nc ponct dernier adj résumé nc ponct l det ponct affaire nc des p d piastres nc ponct catégories nc ponct guerre nc d p indochine npp ponct indochine npp française adj ponct quatrième adj république nc ponct etc say you have this corpus that is a vector of annotated sentences in a var unsurprisingly named corpus to train a model just issue require postagga trainer refer train def model train corpus beware these can be large vars so avoid realizing all of them like printing in your repl we processed one annotated corpus for english postagga fn en edn generated from the framenet project we also processed two annotated corpora for french postagga sequoia fr edn generated from the sequoia corpus from inria postagga tb fr edn generated from the free french tree bank we exposed two of these models as clojure namespaces so you can embed them without using the resource functionality as it is specific to clojure jvm we chose the two lightest ones to limit the possibility cause network issues french model as a namespace postagga fr_tb_model english model as a namespace postagga en_fn_model the suite of tools used to process these two corpora are in the corpuscule project please refer to the licensing of these corpora to see what extent you can use work derived from them we then trained a model out of the above english corpus en_fn_v_model edn and two models out of these two french corpora fr_sequoia_pos_v_model edn fr_tb_v_model edn now you can use that model to assign pos tags to speech note sentences must be fed in the form of a vector of all small case tokens require postagga tagger refer viterbi viterbi model je suis heureux cls v adj patching viterbi s output when the tagger encounters a word it doesn t know about that is was not in the corpus used to generate the viterbi models it arbitrarily assigns it a tag more or less randomly picked by the algorithm to somehow enhance the detection it is possible to patch the output that is look it up in a dictionary of terms of a known type and force the tags accordingly for instance if you have a dictionary for proper nouns in a given language you can patch your hmm generated pos tags by forcing every word happening to be an entry in this dictionary to have the npp tag we provide two dictionaries for proper nouns fr_tr_names cljc for french en_tr_names cljc for english you can see how you can integrate patching in the parsing phase hereafter technically dictionaries are tries to speed up lookup for multiple entries but this may evolve during time and should be considered as mere details implementation meaning of tags a reference to the meaning of tags is provided for english using the tagger to parse free speech now that you have your tagger trained you can use a parser to drill the information from your sentences for our last example say you want postagga to understand how you currently feel or how you look it can be done by detecting the first token as being a subject cls doing a verb v and then having an adjective adj we want to detect who is having what adjective in our sentence for this we ll use the postagga parser namespace first of all require the namespace require postagga parser refer parse tags rules then you ll need to specify rules for the parser we want to grab the word tagged as cls and the word tagged as adj as our information here s what the parser rules look like def sample rules rule tb french je suis heureux id sample rule tb french optional steps rule qui a atep get value cls a state in the parse machine i e a set of possible sets of pos tags or product det get value nc an alternate possible state at this step mood another step v get value adj this deserves some explanation before we carry on with our example the parser is basically a state machine it goes through steps qui mood with each step encompassing multiple states v a state basically refers to words it is matched with tag sets a word can relate to multiple tags if your preferred tagger wants to different tag sets can be assigned to a state for instance to say that in some state we require either a noun npp or a verb v you might put v npp putting the keyword get value in a state tells the parser to grab the word having led to this state put in the yielded parse map and finally assign it to a key representing the step in which that state was in confusing isn t it you ll get it with an example let s say that somewhere we have qui a step get value cls a state with get value under the qui step the value of the word that yielded the tag cls which is je in our example will be reflected on the output map as an entry in some vector associated with the related step which is qui qui je this is what the postagga parser is all about you tell it where to extract information and how you want it structured for upstream processing if we had multiple states with get value flag on we ll find multiple words in the corresponding entry in the output this is why the step key is referring a vector of words in the output map it is also possible to say that a state can be encountered repeatedly using the multi keyword if you say in certain state some step get value multi adj and if you feed postagga the following tokenized sentence il parait beau grand heureux you ll find in the parse map some step beau grand heureux the optional steps stanza tells the parser not to raise an error if a step belonging to this vector is not present at any step you can specify multiple alternatives for capturing different sets of information in the above example you can say that the first step in your sentence might talk about a person captured through the cls attribute or a product captured by specified a det than a nc you specify such alternatives via the or keyword you ll also need to tell the parser how to break down a line of text into a vector of words we call this a tokenizer waiting to develop a full fledged couple with language specific rules we can just start by a naive one that splits strings using space characters hey this one works only on clojure jvm version def sample tokenizer fn clojure string split s back to our sample with sample rules holding a set of rules as defined above you can parse your sentence like so def parse result parse tags rules sample tokenizer fn the tokenizer function partial viterbi model the tagger function curried with a model sample rules the parser rules je suis heureux the sentence to parse and you d have a detailed result like so errors nil the error if any result rule sample rule tb french which rule was detected data qui je the data structure drilled down from the input mood heureux the errors will be reported as a collection mapping each rule to what step and state the parser failed this can be quite large so be careful not to spit the contents of the result directly into your repl you can test on the errors being nil and work with the data value do something with data parse result to integrate patching as discussed above to the parsing you can proceed as follows def patch fr tagger w name a function that wraps viterbi into a patched version patch w entity 0 9 en names trie takes a sentence computes tags with viterbi and afterwards looks if the words are close enough to entries in the french names dictionary in which case it will force them to have nc tag viterbi fr model nc postagga core test patch fr tagger w name parse tags rules sample tokenizer fn p...