Meta tags:
description=Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models. - GitHub - proycon/colibri-core: Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models.;

Headings (most frequently used words):

uh, oh, navigation, installation, saved, searches, colibri, core, files, packages, footer, from, search, code, repositories, users, issues, pull, requests, provide, feedback, proycon, menu, use, to, filter, your, results, more, quickly, folders, and, latest, commit, history, repository, technical, details, demo, publication, about, releases, 30, contributors, languages, python, binding, source, container, usage, topics, resources, license, stars, watchers, forks,

Text of the page (most frequently used words):
the (45), and (36), #colibri (27), core (27), you (20), github (15), this (14), #pattern (14), for (14), pxi (14), with (13), your (13), more (12), build (11), from (11), security (10), python (10), proycon (10), install (10), code (10), patterns (9), view (9), all (9), use (8), container (8), are (8), available (8), reload (7), docker (7), please (6), corpus (6), library (6), size (6), make (6), python3 (6), dev (6), models (6), pxd (6), enterprise (6), can (5), that (5), navigation (5), there (5), repository (5), license (5), skipgram (5), tool (5), gaps (5), memory (5), data (5), command (5), line (5), tools (5), autoconf (5), source (5), which (5), search (5), community (4), was (4), error (4), while (4), loading (4), page (4), packages (4), latest (4), readme (4), resources (4), grams (4), skipgrams (4), one (4), either (4), open (4), software (4), follows (4), pull (4), archive (4), devel (4), documentation (4), model (4), window (4), setup (4), benchmarks (4), actions (4), issues (4), support (4), not (3), information (3), stars (3), gpl (3), nlp (3), topics (3), efficient (3), patternmodeller (3), allows (3), manipulate (3), publication (3), van (3), gram (3), flexgram (3), add (3), run (3), will (3), development (3), image (3), bootstrap (3), configure (3), brew (3), libtool (3), homebrew (3), gcc (3), libtar (3), need (3), installation (3), colibricore (3), occurrence (3), relations (3), extracted (3), specific (3), query (3), over (3), quickly (3), format (3), representation (3), see (3), files (3), changelog (3), include (3), workflows (3), commit (3), name (3), insights (3), requests (3), signed (3), settings (3), another (3), tab (3), refresh (3), session (3), sign (3), saved (3), feedback (3), grade (3), features (3), copilot (3), platform (3), solutions (3), explore (3), manage (2), footer (2), 2026 (2), shell (2), releases (2), forks (2), 131 (2), processing (2), linguistics (2), plus (2), well (2), working (2), basic (2), linguistic (2), constructions (2), such (2), fixed (2), dynamic (2), quick (2), way (2), whi (2), work (2), gompel (2), doi (2), http (2), org (2), following (2), want (2), into (2), various (2), version (2), also (2), root (2), sudo (2), automake (2), pkg (2), config (2), macos (2), zlib (2), pip (2), bzip2 (2), cython3 (2), systems (2), environment (2), distribution (2), note (2), these (2), binding (2), full (2), collection (2), threshold (2), length (2), may (2), indexed (2), statistics (2), informed (2), form (2), simply (2), their (2), count (2), any (2), categories (2), notion (2), reports (2), extraction (2), computed (2), abstracting (2), counting (2), several (2), type (2), binary (2), frequent (2), classes (2), take (2), less (2), space (2), than (2), trivial (2), results (2), large (2), abstract (2), extract (2), unordered_set (2), unordered_map (2), test (2), cfg (2), pyproject (2), toml (2), colibricore_wrapper (2), pyx (2), colibricore_unindexedpatternmodel (2), colibricore_relationmap (2), colibricore_patternset (2), colibricore_patternmodel (2), colibricore_patterndict (2), colibricore_indexedpatternmodel (2), colibricore_classes (2), colibricore_alignmodel (2), tutorial (2), ipynb (2), colibri_wrapper (2), pyxdep (2), codereview_input (2), codemeta (2), json (2), brainstorm2 (2), news (2), makefile (2), manifest (2), doxyfile (2), dockerfile (2), authors (2), travis (2), yml (2), gitignore (2), clang (2), src (2), scripts (2), exp (2), 472 (2), commits (2), last (2), message (2), menu (2), quality (2), wiki (2), projects (2), appearance (2), cancel (2), searches (2), repositories (2), business (2), advanced (2), developer (2), customer (2), services (2), devops (2), app (2), perform, action, time, share, personal, cookies, contact, docs, status, privacy, terms, inc, other, jupyter, notebook, cython, languages, contributors, feb, report, watching, watchers, activity, ngrams, ngram, text, computational, recognition, about, access, cite, here, den, bosch, 2016, modelling, journal, research, e30, 5334, jors, 105, extensively, described, peer, reviewed, demo, parameter, mount, volume, path, frog, interactively, dump, where, arg, builds, stable, release, git, instead, yourself, sure, pre, made, obtained, hub, oci, usage, bash, then, clone, yum, pkgconfig, redhat, based, apt, get, autotools, libbz2, zlib1g, debian, ubuntu, order, sane, necessary, dependencies, compile, explained, later, otherwise, check, has, package, alpine, linux, contain, tap, fbkarsdorp, lamachine, apk, strongly, recommend, virtual, only, unix, like, windows, supported, just, using, consult, https, standalone, technical, details, generation, optionally, parametrised, minimum, maximum, lower, boundary, different, types, instantiate, possible, fillings, much, powerful, inferred, retains, indices, original, cost, diskspace, simplest, stores, unindexed, three, counts, come, two, varieties, heart, sofware, used, enables, generate, statistical, computationally, demanding, but, have, been, implemented, similar, optimisations, flexgrams, turn, directly, basis, mutual, pointwise, performed, intelligently, iteratively, passes, discarding, won, reach, desired, iterative, each, word, assigned, numeric, class, encoded, compact, highly, always, uses, rather, string, both, disk, compressed, seem, fairly, first, few, lines, favourite, scripting, language, move, simple, sliding, store, some, kind, hashmap, approach, however, makes, unnecessarily, high, demand, often, becomes, prohibitive, unleashed, corpora, tries, minimise, requirements, ways, variable, predetermined, multiple, consecutive, words, efficiently, compute, between, employed, construction, encompasses, licensed, under, gplv3, www, gnu, licenses, html, maarten, radboud, university, nijmegen, anaproy, items, history, date, folders, file, tags, branches, master, additional, options, star, fork, must, change, notification, notifications, public, dismiss, alert, switched, accounts, out, resetting, focus, create, qualifiers, our, filter, submit, email, address, contacted, read, every, piece, input, very, seriously, provide, syntax, tips, clear, users, jump, pricing, premium, ons, powered, collections, trending, program, accelerator, maintainer, lab, programs, fund, developers, sponsors, partners, trust, center, forum, skills, ebooks, events, webinars, stories, topic, industries, government, manufacturing, financial, healthcare, industry, cases, devsecops, modernization, case, nonprofits, startups, small, medium, teams, enterprises, company, marketplace, blog, why, stop, leaks, before, they, start, secret, protection, secure, find, fix, vulnerabilities, application, changes, review, plan, track, instant, environments, codespaces, automate, workflow, integrate, external, mcp, registry, new, direct, agents, issue, merge, write, better, creation, toggle, skip, content,

Text of the page (random words):
github proycon colibri core colibri core is an nlp tool as well as a c and python library for working with basic linguistic constructions such as n grams and skipgrams i e patterns with one or more gaps either of fixed or dynamic size in a quick and memory efficient way at the core is the tool colibri patternmodeller whi ch allows you to build view manipulate and query pattern models github skip to content navigation menu toggle navigation sign in appearance settings platform ai code creation github copilot write better code with ai github copilot app direct agents from issue to merge mcp registry new integrate external tools developer workflows actions automate any workflow codespaces instant dev environments issues plan and track work code review manage code changes application security github advanced security find and fix vulnerabilities code security secure your code as you build secret protection stop leaks before they start explore why github documentation blog changelog marketplace view all features solutions by company size enterprises small and medium teams startups nonprofits by use case app modernization devsecops devops ci cd view all use cases by industry healthcare financial services manufacturing government view all industries view all solutions resources explore by topic ai software development devops security view all topics explore by type customer stories events webinars ebooks reports business insights github skills support services documentation customer support community forum trust center partners view all resources open source community github sponsors fund open source developers programs security lab maintainer community accelerator github stars archive program repositories topics trending collections enterprise enterprise solutions enterprise platform ai powered developer platform available add ons github advanced security enterprise grade security features copilot for business enterprise grade ai features premium support enterprise grade 24 7 support pricing search or jump to search code repositories users issues pull requests search clear search syntax tips provide feedback we read every piece of feedback and take your input very seriously include my email address so i can be contacted cancel submit feedback saved searches use saved searches to filter your results more quickly name query to see all available qualifiers see our documentation cancel create saved search sign in sign up appearance settings resetting focus you signed in with another tab or window reload to refresh your session you signed out in another tab or window reload to refresh your session you switched accounts on another tab or window reload to refresh your session dismiss alert message proycon colibri core public notifications you must be signed in to change notification settings fork 20 star 131 code issues 9 pull requests 0 actions projects wiki security and quality 0 insights additional navigation options code issues pull requests actions projects wiki security and quality insights proycon colibri core master branches tags go to file code open more actions menu folders and files name name last commit message last commit date latest commit history 1 472 commits 1 472 commits github workflows github workflows exp exp include include m4 m4 scripts scripts src src clang format clang format gitignore gitignore travis yml travis yml authors authors changelog changelog dockerfile dockerfile doxyfile doxyfile license license manifest in manifest in makefile am makefile am news news readme md readme md benchmarks py benchmarks py benchmarks sh benchmarks sh bootstrap sh bootstrap sh brainstorm2 brainstorm2 codemeta json codemeta json codereview_input codereview_input colibri core pc in colibri core pc in colibri_wrapper pyxdep colibri_wrapper pyxdep colibricore python tutorial ipynb colibricore python tutorial ipynb colibricore_alignmodel pxi colibricore_alignmodel pxi colibricore_classes in pxd colibricore_classes in pxd colibricore_indexedpatternmodel pxi colibricore_indexedpatternmodel pxi colibricore_patterndict pxi colibricore_patterndict pxi colibricore_patternmodel pxi colibricore_patternmodel pxi colibricore_patternset pxi colibricore_patternset pxi colibricore_relationmap pxi colibricore_relationmap pxi colibricore_unindexedpatternmodel pxi colibricore_unindexedpatternmodel pxi colibricore_wrapper in pyx colibricore_wrapper in pyx configure ac configure ac pyproject toml pyproject toml setup cfg setup cfg setup py setup py test py test py unordered_map pxd unordered_map pxd unordered_set pxd unordered_set pxd view all files repository files navigation readme gpl 3 0 license more items colibri core by maarten van gompel proycon anaproy nl radboud university nijmegen licensed under gplv3 see http www gnu org licenses gpl 3 0 html colibri core is software to quickly and efficiently count and extract patterns from large corpus data to extract various statistics on the extracted patterns and to compute relations between the extracted patterns the employed notion of pattern or construction encompasses the following categories n gram n consecutive words skipgram an abstract pattern of predetermined length with one or multiple gaps of specific size flexgram an abstract pattern with one or more gaps of variable size n gram extraction may seem fairly trivial at first with a few lines in your favourite scripting language you can move a simple sliding window of size n over your corpus and store the results in some kind of hashmap this trivial approach however makes an unnecessarily high demand on memory resources this often becomes prohibitive if unleashed on large corpora colibri core tries to minimise these space requirements in several ways compressed binary representation each word type is assigned a numeric class which is encoded in a compact binary format in which highly frequent classes take less space than less frequent classes colibri core always uses this representation rather than a full string representation both on disk and in memory informed iterative counting counting is performed more intelligently by iteratively processing the corpus in several passes and quickly discarding patterns that won t reach the desired occurrence threshold skipgram and flexgram extraction are computationally more demanding but have been implemented with similar optimisations skipgrams are computed by abstracting over n grams and flexgrams in turn are computed either by abstracting over skipgrams or directly from n grams on the basis of co occurrence information mutual pointwise information at the heart of the sofware is the notion of pattern models the core tool to be used from the command line is colibri patternmodeller which enables you to build pattern models generate statistical reports query for specific patterns and relations and manipulate models a pattern model is simply a collection of extracted patterns any of the three categories and their counts from a specific corpus pattern models come in two varieties unindexed pattern model the simplest form which simply stores the patterns and their count indexed pattern model the more informed form which retains all indices to the original corpus at the cost of more memory diskspace the indexed pattern model is much more powerful and allows more statistics and relations to be inferred the generation of pattern models is optionally parametrised by a minimum occurrence threshold a maximum pattern length and a lower boundary on the different types that may instantiate a skipgram i e possible fillings of the gaps technical details colibri core is available as a collection of standalone command line tools as a c library and as a python library please consult the full documentation at https proycon github io colibri core installation python binding for the colibri core python library just install using pip install colibricore we strongly recommend you use a virtual environment for this do note that this is only available for unix like systems windows is not supported installation from packages for the command line tools check if your distribution has a package available there are packages for alpine linux apk add colibri core and for macos with homebrew brew tap fbkarsdorp homebrew lamachine brew install colibri core note that these do not contain the python binding otherwise you will need to either use the container image or to build and install from source installation from source if no packages are available you will need to compile from source or use the container build e g docker as explained later on in order to do so you need a sane build environment install the necessary dependencies for your distribution for debian ubuntu sudo apt get install make gcc g pkg config autoconf archive libtool autotools dev libbz2 dev zlib1g dev libtar dev python3 python3 dev cython3 for redhat based systems run as root yum install pkgconfig libtool autoconf automake autoconf archive make gcc gcc c libtar libtar devel python3 python3 devel zlib zlib devel python3 pip bzip2 bzip2 devel cython3 for macos with homebrew brew install autoconf automake libtool autoconf archive python3 pkg config then clone this repository and install as follows bash bootstrap configure make sudo make install container usage the colibri core command line tools are also available as an oci docker container a pre made container image can be obtained from docker hub as follows docker pull proycon colibri core you can also build a container image yourself as follows make sure you are in the root of this repository docker build t proycon colibri core this builds the latest stable release if you want to use the latest development version from the git repository instead do docker build t proycon colibri core build arg version development run the frog container interactively as follows it will dump you into a shell where the various command line tools are available docker run t i proycon colibri core add the v path to your data data parameter if you want to mount your data volume into the container at data demo publication this software is extensively described in the following peer reviewed publication van gompel m and van den bosch a 2016 efficient n gram skipgram and flexgram modelling with colibri core journal of open research software 4 e30 doi http dx doi org 10 5334 jors 105 access the publication here and please cite it if you make use of colibri core in your work about colibri core is an nlp tool as well as a c and python library for working with basic linguistic constructions such as n grams and skipgrams i e patterns with one or more gaps either of fixed or dynamic size in a quick and memory efficient way at the core is the tool colibri patternmodeller whi ch allows you to build view manipulate a proycon github io colibri core topics python nlp c plus plus library corpus linguistics pattern recognition computational linguistics text processing ngram ngrams skipgram resources readme license gpl 3 0 license uh oh there was an error while loading please reload this page activity stars 131 stars watchers 8 watching forks 20 forks report repository releases 30 v2 5 10 latest feb 2 2026 29 releases packages 0 uh oh there was an error while loading please reload this page uh oh there was an error while loading please reload this page contributors uh oh there was an error while loading please reload this page languages c 70 8 cython 13 4 jupyter notebook 9 8 shell 2 8 python 2 5 m4 0 2 other 0 5 footer 2026 github inc footer navigation terms privacy security status community docs contact manage cookies do not share my personal information you can t perform that action at this time