Meta tags:
description=Pre-trained Transformers for Arabic Language Understanding and Generation (Arabic BERT, Arabic GPT2, Arabic ELECTRA) - aub-mind/arabert;

Headings (most frequently used words):

and, dataset, uh, oh, aragpt2, araelectra, model, navigation, compute, saved, searches, arabert, arabertv2, models, more, files, new, footer, search, code, repositories, users, issues, pull, requests, provide, feedback, aub, mind, update, installation, preprocessing, examples, notebooks, tensorflow, if, you, used, this, please, cite, us, as, acknowledgments, contacts, menu, use, to, filter, your, results, quickly, folders, latest, commit, history, repository, what, better, pre, processing, vocab, bigger, about, releases, contributors, languages, the, arabertpreprocessor, class, topics, resources, stars, watchers, forks,

Text of the page (most frequently used words):
the (62), and (60), aragpt2 (30), #arabert (28), arabic (26), for (26), base (26), model (22), araelectra (19), true (18), github (16), more (15), with (14), bert (14), from (14), large (14), you (13), name (13), dataset (13), code (13), new (12), all (12), this (11), huggingface (11), arabertv2 (11), security (10), pre (10), defaults (10), optional (10), bool (10), 2021 (9), models (9), examples (9), tpuv3 (9), size (9), readme (8), aubmindlab (8), trained (8), language (8), available (8), 128 (8), arabertv0 (8), was (7), reload (7), page (7), aub (7), twitter (7), antoun (7), use (7), model_name (7), set (7), preprocessing (7), num (7), are (7), view (7), can (6), that (6), please (6), dot (6), arabertv1 (6), false (6), force (6), none (6), arabertv02 (6), words (6), install (6), mega (6), png (6), your (6), enterprise (6), navigation (5), while (5), electra (5), baly (5), fady (5), wissam (5), support (5), data (5), https (5), mind (5), training (5), used (5), folder (5), numbers (5), vocabulary (5), arabertpreprocessor (5), 77gb (5), steps (5), medium (5), read (5), 543mb (5), 136m (5), tweets (5), search (5), time (4), community (4), there (4), error (4), loading (4), repository (4), resources (4), transformer (4), understanding (4), also (4), another (4), text (4), under (4), old (4), will (4), segmentation (4), issue (4), before (4), remove (4), نبالغ (4), إذا (4), هاتف (4), كمبيوتر (4), هذا (4), ضروري (4), function (4), preprocess (4), batch (4), 512 (4), 60m (4), see (4), 38g (4), now (4), better (4), 200m (4), actions (4), issues (4), requirements (3), stars (3), gpt2 (3), topics (3), transformers (3), generation (3), com (3), thanks (3), assafir (3), www (3), inproceedings (3), title (3), author (3), hajj (3), hazem (3), booktitle (3), natural (3), processing (3), workshop (3), apr (3), address (3), pages (3), 2020 (3), our (3), any (3), find (3), pytorch (3), tf2 (3), tf1 (3), library (3), work (3), automatically (3), disable (3), enable (3), when (3), non (3), arabert_prep (3), characters (3), import (3), pip (3), corpus (3), 256 (3), days (3), seq (3), len (3), hardware (3), compute (3), 135m (3), params (3), source (3), section (3), 300k (3), 420m (3), 207m (3), yes (3), 371m (3), files (3), workflows (3), commit (3), open (3), insights (3), pull (3), requests (3), signed (3), settings (3), tab (3), window (3), refresh (3), session (3), sign (3), saved (3), documentation (3), feedback (3), grade (3), features (3), copilot (3), platform (3), solutions (3), explore (3), not (2), manage (2), footer (2), 2022 (2), latest (2), forks (2), 148 (2), 727 (2), farasa (2), nlp (2), mail (2), edu (2), gmail (2), linkedin (2), tensorflow (2), cloud (2), access (2), have (2), done (2), without (2), program (2), lab (2), etal (2), rabic (2), proceedings (2), sixth (2), month (2), year (2), kyiv (2), ukraine (2), virtual (2), publisher (2), association (2), computational (2), linguistics (2), url (2), aclweb (2), org (2), anthology (2), wanlp (2), missing (2), check (2), file (2), notebooks (2), apply_farasa_segmentation (2), replaces (2), present (2), default (2), map_hindi_numbers_to_arabic (2), replace_slash_with_dash (2), replace (2), character (2), remove_non_digit_repetition (2), insert (2), whitespace (2), after (2), digits (2), english (2), then (2), between (2), insert_white_spaces (2), strip_tatweel (2), strip_tashkeel (2), whether (2), email (2), replace_urls_emails_mentions (2), should (2), remove_html_markup (2), keep_emojis (2), ولن (2), قلنا (2), المكتب (2), زمننا (2), output_text (2), unpreprocess (2), مكتب (2), زمن (2), changes (2), around (2), content (2), farasapy (2), wikipedia (2), oscar (2), unshuffled (2), filtered (2), added (2), filter (2), out (2), 516mb (2), discriminator (2), generator (2), details (2), checkpoints (2), formats (2), detector (2), long (2), 550k (2), 2056 (2), 13440 (2), 250k (2), 384 (2), 2560 (2), total (2), change (2), using (2), supports (2), implementation (2), wordpiece (2), vocab (2), space (2), 77m (2), 23gb (2), same (2), v02 (2), multi (2), dialect (2), two (2), emojies (2), powered (2), git (2), clone (2), scratch (2), setup (2), txt (2), arabert_logo (2), __init__ (2), gitignore (2), 122 (2), commits (2), last (2), message (2), menu (2), quality (2), discussions (2), appearance (2), cancel (2), searches (2), repositories (2), business (2), advanced (2), developer (2), customer (2), services (2), devops (2), app (2), perform, action, share, personal, information, cookies, contact, docs, status, privacy, terms, 2026, inc, 100, python, languages, contributors, release, aug, adds, emoji, fixed, ver, releases, report, watching, watchers, custom, properties, activity, classification, about, fgb06, wfa07, contacts, research, tfrc, free, tpus, couldn, members, continous, storage, habib, rahal, putting, face, behance, net, rahalhabib, yakshof, acknowledgments, discriminators, 191, 195, gpt, 196, 207, antoun2020arabert, based, lrec, evaluation, conference, may, google, scholar, has, bibtex, wrong, instead, cite, where, wget, resolve, main, tf1_model, tar, username, links, colab, hindi, corresponding, one, ١٩٩٥, 1995, behavior, fixes, caused, bug, inserting, white, spaces, since, repetition, than, digit, alphabet, brackets, inserts, tatweel, u0640, diacritics, fathatan, dammatan, kasratan, fatha, damma, kasra, sukun, shadda, urls, mentions, special, tokens, html, artfacts, tydi, don, emojis, tag, preprocessor, found, str, class, reverse, fixing, spacing, alphabetical, segmenting, selected, need, highly, recommend, unprocessing, generated, make, look, recommended, apply, testing, segment, news, articles, huge, thank, osian, dump, thoroughly, previous, but, websites, previously, crawled, consists, 200, 095, 961, lines, 655, 948, 860, 232, 988, 358, chars, applying, pretraining, 227mb, 800k, 220k, 125k, 1792, 1024, 5gb, 46b, 98gb, 792m, 370m, 527mb, 900k, tpuv2, 520m, 245m, times, longer, sources, bigger, how, codes, just, learnt, fast, tokenizer, tokenizers, bertwordpiecetokenizer, identified, came, punctuations, were, still, attached, punctuation, arabertv01, sentences, nwords, detail, paper, comes, variants, had, their, addition, common, weren, first, max, sentence, length, only, epoch, dialects, continuing, mlm, task, collection, 100m, what, gpt2lmheadmodel, modeling_gpt2, grover, follows, pypi, installation, system, demo, sep, oct, via, jul, update, checkout, 6a58ca118911ef311cbe8cdcdcc1d03601123291, want, versions, original, contains, items, history, date, folders, tags, branches, master, additional, options, star, fork, must, notification, notifications, public, dismiss, alert, switched, accounts, resetting, focus, create, qualifiers, query, results, quickly, submit, include, contacted, every, piece, take, input, very, seriously, provide, syntax, tips, clear, users, jump, pricing, premium, add, ons, collections, trending, archive, accelerator, maintainer, programs, fund, developers, sponsors, partners, trust, center, forum, skills, ebooks, reports, events, webinars, stories, type, software, development, topic, industries, government, manufacturing, financial, healthcare, industry, cases, devsecops, modernization, case, nonprofits, startups, small, teams, enterprises, company, marketplace, changelog, blog, why, stop, leaks, they, start, secret, protection, secure, build, fix, vulnerabilities, application, review, plan, track, instant, dev, environments, codespaces, automate, workflow, integrate, external, tools, mcp, registry, direct, agents, merge, write, creation, toggle, skip,

Text of the page (random words):
github aub mind arabert pre trained transformers for arabic language understanding and generation arabic bert arabic gpt2 arabic electra github skip to content navigation menu toggle navigation sign in appearance settings platform ai code creation github copilot write better code with ai github copilot app direct agents from issue to merge mcp registry new integrate external tools developer workflows actions automate any workflow codespaces instant dev environments issues plan and track work code review manage code changes application security github advanced security find and fix vulnerabilities code security secure your code as you build secret protection stop leaks before they start explore why github documentation blog changelog marketplace view all features solutions by company size enterprises small and medium teams startups nonprofits by use case app modernization devsecops devops ci cd view all use cases by industry healthcare financial services manufacturing government view all industries view all solutions resources explore by topic ai software development devops security view all topics explore by type customer stories events webinars ebooks reports business insights github skills support services documentation customer support community forum trust center partners view all resources open source community github sponsors fund open source developers programs security lab maintainer community accelerator github stars archive program repositories topics trending collections enterprise enterprise solutions enterprise platform ai powered developer platform available add ons github advanced security enterprise grade security features copilot for business enterprise grade ai features premium support enterprise grade 24 7 support pricing search or jump to search code repositories users issues pull requests search clear search syntax tips provide feedback we read every piece of feedback and take your input very seriously include my email address so i can be contacted cancel submit feedback saved searches use saved searches to filter your results more quickly name query to see all available qualifiers see our documentation cancel create saved search sign in sign up appearance settings resetting focus you signed in with another tab or window reload to refresh your session you signed out in another tab or window reload to refresh your session you switched accounts on another tab or window reload to refresh your session dismiss alert message uh oh there was an error while loading please reload this page aub mind arabert public notifications you must be signed in to change notification settings fork 148 star 727 code issues 2 pull requests 0 discussions actions security and quality 0 insights additional navigation options code issues pull requests discussions actions security and quality insights aub mind arabert master branches tags go to file code open more actions menu folders and files name name last commit message last commit date latest commit history 122 commits 122 commits github workflows github workflows arabert arabert araelectra araelectra aragpt2 aragpt2 examples examples gitignore gitignore araelectra png araelectra png aragpt2 png aragpt2 png readme md readme md __init__ py __init__ py arabert_logo png arabert_logo png preprocess py preprocess py requirements txt requirements txt setup py setup py view all files repository files navigation readme more items arabertv2 aragpt2 araelectra this repository now contains code and implementation for arabert v0 1 v1 original arabert v0 2 v2 base and large versions with better vocabulary more data more training read more aragpt2 base medium large and mega trained from scratch on arabic read more araelectra trained from scratch on arabic read more if you want to clone the old repository git clone https github com aub mind arabert cd arabert git checkout 6a58ca118911ef311cbe8cdcdcc1d03601123291 update 17 jul 2022 you can now install arabert via pip install arabert 8 oct 2021 new arabert models that better supports tweets and emojies 13 sep 2021 arabic nlp demo space on huggingface 02 apr 2021 araelectra powered arabic wikipedia qa system installation install arabert from pypi pip install arabert then use it as follows from arabert import arabertpreprocessor from arabert aragpt2 grover modeling_gpt2 import gpt2lmheadmodel arabertv2 what s new arabertv0 2 twitter base large are two new models for arabic dialects and tweets trained by continuing the pre training using the mlm task on 60m arabic tweets filtered from a collection on 100m the two new models have had emojies added to their vocabulary in addition to common words that weren t at first present the pre training was done with a max sentence length of 64 only for 1 epoch models arabert comes in 6 variants more detail in the arabert folder and in the readme and in the arabert paper model huggingface model name size mb params pre segmentation dataset sentences size nwords arabertv0 2 twitter base bert base arabertv02 twitter 543mb 136m no same as v02 60m multi dialect tweets arabertv0 2 twitter large bert large arabertv02 twitter 1 38g 371m no same as v02 60m multi dialect tweets arabertv0 2 base bert base arabertv02 543mb 136m no 200m 77gb 8 6b arabertv0 2 large bert large arabertv02 1 38g 371m no 200m 77gb 8 6b arabertv2 base bert base arabertv2 543mb 136m yes 200m 77gb 8 6b arabertv2 large bert large arabertv2 1 38g 371m yes 200m 77gb 8 6b arabertv0 1 base bert base arabertv01 543mb 136m no 77m 23gb 2 7b arabertv1 base bert base arabert 543mb 136m yes 77m 23gb 2 7b all models are available in the huggingface model page under the aubmindlab name checkpoints are available in pytorch tf2 and tf1 formats better pre processing and new vocab we identified an issue with arabertv1 s wordpiece vocabulary the issue came from punctuations and numbers that were still attached to words when we trained the wordpiece vocab we now insert a space between numbers and characters and around punctuation characters the new vocabulary was learnt using the bertwordpiecetokenizer from the tokenizers library and now supports the fast tokenizer implementation from the transformers library p s all the old bert codes should work with the new bert just change the model name and check the new preprocessing function please read the section on how to use the preprocessing function bigger dataset and more compute we used 3 5 times more data and trained for longer for dataset sources see the dataset section model hardware num of examples with seq len 128 512 128 batch size num of steps 512 batch size num of steps total steps total time in days arabertv0 2 base tpuv3 8 420m 207m 2560 1m 384 2m 3m 36 arabertv0 2 large tpuv3 128 420m 207m 13440 250k 2056 300k 550k 7 arabertv2 base tpuv3 8 420m 207m 2560 1m 384 2m 3m 36 arabertv2 large tpuv3 128 520m 245m 13440 250k 2056 300k 550k 7 arabert base v1 v0 1 tpuv2 8 512 900k 128 300k 1 2m 4 aragpt2 more details and code are available in the aragpt2 folder and readme model model huggingface model name size params aragpt2 base aragpt2 base 527mb 135m aragpt2 medium aragpt2 medium 1 38g 370m aragpt2 large aragpt2 large 2 98gb 792m aragpt2 mega aragpt2 mega 5 5gb 1 46b aragpt2 mega detector long aragpt2 mega detector long 516mb 135m all models are available in the huggingface model page under the aubmindlab name checkpoints are available in pytorch tf2 and tf1 formats dataset and compute for dataset source see the dataset section model hardware num of examples seq len 1024 batch size num of steps time in days aragpt2 base tpuv3 128 9 7m 1792 125k 1 5 aragpt2 medium tpuv3 8 9 7m 80 1m 15 aragpt2 large tpuv3 128 9 7m 256 220k 3 aragpt2 mega tpuv3 128 9 7m 256 800k 9 araelectra more details and code are available in the araelectra folder and readme model model huggingface model name size mb params araelectra base generator araelectra base generator 227mb 60m araelectra base discriminator araelectra base discriminator 516mb 135m dataset and compute model hardware num of examples seq len 512 batch size num of steps time in days electra base tpuv3 8 256 2m 24 dataset the pretraining data used for the new arabert model is also used for aragpt2 and araelectra the dataset consists of 77gb or 200 095 961 lines or 8 655 948 860 words or 82 232 988 358 chars before applying farasa segmentation for the new dataset we added the unshuffled oscar corpus after we thoroughly filter it to the previous dataset used in arabertv1 but with out the websites that we previously crawled oscar unshuffled and filtered arabic wikipedia dump from 2020 09 01 the 1 5b words arabic corpus the osian corpus assafir news articles huge thank you for assafir for the data preprocessing it is recommended to apply our preprocessing function before training testing on any dataset install farasapy to segment text for arabert v1 v2 pip install farasapy from arabert preprocess import arabertpreprocessor model_name aubmindlab bert base arabertv2 arabert_prep arabertpreprocessor model_name model_name text ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري arabert_prep preprocess text و لن نبالغ إذا قل نا إن هاتف أو كمبيوتر ال مكتب في زمن نا هذا ضروري you can also use the unpreprocess function to reverse the preprocessing changes by fixing the spacing around non alphabetical characters and also de segmenting if the model selected need pre segmentation we highly recommend unprocessing generated content of aragpt2 model to make it look more natural output_text و لن نبالغ إذا قل نا إن هاتف أو كمبيوتر ال مكتب في زمن نا هذا ضروري arabert_prep unpreprocess output_text ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري the arabertpreprocessor class arabertpreprocessor model_name keep_emojis false remove_html_markup true replace_urls_emails_mentions true strip_tashkeel true strip_tatweel true insert_white_spaces true remove_non_digit_repetition true replace_slash_with_dash none map_hindi_numbers_to_arabic none apply_farasa_segmentation none model_name str model name from the huggingface models page without the aubmindlab tag will default to a base arabic preprocessor if model name was not found keep_emojis bool optional defaults to false don t remove emojis while preprocessing remove_html_markup bool optional defaults to true whether to remove html artfacts should be set to false when preprocessing tydi qa replace_urls_emails_mentions bool optional defaults to true whether to replace email urls and mentions by special tokens strip_tashkeel bool optional defaults to true remove diacritics fathatan dammatan kasratan fatha damma kasra sukun shadda strip_tatweel bool optional defaults to true remove tatweel u0640 insert_white_spaces bool optional defaults to true insert whitespace before and after all non arabic digits or english digits or arabic and english alphabet or the 2 brackets then inserts whitespace between words and numbers or numbers and words remove_non_digit_repetition bool optional defaults to true replace repetition of more than 2 non digit character with 2 of this character replace_slash_with_dash bool optional defaults to none will be automatically set to true in arabertv02 araelectra and aragpt2 set to false to force disable and true to force enable replaces the with since is missing from arabertv2 araelectra and aragpt2 vocabulary map_hindi_numbers_to_arabic bool optional defaults to none will be automatically set to true in arabertv02 araelectra and aragpt2 set to false to force disable and true to force enable replaces hindi numbers with the corresponding arabic one ex ١٩٩٥ 1995 this is behavior is present by default in arabertv1 and v2 with pre segmentation and fixes the issue of caused by a bug when inserting white spaces apply_farasa_segmentation bool optional defaults to none will be automatically set to true in arabertv2 and arabertv1 set to false to force disable and true to force enable examples notebooks you can find the old examples that work with arabertv1 in the examples old folder check the readme md file in the examples folder for new links to colab notebooks tensorflow 1 x models you can find the pytorch tf2 and tf1 models in huggingface s transformer library under the aubmindlab username wget https huggingface co aubmindlab model_name resolve main tf1_model tar gz where model_name is any model under the aubmindlab name if you used this model please cite us as arabert google scholar has our bibtex wrong missing name use this instead inproceedings antoun2020arabert title arabert transformer based model for arabic language understanding author antoun wissam and baly fady and hajj hazem booktitle lrec 2020 workshop language resources and evaluation conference 11 16 may 2020 pages 9 aragpt2 inproceedings antoun etal 2021 aragpt2 title a ra gpt 2 pre trained transformer for a rabic language generation author antoun wissam and baly fady and hajj hazem booktitle proceedings of the sixth arabic natural language processing workshop month apr year 2021 address kyiv ukraine virtual publisher association for computational linguistics url https www aclweb org anthology 2021 wanlp 1 21 pages 196 207 araelectra inproceedings antoun etal 2021 araelectra title a ra electra pre training text discriminators for a rabic language understanding author antoun wissam and baly fady and hajj hazem booktitle proceedings of the sixth arabic natural language processing workshop month apr year 2021 address kyiv ukraine virtual publisher association for computational linguistics url https www aclweb org anthology 2021 wanlp 1 20 pages 191 195 acknowledgments thanks to tensorflow research cloud tfrc for the free access to cloud tpus couldn t have done it without this program and to the aub mind lab members for the continous support also thanks to yakshof and assafir for data and storage access another thanks for habib rahal https www behance net rahalhabib for putting a face to arabert contacts wissam antoun linkedin twitter github wfa07 at mail dot aub dot edu wissam antoun at gmail dot com fady baly linkedin twitter github fgb06 at mail dot aub dot edu baly fady at gmail dot com about pre trained transformers for arabic language understanding and generation arabic bert arabic gpt2 arabic electra huggingface co aubmindlab topics arabic bert arabic nlp electra farasa gpt2 huggingface transformer arabic classification arabert resources readme uh oh there was an error while loading please reload this page activity custom properties stars 727 stars watchers 29 watching forks 148 forks report repository releases 2 v1 0 1 adds emoji with fixed ver to requirements latest aug 1 2022 1 release uh oh there was an error while loading please reload this page contributors uh oh there was an error while loading please reload this page languages python 100 0 footer 2026 github inc footer navigation terms privacy security status community docs contact manage cookies do not share my personal information you can t perform that action at this time