Meta tags:
description=Apache Airflow - A platform to programmatically author, schedule, and monitor workflows - apache/airflow;

Headings (most frequently used words):

airflow, apache, github, launching, requirements, packages, who, the, desktop, cookies, latest, commit, git, stats, files, readme, md, project, focus, principles, getting, started, installing, from, pypi, official, source, code, convenience, user, interface, contributing, uses, maintains, can, use, logo, in, my, presentation, merchandise, links, about, releases, 211, 54, contributors, 395, languages, join, today, is, where, world, builds, software, additional, notes, on, python, version, topics, resources, license, learn, contribute, connect, with, others, xcode, visual, studio, essential, always, active, analytics,

Text of the page (most frequently used words):
airflow (75), 2020 (62), the (61), and (51), #apache (41), nov (29), for (28), #github (27), are (20), use (17), you (17), dec (17), with (16), code (15), can (14), python (13), rst (13), more (12), that (12), files (12), time (11), packages (11), using (10), latest (9), official (9), installing (9), our (8), they (8), build (8), your (8), this (8), dag (8), not (8), release (8), install (8), images (8), aug (8), used (7), workflows (7), source (7), asf (7), docs (7), docker (7), pip (7), from (7), constraints (7), version (7), setup (7), data (7), enable (7), breeze (7), cookies (6), them (6), about (6), learn (6), security (6), releases (6), check (6), license (6), platform (6), have (6), software (6), but (6), view (6), tasks (6), dependencies (6), all (6), user (6), stable (6), 2019 (6), added (6), json (6), schema (6), fix (6), information (5), com (5), author (5), monitor (5), documentation (5), work (5), who (5), dags (5), those (5), git (5), project (5), pypi (5), backport (5), constraint (5), when (5), scripts (5), auto (5), sep (5), refactor (5), add (5), commit (5), back (5), download (5), desktop (5), sign (5), analytics (4), how (4), task (4), update (4), out (4), help (4), readme (4), org (4), programmatically (4), schedule (4), requirements (4), contributing (4), development (4), policy (4), kubernetes (4), providers (4), https (4), keep (4), set (4), note (4), branch (4), master (4), open (4), should (4), will (4), tests (4), pipelines (4), static (4), projects (4), yamllint (4), yml (4), black (4), provider (4), style (4), nothing (4), happens (4), try (4), again (4), launching (4), jump (4), preferences (3), understand (3), make (3), better (3), visit (3), essential (3), website (3), signed (3), dockerfile (3), scheduler (3), foundation (3), most (3), requests (3), review (3), uses (3), duration (3), their (3), environment (3), interface (3), following (3), tool (3), convenience (3), process (3), google (3), txt (3), extras (3), working (3), package (3), same (3), described (3), getting (3), started (3), sqlite (3), message (3), issues (3), checks (3), config (3), oct (3), 12682 (3), validate (3), output (3), testing (3), yaml (3), markdownlint (3), 12720 (3), speed (3), prefix (3), permissions (3), migration (3), web (3), actions (3), repository (3), reject (2), accept (2), always (2), perform (2), third (2), party (2), products (2), see (2), privacy (2), cookie (2), optional (2), another (2), tab (2), window (2), refresh (2), session (2), reload (2), api (2), pricing (2), contact (2), status (2), css (2), shell (2), contributors (2), 211 (2), topics (2), links (2), would (2), etc (2), merchandise (2), logo (2), presentation (2), around (2), feature (2), become (2), please (2), core (2), community (2), maintains (2), want (2), different (2), over (2), run (2), tree (2), artifacts (2), officially (2), released (2), sources (2), pre (2), via (2), tags (2), helm (2), order (2), ways (2), other (2), users (2), rules (2), postgres (2), raw (2), githubusercontent (2), installation (2), however (2), known (2), branches (2), per (2), versions (2), url (2), both (2), library (2), libraries (2), usually (2), needed (2), complete (2), production (2), mysql (2), running (2), has (2), workers (2), dynamic (2), allows (2), principles (2), solution (2), results (2), pass (2), one (2), next (2), metadata (2), best (2), type (2), similar (2), include (2), focus (2), rich (2), licences (2), yam (2), 5180 (2), 12636 (2), get (2), non (2), conflicting (2), replace (2), remove (2), pytest (2), ini (2), 12735 (2), friendly (2), 12548 (2), restructure (2), 12704 (2), list (2), rendering (2), commands (2), host (2), jul (2), inthewild (2), 12738 (2), pins (2), dockerfiles (2), code_of_conduct (2), changelog (2), 12427 (2), rule (2), md003 (2), heading (2), header (2), improve (2), 9550 (2), formmatter (2), context (2), chart (2), 386f6b2 (2), visual (2), studio (2), xcode (2), cli (2), dismiss (2), developers (2), world (2), together (2), insights (2), discussions (2), pull (2), watch (2), star (2), stars (2), education (2), plans (2), explore (2), team (2), management (2), jan (2), save, websites, gather, pages, many, clicks, need, accomplish, active, functions, log, selection, clicking, bottom, page, statement, action, blog, training, terms, inc, javascript, html, languages, 384, 395, 210, upgrade, resources, workflow, chat, love, stickers, shirt, then, redbubble, shop, yes, sure, abide, date, logos, found, repo, brandbook, trademark, policies, responsible, reviewing, merging, prs, well, steering, conversation, new, like, maintainer, committer, committers, maintainers, than, 350, organizations, wild, quick, way, overlap, gantt, total, spent, visualization, current, specific, graph, representation, spans, across, overview, prepared, some, ones, clearly, marked, such, retrieve, were, generate, charts, read, customising, extending, details, internals, document, swarm, compose, standard, common, people, there, methods, stated, themselves, must, sufficient, test, provided, access, appropriate, tools, voted, pmc, members, during, approval, cryptographically, manager, downloaded, distribution, directory, follow, example, just, repeatable, introduced, updated, also, orphan, separately, major, minor, specify, correct, tag, publish, might, sometimes, tricky, because, bit, application, applications, pin, neither, decided, possible, means, plain, produce, unusable, container, roadmap, improvement, proposals, aips, wiki, looking, find, walking, through, tutorial, least, requires, additional, notes, mariadb, unable, limitations, multiple, schedulers, postgresql, 0dev, tested, modular, architecture, queue, orchestrate, arbitrary, number, scalable, lean, explicit, parameterizing, built, into, powerful, templating, engine, jinja, elegant, easily, define, own, operators, executors, extend, fits, level, abstraction, suits, extensible, configuration, allowing, pipeline, generation, writing, instantiates, dynamically, streaming, often, real, pulling, off, streams, batches, commonly, opinion, ideally, idempotent, create, duplicated, destination, system, large, quantities, though, high, volume, intensive, practice, delegate, external, services, specialize, xcom, works, mostly, slowly, changing, structure, clarity, unit, continuity, azkaban, oozie, luigi, table, contents, directed, acyclic, graphs, executes, array, while, specified, command, line, utilities, performing, complex, surgeries, snap, makes, easy, visualize, progress, troubleshoot, defined, maintainable, versionable, testable, collaborative, simply, pkg_resources, importlib, avoid, versionconflic, cfg, 10923, unknown, setting, 10545, connexion, folders, pyproject, toml, 10930, introduce, taskmixin, pylintrc, 10695, implement, conventions, script, confirm, 11709, enforce, strict, codecov, upgrading_to_2, updating, 12374, static_code_checks, 12727, stronger, language, 12433, case, pull_request_workflow, apr, 8071, housekeeping, auth, backend, doc, notice, 12383, adds, mechanism, discovery, manifest, 9889, now, maintained, automatically, local_virtualenv, 12204, warnings, visible, 12734, headout, 12652, don, timezone, utcnow, calculations, 9715, 12623, 12551, rtd, readthedocs, 11487, support, rat, excludes, 12729, 12715, 12286, alias, shortlog, mailmap, hadolint, 12444, separate, building, gitignore, 10543, flake8, feb, 7338, magic, comments, utf, 6714, editorconfig, 11693, framework, dockerignore, 10143, bring, coverage, coveragerc, 4932, simplified, 3611, bash_completion, 12525, protection, 12674, slackwebhookhook, password, instead, extra, 12200, provider_packages, metastore_browser, 12181, simplifies, whether, image, rebuilt, manifests, 11035, toggle, pure, licenses, consistent, 5234, templates, 12479, kube, kubernetes_tests, 12163, k9s, integrated, 9653, group, subdirectories, hooks, may, 8891, prepare, candidate, empty, allow, _cmd, _secret, webserver, secret_key, 12078, typo, dev, clients, 12646, jobs, delete, improved, idempotency, 12718, verification, name, failed, load, permalink, commits, 945, stats, jhtimmins, extension, zip, fast, checkout, svn, clone, file, free, millions, companies, ship, maintain, largest, advanced, where, builds, home, million, manage, join, today, 188, 716, forks, fork, 741, suggested, nonprofit, sales, compare, marketplace, program, forum, events, connect, others, guides, learning, lab, trending, collections, contribute, enterprise, customer, stories, mobile, hosting, integrations, features, why, skip, content, wayback, machine, http, archive, 20201203033224, timestamps, capture, fail, success, 2021, 2026, 911, captures,

Text of the page (random words):
apache 2 0 license 19 4k stars 7 6k forks star watch code issues 716 pull requests 188 discussions actions projects 4 security insights more code issues pull requests discussions actions projects security insights dismiss join github today github is home to over 50 million developers working together to host and review code manage projects and build software together sign up github is where the world builds software millions of developers and companies build ship and maintain their software on github the largest and most advanced development platform in the world sign up for free dismiss master 15 branches 211 tags go to file code clone https github cli use git or checkout with svn using the web url work fast with our official cli learn more open with github desktop download zip launching github desktop if nothing happens download github desktop and try again go back launching github desktop if nothing happens download github desktop and try again go back launching xcode if nothing happens download xcode and try again go back launching visual studio if nothing happens download the github extension for visual studio and try again go back latest commit jhtimmins refactor and speed up dag prefix permissions migration 12720 386f6b2 dec 2 2020 refactor and speed up dag prefix permissions migration 12720 386f6b2 git stats 10 945 commits files permalink failed to load latest commit information type name latest commit message commit time github improve verification of images with pip check 12718 dec 1 2020 airflow refactor and speed up dag prefix permissions migration 12720 dec 2 2020 chart fix chart jobs delete policy for improved idempotency 12646 dec 1 2020 clients enable markdownlint rule md003 heading style header style 12427 nov 18 2020 dags enable black python auto formmatter 9550 nov 3 2020 dev user friendly output of breeze and ci scripts 12735 dec 1 2020 docker context files fix typo in docker context files readme md 12078 nov 4 2020 docs allow using _cmd _secret to set webserver secret_key config 1 dec 2 2020 empty prepare release candidate for backport packages 8891 may 17 2020 hooks group ci scripts in subdirectories 9653 jul 16 2020 images added k9s as integrated tool to help with kubernetes testing 12163 nov 11 2020 kubernetes_tests fix kube tests 12479 nov 19 2020 license templates airflow 5234 rst files have consistent auto added license aug 18 2019 licenses replace js package toggle w pure css solution 11035 sep 21 2020 manifests simplifies check whether the ci image should be rebuilt 12181 nov 13 2020 metastore_browser enable black python auto formmatter 9550 nov 3 2020 provider_packages include data files in backport provider packages 12200 nov 9 2020 scripts refactor list rendering in commands 12704 dec 2 2020 tests slackwebhookhook use password instead of extra 12674 dec 2 2020 asf yaml enable v1 10 stable branch protection 12525 nov 21 2020 bash_completion airflow 3611 simplified development environment 4932 aug 27 2019 coveragerc bring back code coverage 10143 aug 5 2020 dockerignore add python helm testing framework 11693 oct 28 2020 editorconfig airflow 6714 remove magic comments about utf 8 7338 feb 2 2020 flake8 enable black on providers packages 10543 aug 25 2020 gitignore separate out documentation building per provider 12444 nov 20 2020 hadolint yaml airflow 5180 added static checks yamllint auto licences for yam aug 22 2019 mailmap add an alias to improve git shortlog output 12286 nov 11 2020 markdownlint yml enable markdownlint rule md003 heading style header style 12427 nov 18 2020 pre commit config yaml fix static checks 12715 12729 dec 1 2020 rat excludes added support for provider packages for airflow 2 0 11487 oct 13 2020 readthedocs yml fix build on rtd 12551 nov 23 2020 breeze rst validate json schema files with json schema 12682 nov 28 2020 changelog txt add changelog for airflow 1 10 13 12623 nov 26 2020 ci rst update setup py to get non conflicting set of dependencies 12636 nov 27 2020 code_of_conduct md add apache airflow code_of_conduct md 9715 aug 5 2020 contributing rst don t use time time or timezone utcnow for duration calculations nov 29 2020 dockerfile pins pip to 20 2 4 in our dockerfiles 12738 dec 1 2020 dockerfile ci pins pip to 20 2 4 in our dockerfiles 12738 dec 1 2020 images rst add 1 10 13 to ci breeze and docs 12652 nov 27 2020 install restructure the extras in setup py and described them 12548 nov 27 2020 inthewild md added headout to inthewild 12734 dec 1 2020 license make warnings more visible 12204 nov 9 2020 local_virtualenv rst constraint files are now maintained automatically 9889 jul 20 2020 manifest in adds mechanism for provider package discovery 12383 nov 17 2020 notice housekeeping of auth backend update security doc 8071 apr 3 2020 pull_request_workflow rst fix case of github 12433 nov 18 2020 readme md stronger language re sqlite 12727 dec 1 2020 static_code_checks rst validate json schema files with json schema 12682 nov 28 2020 testing rst all kubernetes tests use the same host python version 12374 nov 15 2020 updating md refactor list rendering in commands 12704 dec 2 2020 upgrading_to_2 0 md restructure the extras in setup py and described them 12548 nov 27 2020 breeze user friendly output of breeze and ci scripts 12735 dec 1 2020 breeze complete validate json schema files with json schema 12682 nov 28 2020 codecov yml enforce strict rules for yamllint 11709 oct 21 2020 confirm implement google shell conventions for breeze script 10695 sep 2 2020 pylintrc introduce taskmixin 10930 sep 16 2020 pyproject toml enable black on connexion api folders 10545 aug 25 2020 pytest ini remove unknown pytest ini setting 10923 sep 14 2020 setup cfg replace pkg_resources with importlib metadata to avoid versionconflic nov 29 2020 setup py update setup py to get non conflicting set of dependencies 12636 nov 27 2020 yamllint config yml airflow 5180 added static checks yamllint auto licences for yam aug 22 2019 view code readme md apache airflow apache airflow or simply airflow is a platform to programmatically author schedule and monitor workflows when workflows are defined as code they become more maintainable versionable testable and collaborative use airflow to author workflows as directed acyclic graphs dags of tasks the airflow scheduler executes your tasks on an array of workers while following the specified dependencies rich command line utilities make performing complex surgeries on dags a snap the rich user interface makes it easy to visualize pipelines running in production monitor progress and troubleshoot issues when needed table of contents project focus principles requirements getting started installing from pypi official source code convenience packages user interface contributing who uses apache airflow who maintains apache airflow can i use the apache airflow logo in my presentation airflow merchandise links project focus airflow works best with workflows that are mostly static and slowly changing when dag structure is similar from one run to the next it allows for clarity around unit of work and continuity other similar projects include luigi oozie and azkaban airflow is commonly used to process data but has the opinion that tasks should ideally be idempotent i e results of the task will be the same and will not create duplicated data in a destination system and should not pass large quantities of data from one task to the next though tasks can pass metadata using airflow s xcom feature for high volume data intensive tasks a best practice is to delegate to external services that specialize on that type of work airflow is not a streaming solution but it is often used to process real time data pulling data off streams in batches principles dynamic airflow pipelines are configuration as code python allowing for dynamic pipeline generation this allows for writing code that instantiates pipelines dynamically extensible easily define your own operators executors and extend the library so that it fits the level of abstraction that suits your environment elegant airflow pipelines are lean and explicit parameterizing your scripts is built into the core of airflow using the powerful jinja templating engine scalable airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers requirements apache airflow is tested with master version 2 0 0dev stable version 1 10 13 python 3 6 3 7 3 8 2 7 3 5 3 6 3 7 3 8 postgresql 9 6 10 11 12 13 9 6 10 11 12 13 mysql 5 7 8 5 6 5 7 sqlite latest stable latest stable kubernetes 1 16 9 1 17 5 1 18 6 1 16 9 1 17 5 1 18 6 note mariadb and mysql 5 x are unable to or have limitations with running multiple schedulers please see the scheduler docs note sqlite is used in airflow tests do not use it in production additional notes on python version requirements stable version requires at least python 3 5 3 when using python 3 getting started visit the official airflow website documentation latest stable release for help with installing airflow getting started or walking through a more complete tutorial note if you re looking for documentation for master branch latest development branch you can find it on s apache org airflow docs for more information on airflow s roadmap or airflow improvement proposals aips visit the airflow wiki official docker container images for apache airflow are described in images rst installing from pypi we publish apache airflow as apache airflow package in pypi installing it however might be sometimes tricky because airflow is a bit of both a library and application libraries usually keep their dependencies open and applications usually pin them but we should do neither and both at the same time we decided to keep our dependencies as open as possible in setup py so users can install different versions of libraries if needed this means that from time to time plain pip install apache airflow will not work or will produce unusable airflow installation in order to have repeatable installation however introduced in airflow 1 10 10 and updated in airflow 1 10 12 we also keep a set of known to be working constraint files in the orphan constraints master and constraints 1 10 branches we keep those known to be working constraints files separately per major minor python version you can use them as constraint files when installing airflow from pypi note that you have to specify correct airflow tag version branch and python versions in the url installing just airflow pip install apache airflow 1 10 13 constraint https raw githubusercontent com apache airflow constraints 1 10 13 constraints 3 7 txt installing with extras for example postgres google pip install apache airflow postgres google 1 10 13 constraint https raw githubusercontent com apache airflow constraints 1 10 13 constraints 3 7 txt for information on installing backport providers check docs backport providers rst docs backport providers rst official source code apache airflow is an apache software foundation asf project and our official source code releases follow the asf release policy can be downloaded from the asf distribution directory are cryptographically signed by the release manager are officially voted on by the pmc members during the release approval process following the asf rules the source packages released must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools convenience packages there are other ways of installing and using airflow those are convenience methods they are not official releases as stated by the asf release policy but they can be used by the users who do not want to build the software themselves those are in the order of most common ways people install airflow pypi releases to install airflow using standard pip tool docker images to install airflow via docker tool use them in kubernetes helm charts docker compose docker swarm etc you can read more about using customising and extending the images in the latest docs and learn details on the internals in the images rst document tags in github to retrieve the git project sources that were used to generate official source packages via git all those artifacts are not official releases but they are prepared using officially released sources some of those artifacts are development or pre release ones and they are clearly marked as such following the asf policy user interface dags overview of all dags in your environment tree view tree representation of a dag that spans across time graph view visualization of a dag s dependencies and their current status for a specific run task duration total time spent on different tasks over time gantt view duration and overlap of a dag code view quick way to view source code of a dag contributing want to help build apache airflow check out our contributing documentation who uses apache airflow more than 350 organizations are using apache airflow in the wild who maintains apache airflow airflow is the work of the community but the core committers maintainers are responsible for reviewing and merging prs as well as steering conversation around new feature requests if you would like to become a maintainer please review the apache airflow committer requirements can i use the apache airflow logo in my presentation yes be sure to abide by the apache foundation trademark policies and the apache airflow brandbook the most up to date logos are found in this repo and on the apache software foundation website airflow merchandise if you would love to have apache airflow stickers t shirt etc then check out redbubble shop links documentation chat about apache airflow a platform to programmatically author schedule and monitor workflows airflow apache org topics airflow apache apache airflow python scheduler workflow resources readme license apache 2 0 license releases 211 apache airflow upgrade check 1 0 0 latest dec 2 2020 210 releases packages 54 51 packages contributors 1 395 1 384 contributors languages python 94 5 shell 3 6 html 1 4 javascript 0 2 dockerfile 0 2 css 0 1 2020 github inc terms privacy cookie preferences security status help contact github pricing api training blog about you can t perform that action at this time you signed in with another tab or window reload to refresh your session you signed out in another tab or window reload to refresh your session we use optional third party analytics cookies to understand how you use github com so we can build better products learn more accept reject we use optional third party analytics cookies to understand how you use github com so we can build better products you can always update your selection by clicking cookie preferences at the bottom of the page for more information see our privacy statement essential cookies we use essential cookies to perform essential website functions e g they re used to log you in learn more always active analytics cookies we use analytics cookies to understand h...