Meta tags:
description=Dask is a flexible open-source Python library for parallel computing maintained by OSS contributors across dozens of companies including Anaconda, Coiled, SaturnCloud, and nvidia.;

Headings (most frequently used words):

dask, for, you, can, on, use, to, parallel, submit, fast, and, easy, what, big, machine, scale, where, run, model, python, new, event, news, feature, sign, up, newsletter, do, with, pandas, loops, arrays, learning, performance, at, machines, made, humans, cheap, efficient, open, source, deployment, managed, cloud, users, say, about, organizations, that, cases, super, get, started, most, people, just, their, laptops, out, 100, gib, datasets, works, great, single, nyc, uber, lyft, scalable, etl, pipeline, national, water, tpc, benchmark, nasa, satellite, imagery, xgboost, tuning, want, more, examples,

Text of the page (most frequently used words):
dask (55), data (22), you (18), and (17), the (16), your (13), with (12), use (11), import (11), easy (10), #python (10), cloud (10), get (8), scale (8), can (8), parquet (8), tools (8), code (8), for (8), documentation (7), run (7), model (7), cluster (7), has (7), client (7), that (6), distributed (6), how (6), pandas (6), read_parquet (6), value (6), compute (6), parallel (6), page (5), blog (5), just (5), machine (5), xarray (5), spark (5), been (5), dataframe (5), started (4), xgboost (4), like (4), open (4), fast (4), humans (4), sum (4), big (4), submit (4), new (3), youtube (3), demo (3), day (3), powered (3), help (3), community (3), 100 (3), datasets (3), install (3), now (3), files (3), coiled (3), zarr (3), uber (3), did (3), users (3), across (3), making (3), are (3), computing (3), what (3), from (3), mean (3), source (3), account (3), deployment (3), process (3), machines (3), cheap (3), xgb (3), learning (3), oops (3), something (3), went (3), wrong (3), while (3), submitting (3), form (3), thank (3), submission (3), received (3), news (3), brand (2), people (2), out (2), works (2), single (2), lightweight (2), super (2), channel (2), netcdf (2), standard (2), benchmark (2), results (2), tpc (2), 250 (2), water (2), dataframes (2), nyc (2), work (2), all (2), sooner (2), easily (2), clusters (2), deploy (2), locally (2), using (2), processes (2), way (2), runs (2), few (2), quickly (2), any (2), ryan (2), climate (2), hpc (2), structures (2), research (2), made (2), large (2), multi (2), dimensional (2), about (2), capital (2), one (2), training (2), get_client (2), video (2), kubernetes (2), saas (2), managed (2), where (2), localcluster (2), numpy (2), groupby (2), mybucket (2), myfile (2), matures (2), benchmarks (2), faster (2), than (2), performance (2), dtrain (2), train (2), array (2), arrays (2), futures (2), loops (2), much (2), sign (2), newsletter (2), feature (2), event (2), copyright, core, developers, bsd, licensed, 2022, discourse, group, stackoverflow, twitter, github, guidelines, contribute, latest, proceed, most, their, laptops, gib, great, pip, complete, conda, right, computer, simple, visit, want, more, examples, tuning, geotiff, nasa, satellite, imagery, sql, polars, duckdb, flox, national, json, records, prefect, streamlit, deltalake, scalable, etl, pipeline, lyft, anything, cases, organizations, john, renken, rebuy, used, time, move, grateful, meenakshi, sharma, wayfair, further, accelerate, our, ability, expanded, this, setting, pre, configured, horovod, hugues, demers, grubhub, also, makes, multiple, nearly, identical, full, production, load, jacqueline, nolis, fanatics, inc, adapt, entire, very, other, let, going, language, abernathy, earthmover, transformed, world, interacts, weather, geospatial, processing, pipelines, things, seemed, impossible, five, years, ago, routine, thanks, ajith, aravind, simeio, shines, when, dealing, generic, which, don, conform, table, pyspark, rdds, but, who, wants, debug, verbose, java, logs, paige, martin, pangeo, science, possible, integrates, seamlessly, computations, focus, instead, thinking, parallelism, mcentee, early, implementations, have, reduced, times, within, months, development, effort, wes, mckinney, trailblazer, memory, accessible, everyone, solved, problem, massive, won, alone, say, n_workers, region, east, solution, free, individuals, modest, anyone, false, path, laptop, trivial, resource, manager, job, schedulers, services, even, legacy, hadoop, often, per, tib, cost, 1000000000000, rows, computed, efficient, dashboard, guides, towards, efficiency, teaching, become, expert, fortunately, already, know, computers, expensive, stack, today, raw, without, getting, virtualization, compilers, daskdmatrix, tree_method, hist, example, common, libraries, predict, increasing, accuracy, image, natively, open_mfdataset, dimensions, dims, lat, lon, aggregate, churn, through, terabytes, formats, hdf, tiff, define, own, def, return, map, range, gather, 000, parallelize, matter, complex, flexible, supports, arbitrary, dependencies, fine, grained, task, scheduling, pay, base_passenger_fare, drivers, make, driver_pay, under, hood, current, likely, easier, too, does, need, docs, events, mailing, list, love,

Text of the page (random words):
dask scale the python tools you love get started community community demo day get help powered by mailing list get help powered by news events blog blog docs submit new event submit news feature sign up for newsletter submit new event thank you your submission has been received oops something went wrong while submitting the form submit news feature thank you your submission has been received oops something went wrong while submitting the form sign up for newsletter thank you your submission has been received oops something went wrong while submitting the form parallel python fast and easy easy parallel python that does what you need get started what you can do with dask big pandas parallel for loops big arrays machine learning big pandas dask dataframes use pandas under the hood so your current code likely just works it s faster than spark and easier too documentation performance benchmarks import dask dataframe as dd df dd read_parquet s3 data uber how much did nyc pay uber df base_passenger_fare sum compute and how much did drivers make df driver_pay sum compute parallel for loops parallelize your python code no matter how complex dask is flexible and supports arbitrary dependencies and fine grained task scheduling documentation process 5 000 files in parallel from dask distributed import client client client define your own code def f x return x 1 run your code in parallel futures client map f range 100 results client gather futures big arrays use dask and numpy xarray to churn through terabytes of multi dimensional array data in formats like hdf netcdf tiff or zarr documentation aggregate 250 tb of water model data import xarray as xr open image array files natively ds xr open_mfdataset data nc process across dimensions ds mean dims lat lon compute machine learning use dask with common machine learning libraries to train or predict on large datasets increasing model accuracy by using all of your data documentation example xgboost model training import xgboost as xgb import dask dataframe as dd df dd read_parquet s3 my data dtrain xgb dask daskdmatrix df model xgb dask train dtrain tree_method hist performance at scale fast on machines dask is lightweight and runs your raw code on your machines without getting in the way no virtualization or compilers as the python stack matures your code matures today dask is 50 faster than spark on standard benchmarks import pandas as pd df pd read_parquet s3 mybucket myfile parquet df df df value 0 df groupby account value sum import dask dataframe as dd df dd read_parquet s3 mybucket myfile parquet df df df value 0 df groupby account value sum compute made for humans computers are cheap humans are expensive fortunately humans already know how to use dask it s just python it s just pandas it s just numpy dask s dashboard guides you towards efficiency quickly teaching you to become a distributed computing expert cheap and efficient fast humans fast machines cheap computing rows of data computed 1000000000000 cost 0 00 dask users often process cloud data at 0 10 per tib open source deployment managed cloud where you can run dask open source deployment run dask on your laptop it s trivial or deploy it on any resource manager like kubernetes an hpc job schedulers cloud saas services or even legacy hadoop spark clusters documentation dask deployment video from dask distributed import localcluster cluster localcluster processes false client cluster get_client use dask locally import dask dataframe as dd df dd read_parquet path to data parquet df value mean compute where you can run dask managed cloud run dask in the cloud with open source kubernetes or with an easy saas solution coiled is free for individuals with modest use and easy for anyone with a cloud account documentation video dask in the cloud from coiled import cluster cluster cluster n_workers 100 region us east 2 client cluster get_client use dask on the cloud import dask dataframe as dd df dd read_parquet s3 data parquet df value mean compute what users say about dask people use dask and like it you won t be alone it s easy it s massive it solved my problem dask has been a trailblazer in making distributed and out of memory computing in python easy and accessible for everyone wes mckinney pandas at capital one early implementations of dask have reduced model training times by 91 within a few months of development effort ryan mcentee capital one my climate science research has been made possible by dask dask integrates seamlessly with xarray making it easy to run large scale computations on multi dimensional datasets i can focus on my research instead of thinking about parallelism paige martin pangeo dask shines when dealing with generic data structures which don t conform to table like structures pyspark has rdds but who wants to code in python and debug verbose java logs ajith aravind simeio dask has transformed how the world interacts with weather climate and geospatial data by making it super easy to scale up data processing pipelines on hpc or cloud things that seemed impossible five years ago are now routine thanks to dask ryan abernathy earthmover with dask i can easily adapt code that runs on a single machine and scale it across an entire cluster very few other tools let you get going that quickly across any language jacqueline nolis fanatics inc dask also makes it easy to deploy distributed work locally using multiple python processes in a way that is nearly identical to how full production load is distributed hugues demers grubhub to further accelerate our users ability to scale easily on the cloud we expanded this by setting up pre configured horovod and dask clusters meenakshi sharma wayfair i used to work in spark all the time the sooner you move to dask the sooner you ll be grateful you did john renken rebuy organizations that use dask use cases anything you can do with python you can scale with dask nyc uber lyft tools dataframes parquet data 50 gb of parquet data scalable etl pipeline tools deltalake streamlit prefect data 2 tb of tpc h json records national water model tools xarray zarr flox data 250 tb of zarr data tpc h benchmark tools spark duckdb polars data standard sql benchmark results nasa satellite imagery tools geotiff xarray coiled data 1 tb of netcdf files xgboost model tuning tools xgboost parquet data 75 gb of parquet data want more examples visit the dask blog or dask youtube channel dask demo day dask youtube channel super easy to get started you can run dask right now on your computer it s simple and lightweight install now conda install dask pip install dask complete most people use dask just on their laptops to scale out to 100 gib datasets dask works great on a single machine proceed to documentation get started latest community get help powered by demo day blog dask blog contribute brand brand guidelines go to github page go to twitter page go to youtube page go to stackoverflow page go to dask discourse group page copyright 2022 dask core developers new bsd licensed