Meta tags:

Headings (most frequently used words):

dask, how, to, contents, use, install, deploy, learn, with, examples,

Text of the page (most frequently used words):
dask (167), dataframe (104), series (70), api (33), and (25), rolling (24), for (17), learn (14), #documentation (13), data (13), use (12), the (12), more (12), python (12), cluster (12), how (11), you (11), memory (10), example (10), with (9), import (9), client (8), distributed (8), parallel (8), array (8), cat (8), install (7), that (7), arrays (7), result (7), deploy (6), your (6), kubernetes (6), bag (6), futures (6), performance (6), examples (5), get_client (5), popular (5), process (5), localcluster (5), datasets (5), see (5), processing (5), dataframes (5), mean (5), xarray (5), providing (5), execution (5), results (5), best (5), work (4), easy (4), this (4), can (4), used (4), large (4), scale (4), from (4), clusters (4), coiled (4), local (4), bags (4), are (4), one (4), collection (4), computers (4), compute (4), time (4), numpy (4), computation (4), allowing (4), larger (4), than (4), library (4), pandas (4), to_bag (4), graphs (4), practices (4), powerful (3), reference (3), project (3), cloud (3), create (3), single (3), conda (3), pip (3), json (3), simply (3), different (3), month (3), intuitive (3), var (3), random (3), parallelize (3), submit (3), second (3), custom (3), contents (2), fun (2), computing (2), name (2), kubecluster (2), operator (2), provides (2), jobqueue (2), 100 (2), account (2), pbscluster (2), deploys (2), hpc (2), like (2), skip (2), within (2), set (2), machine (2), records (2), value (2), map (2), lines (2), mybucket (2), read (2), similar (2), regard (2), spark (2), iterators (2), especially (2), text (2), simple (2), groupby (2), axis (2), 10000 (2), but (2), familiar (2), model (2), terabyte (2), sized (2), faster (2), machines (2), available (2), ram (2), parquet (2), gather (2), back (2), filename (2), load (2), several (2), tasks (2), guidelines (2), extend (2), debug (2), command (2), line (2), delayed (2), sum (2), std (2), skew (2), quantile (2), min (2), median (2), max (2), kurt (2), count (2), apply (2), ordered (2), known (2), codes (2), categories (2), year (2), weekofyear (2), weekday (2), week (2), timetz (2), quarter (2), nanosecond (2), minute (2), microsecond (2), is_year_start (2), is_year_end (2), is_quarter_start (2), is_quarter_end (2), is_month_start (2), is_month_end (2), is_leap_year (2), hour (2), freq (2), daysinmonth (2), days_in_month (2), dayofyear (2), dayofweek (2), day_of_year (2), day_of_week (2), day (2), date (2), scheduling (2), understanding (2), internals (2), diagnostics (2), task (2), high (2), collections (2), advanced (2), additional (2), information (2), internal (2), design (2), ctrl (2), copyright, 2026, developers, installation, next, was, designed, hope, able, help, have, additionally, encourage, look, through, website, related, most, closely, matches, application, youtube, channel, about, applications, following, sources, widely, across, all, industries, scales, anywhere, people, experience, pain, due, intense, 64gi, limits, 2gi, requests, resources, ghcr, latest, image, dask_kubernetes, deploying, jobs, regular, queue, 100gb, cores, dask_jobqueue, job, submission, systems, slurm, pbs, sge, lsf, torque, condor, others, spot_with_fallback, spot_policy, gib, worker_memory, east, region, n_workers, commercial, saas, product, platforms, aws, gcp, azure, alternatively, part, will, operate, thread, pool, contained, entirely, normal, itself, easily, session, object, which, sets, everything, hardware, complete, installing, to_dask_dataframe, lambda, filter, loads, read_text, rdds, vanilla, structures, prior, ingestion, into, good, preprocessing, streaming, way, minimizes, low, offering, reduce, functionality, lists, commonly, raw, objects, they, open_mfdataset, wraps, downstream, labeled, axes, simultaneously, tracking, many, together, resulting, analyses, accounts, majority, today, geospatial, imaging, communities, allow, scientists, researchers, perform, sophisticated, operations, plot, get, amount, read_parquet, apache, computer, append, filenames, happen, form, foundation, other, feedback, helpful, dashboards, responsive, scaling, techniques, thousand, per, tooling, construct, pipelines, workflows, flexible, arbitrary, loop, style, code, apis, choose, works, unlocking, complex, algorithms, just, pdf, rst, maintainer, talks, tutorials, gpus, selecting, backend, sizeof, connect, remote, configuration, changelog, development, interface, query, planning, expression, system, depth, shared, opportunistic, caching, ordering, stages, user, interfaces, phases, dashboard, visualize, debugging, level, graph, manipulation, optimization, specification, learning, working, hive, partitioning, categoricals, shuffling, join, sql, indexing, optimizer, save, compatibility, functions, number, generation, generalized, ufuncs, stack, concatenate, block, assignment, slicing, stats, sparse, overlapping, computations, chunks, faq, deployment, considerations, customize, initialization, prometheus, manage, environments, docker, images, adaptive, deployments, ssh, minutes, getting, started, search, top, community, ecosystem, main, content,

Text of the page (random words):
dask dask documentation skip to main content dask distributed dask ml examples ecosystem community back to top ctrl k dask documentation search ctrl k getting started install dask 10 minutes to dask deploy dask clusters python api cloud high performance computers kubernetes command line ssh additional information adaptive deployments docker images python api advanced manage environments prometheus customize initialization deployment considerations best practices faq how to use array best practices chunks create dask arrays overlapping computations internal design sparse arrays stats slicing assignment stack concatenate and block generalized ufuncs random number generation api compatibility with numpy functions bag create dask bags api dask dataframe dataframe to_bag dask dataframe series to_bag dataframe load and save data internal design optimizer best practices api dask dataframe series dt date dask dataframe series dt day dask dataframe series dt day_of_week dask dataframe series dt day_of_year dask dataframe series dt dayofweek dask dataframe series dt dayofyear dask dataframe series dt days_in_month dask dataframe series dt daysinmonth dask dataframe series dt freq dask dataframe series dt hour dask dataframe series dt is_leap_year dask dataframe series dt is_month_end dask dataframe series dt is_month_start dask dataframe series dt is_quarter_end dask dataframe series dt is_quarter_start dask dataframe series dt is_year_end dask dataframe series dt is_year_start dask dataframe series dt microsecond dask dataframe series dt minute dask dataframe series dt month dask dataframe series dt nanosecond dask dataframe series dt quarter dask dataframe series dt second dask dataframe series dt time dask dataframe series dt timetz dask dataframe series dt tz dask dataframe series dt week dask dataframe series dt weekday dask dataframe series dt weekofyear dask dataframe series dt year dask dataframe series cat categories dask dataframe series cat codes dask dataframe series cat known dask dataframe series cat ordered dask dataframe api rolling apply dask dataframe api rolling count dask dataframe api rolling kurt dask dataframe api rolling max dask dataframe api rolling mean dask dataframe api rolling median dask dataframe api rolling min dask dataframe api rolling quantile dask dataframe api rolling skew dask dataframe api rolling std dask dataframe api rolling sum dask dataframe api rolling var additional information parquet indexing sql join performance shuffling performance categoricals extend hive partitioning delayed working with collections best practices futures machine learning internals understanding performance scheduling task graphs specification custom graphs optimization advanced graph manipulation custom collections high level graphs debugging and performance debug visualize task graphs dashboard diagnostics local diagnostics distributed phases of computation dask internals user interfaces understanding performance stages of computation ordering opportunistic caching shared memory scheduling in depth query planning with expression system reference api reference array dataframe dask dataframe series dt date dask dataframe series dt day dask dataframe series dt day_of_week dask dataframe series dt day_of_year dask dataframe series dt dayofweek dask dataframe series dt dayofyear dask dataframe series dt days_in_month dask dataframe series dt daysinmonth dask dataframe series dt freq dask dataframe series dt hour dask dataframe series dt is_leap_year dask dataframe series dt is_month_end dask dataframe series dt is_month_start dask dataframe series dt is_quarter_end dask dataframe series dt is_quarter_start dask dataframe series dt is_year_end dask dataframe series dt is_year_start dask dataframe series dt microsecond dask dataframe series dt minute dask dataframe series dt month dask dataframe series dt nanosecond dask dataframe series dt quarter dask dataframe series dt second dask dataframe series dt time dask dataframe series dt timetz dask dataframe series dt tz dask dataframe series dt week dask dataframe series dt weekday dask dataframe series dt weekofyear dask dataframe series dt year dask dataframe series cat categories dask dataframe series cat codes dask dataframe series cat known dask dataframe series cat ordered dask dataframe api rolling apply dask dataframe api rolling count dask dataframe api rolling kurt dask dataframe api rolling max dask dataframe api rolling mean dask dataframe api rolling median dask dataframe api rolling min dask dataframe api rolling quantile dask dataframe api rolling skew dask dataframe api rolling std dask dataframe api rolling sum dask dataframe api rolling var bag dask dataframe dataframe to_bag dask dataframe series to_bag delayed futures command line interface development guidelines changelog configuration how to connect to remote data debug extend sizeof selecting the collection backend use gpus talks tutorials maintainer guidelines rst pdf dask contents how to use dask how to install dask how to deploy dask learn with examples dask dask is a python library for parallel and distributed computing dask is easy to use and set up it s just a python library powerful at providing scale and unlocking complex algorithms and fun how to use dask dask provides several apis choose one that works best for you tasks dask futures parallelize arbitrary for loop style python code providing flexible tooling allowing you to construct custom pipelines and workflows powerful scaling techniques processing several thousand tasks per second responsive feedback allowing for intuitive execution and helpful dashboards dask futures form the foundation for other dask work learn more at futures documentation or see an example at futures example from dask distributed import localcluster client localcluster get_client submit work to happen in parallel results for filename in filenames data client submit load filename result client submit process data results append result gather results back to local computer results client gather results dataframes dask dataframes parallelize the popular pandas library providing larger than memory execution for single machines allowing you to process data that is larger than your available ram parallel execution for faster processing distributed computation for terabyte sized datasets dask dataframes are similar in this regard to apache spark but use the familiar pandas api and memory model one dask dataframe is simply a collection of pandas dataframes on different computers learn more at dataframe documentation or see an example at dataframe example import dask dataframe as dd read large datasets in parallel df dd read_parquet s3 mybucket data parquet df df df value 0 result df groupby df name amount mean result result compute compute to get pandas result result plot arrays dask arrays parallelize the popular numpy library providing larger than memory execution for single machines allowing you to process data that is larger than your available ram parallel execution for faster processing distributed computation for terabyte sized datasets dask arrays allow scientists and researchers to perform intuitive and sophisticated operations on large datasets but use the familiar numpy api and memory model one dask array is simply a collection of numpy arrays on different computers learn more at array documentation or see an example at array example import dask array as da x da random random 10000 10000 y x x t x mean axis 1 z y var axis 0 compute xarray wraps dask array and is a popular downstream project providing labeled axes and simultaneously tracking many dask arrays together resulting in more intuitive analyses xarray is popular and accounts for the majority of dask array use today especially within geospatial and imaging communities learn more at xarray documentation or see an example at xarray example import xarray as xr ds xr open_mfdataset data nc da groupby time month mean time compute bags dask bags are simple parallel python lists commonly used to process text or raw python objects they are simple offering easy map and reduce functionality low memory processing data in a streaming way that minimizes memory use good for preprocessing especially for text or json data prior ingestion into dataframes dask bags are similar in this regard to spark rdds or vanilla python data structures and iterators one dask bag is simply a collection of python iterators processing in parallel on different computers learn more at bag documentation or see an example at bag example import dask bag as db read large datasets in parallel lines db read_text s3 mybucket data json records lines map json loads filter lambda d d value 0 df records to_dask_dataframe how to install dask installing dask is easy with pip or conda learn more at install documentation pip python m pip install dask complete conda conda install dask how to deploy dask you can use dask on a single machine or deploy it on distributed hardware learn more at deploy documentation local dask can set itself up easily in your python session if you create a localcluster object which sets everything up for you from dask distributed import localcluster cluster localcluster client cluster get_client normal dask work alternatively you can skip this part and dask will operate within a thread pool contained entirely with your local process cloud coiled is a commercial saas product that deploys dask clusters on cloud platforms like aws gcp and azure import coiled cluster coiled cluster n_workers 100 region us east 2 worker_memory 16 gib spot_policy spot_with_fallback client cluster get_client learn more at coiled documentation hpc the dask jobqueue project deploys dask clusters on popular hpc job submission systems like slurm pbs sge lsf torque condor and others from dask_jobqueue import pbscluster cluster pbscluster cores 24 memory 100gb queue regular account my account cluster scale jobs 100 client cluster get_client learn more at dask jobqueue documentation kubernetes the dask kubernetes project provides a dask kubernetes operator for deploying dask on kubernetes clusters from dask_kubernetes operator import kubecluster cluster kubecluster name my dask cluster image ghcr io dask dask latest resources requests memory 2gi limits memory 64gi cluster scale 10 client cluster get_client learn more at dask kubernetes documentation learn with examples dask is used widely across all industries and scales dask is used anywhere python is used and people experience pain due to large scale data or intense computing you can learn more about dask applications at the following sources dask examples dask youtube channel additionally we encourage you to look through the reference documentation on this website related to the api that most closely matches your application dask was designed to be easy to use and powerful we hope that it s able to help you have fun with your work next dask installation contents how to use dask how to install dask how to deploy dask learn with examples by dask developers copyright 2026