Meta tags:

Headings (most frequently used words):

groupby, dask, dataframes, client, aggregations, info, scheduler, examples, documentation, contents, start, for, dashboard, artifical, dataset, custom, many, groups, apply, cluster, localcluster, workers, worker,

Text of the page (most frequently used words):
dask (40), #groupby (32), the (29), linearregression (28), name (14), with (14), aggregations (14), dataframes (13), pandas (12), mean (11), client (11), and (10), total (10), this (10), time (9), you (9), for (9), dashboard (9), user (8), groups (8), data (7), examples (7), dataframe (7), 127 (7), object (6), will (6), apply (6), can (6), small (6), custom (6), dataset (6), from (5), train (5), compute (5), partition (5), learning (5), your (5), documentation (5), memory (5), status (5), alice (4), cpu (4), times (4), sys (4), wall (4), meta (4), output (4), fit (4), machine (4), like (4), more (4), aggregation (4), are (4), into (4), number (4), many (4), results (4), start (4), large (4), cluster (4), 2000 (4), datasets (4), http (4), notebook (4), read (3), bob (3), charlie (3), dan (3), edith (3), dtype (3), not (3), work (3), est (3), return (3), values (3), import (3), operations (3), generally (3), just (3), single (3), aggregate (3), same (3), std (3), float64 (3), timeseries (3), artifical (3), worker (3), tcp (3), gib (3), threads (3), workers (3), 8787 (3), scheduler (3), distributed (3), survey (3), workloads (3), developers (2), gotcha (2), write (2), frank (2), george (2), hannah (2), ingrid (2), jerry (2), kevin (2), laura (2), michael (2), norbert (2), oliver (2), patricia (2), quinn (2), ray (2), sarah (2), tim (2), ursula (2), victor (2), wendy (2), xavier (2), yvonne (2), zelda (2), scikit (2), learn (2), every (2), some (2), however (2), group (2), but (2), method (2), quite (2), around (2), too (2), penwidth (2), node_attr (2), visualize (2), split_out (2), computational (2), graph (2), these (2), result (2), may (2), usually (2), see (2), common (2), performance (2), scalable (2), 000263 (2), 000685 (2), 001401 (2), 000536 (2), 001977 (2), 000260 (2), count (2), which (2), persist (2), now (2), create (2), comm (2), info (2), true (2), using (2), processes (2), localcluster (2), when (2), side (2), both (2), useful (2), computation (2), use (2), live (2), contents (2), toggle (2), navigation (2), web (2), models (2), scale (2), arrays (2), copyright, 2018, next, previous, 676, 723, usr, share, miniconda3, envs, lib, python3, site, packages, core, 6335, futurewarning, valid, map_partitions, expects, try, passing, dict, tuple, representing, columns, future, passed, warnings, warn, sort_index, empty, def, sklearn, linear_model, following, example, simple, model, person, function, applied, requires, given, record, someone, named, force, great, deal, communication, expensive, still, possible, should, avoided, works, all, fast, because, they, broken, down, easily, well, known, doesn, have, move, much, pass, intermediate, across, network, four, outputs, larger, 1000000, 1000, sometimes, people, want, millions, cases, full, need, split, multiple, partitions, control, parameter, default, sum, their, good, choice, available, about, speaking, roughly, 100010, 001361, 577649, 99687, 002160, 577608, 99326, 001523, 577126, 99283, 000872, 577571, 99471, 577759, 355, agg, head, 452, 235, depend, key, 000048, 001241, 002285, 002062, 000779, 002219, 000472, 003100, 001520, 001986, 000934, 000233, 001600, 000426, 001102, 002923, 000821, 002330, 001805, 004697, fairly, efficient, assuming, that, less, than, million, implement, commonly, used, subset, api, would, skip, step, becomes, enough, make, tasks, int64, npartitions, structure, artificial, help, home, runner, space, 8cgbsa8t, local, directory, 36749, nanny, 44441, 34411, started, 37903, c95eb614, e548, 45f8, bbbb, 9867335584b4, running, 40f50cf8, type, connection, f66fcc00, 0de0, 11ed, a18c, 000d3a8f7959, 2gb, memory_limit, threads_per_worker, n_workers, link, become, visible, below, recommend, having, open, one, screen, while, other, take, effort, arrange, windows, seeing, them, very, starting, optional, provide, gain, insight, uses, discuss, best, practices, run, view, github, session, pdf, ipynb, theme, executable, book, project, 2019, 2020, 2021, surveys, series, forecasting, stencil, computations, numba, etl, pipelines, prefect, image, processing, handle, evolving, workflows, embarrassingly, parallel, asynchronous, servers, async, await, non, blocking, execution, analyze, hosted, json, applications, singular, value, decomposition, generalized, linear, automate, tpot, voting, classifiers, xgboost, hyperparameter, optimization, text, vectorization, pipeline, incrementally, batch, prediction, pytorch, score, predict, problems, blockwise, ensemble, methods, reading, messy, resilience, against, hardware, failures, xarray, operating, sql, futures, delayed, bags, basic, sidebar, community, ecosystem,

Text of the page (random words):
dataframes groupby dask examples documentation dask distributed dask ml examples ecosystem community toggle navigation sidebar dask examples documentation basic examples dask arrays dask bags dask dataframes custom workloads with dask delayed custom workloads with futures dask for machine learning operating on dask dataframes with sql xarray with dask arrays resilience against hardware failures dataframes dataframes read and write data dataframes groupby gotcha s from pandas to dask dataframes reading in messy data machine learning blockwise ensemble methods scale scikit learn for small data problems score and predict large datasets batch prediction with pytorch train models on large datasets incrementally train large datasets text vectorization pipeline hyperparameter optimization with dask scale xgboost use voting classifiers automate machine learning with tpot generalized linear models singular value decomposition applications analyze web hosted json data async await and non blocking execution asynchronous computation web servers dask embarrassingly parallel workloads handle evolving workflows image processing etl pipelines with prefect stencil computations with numba time series forecasting user surveys 2021 dask user survey results 2020 dask user survey results 2019 dask user survey results theme by the executable book project toggle navigation ipynb pdf contents start dask client for dashboard artifical dataset groupby aggregations custom aggregations many groups groupby apply dataframes groupby contents start dask client for dashboard artifical dataset groupby aggregations custom aggregations many groups groupby apply live notebook you can run this notebook in a live session or view it on github dataframes groupby this notebook uses the pandas groupby aggregate and groupby apply on scalable dask dataframes it will discuss both common use and best practices start dask client for dashboard starting the dask client is optional it will provide a dashboard which is useful to gain insight on the computation the link to the dashboard will become visible when you create the client below we recommend having it open on one side of your screen while using your notebook on the other side this can take some effort to arrange your windows but seeing them both at the same is very useful when learning 1 from dask distributed import client client client n_workers 1 threads_per_worker 4 processes true memory_limit 2gb client 1 client client f66fcc00 0de0 11ed a18c 000d3a8f7959 connection method cluster object cluster type distributed localcluster dashboard http 127 0 0 1 8787 status cluster info localcluster 40f50cf8 dashboard http 127 0 0 1 8787 status workers 1 total threads 4 total memory 1 86 gib status running using processes true scheduler info scheduler scheduler c95eb614 e548 45f8 bbbb 9867335584b4 comm tcp 127 0 0 1 37903 workers 1 dashboard http 127 0 0 1 8787 status total threads 4 started just now total memory 1 86 gib workers worker 0 comm tcp 127 0 0 1 34411 total threads 4 dashboard http 127 0 0 1 44441 status memory 1 86 gib nanny tcp 127 0 0 1 36749 local directory home runner work dask examples dask examples dataframes dask worker space worker 8cgbsa8t artifical dataset we create an artificial timeseries dataset to help us work with groupby operations 2 import dask df dask datasets timeseries df 2 dask dataframe structure id name x y npartitions 30 2000 01 01 int64 object float64 float64 2000 01 02 2000 01 30 2000 01 31 dask name make timeseries 30 tasks this dataset is small enough to fit in the cluster s memory so we persist it now you would skip this step if your dataset becomes too large to fit into memory 3 df df persist groupby aggregations dask dataframes implement a commonly used subset of the pandas groupby api see pandas groupby documentation we start with groupby aggregations these are generally fairly efficient assuming that the number of groups is small less than a million 4 df groupby name x mean compute 4 name alice 0 000260 bob 0 000536 charlie 0 001401 dan 0 000685 edith 0 000263 frank 0 000048 george 0 001241 hannah 0 002285 ingrid 0 002062 jerry 0 000779 kevin 0 002219 laura 0 000472 michael 0 003100 norbert 0 001520 oliver 0 001986 patricia 0 000934 quinn 0 000233 ray 0 001600 sarah 0 000426 tim 0 001102 ursula 0 002923 victor 0 000821 wendy 0 001977 xavier 0 002330 yvonne 0 001805 zelda 0 004697 name x dtype float64 performance will depend on the aggregation you do mean vs std the key on which you group name vs id and the number of total groups 5 time _ df groupby id x mean compute cpu times user 63 9 ms sys 3 83 ms total 67 7 ms wall time 235 ms 6 time _ df groupby name x mean compute cpu times user 87 4 ms sys 0 ns total 87 4 ms wall time 452 ms 7 time df groupby name agg x mean std y mean count compute head cpu times user 66 1 ms sys 4 25 ms total 70 4 ms wall time 355 ms 7 x y mean std mean count name alice 0 000260 0 577759 0 001977 99471 bob 0 000536 0 577571 0 000872 99283 charlie 0 001401 0 577126 0 001523 99326 dan 0 000685 0 577608 0 002160 99687 edith 0 000263 0 577649 0 001361 100010 this is the same as with pandas generally speaking dask dataframe groupby aggregations are roughly same performance as pandas groupby aggregations just more scalable you can read more about pandas common aggregations in the pandas documentation custom aggregations dask dataframe aggregate is available for custom aggregations see dask dataframe aggregate documentation many groups by default groupby aggregations like groupby mean or groupby sum return the result as a single partition dask dataframe their results are usually quite small so this is usually a good choice however sometimes people want to do groupby aggregations on many groups millions or more in these cases the full result may not fit into a single pandas dataframe output and you may need to split your output into multiple partitions you can control this with the split_out parameter 8 computational graph of a single output aggregation for a small number of groups like 1000 df groupby name x mean visualize node_attr penwidth 6 8 9 computational graph of an aggregation to four outputs for a larger number of groups like 1000000 df groupby id x mean split_out 4 visualize node_attr penwidth 6 9 groupby apply groupby aggregations are generally quite fast because they can be broken down easily into well known operations the data doesn t have to move around too much and we can just pass around small intermediate values across the network for some operations however the function to be applied requires all data from a given group like every record of someone named alice this will force a great deal of communication and be more expensive but is still possible with the groupby apply method this should be avoided if a groupby aggregation works in the following example we train a simple scikit learn machine learning model on every person s name 10 from sklearn linear_model import linearregression def train partition if partition empty return est linearregression est fit partition x id values partition y values return est 11 time df groupby name apply train meta object compute sort_index usr share miniconda3 envs dask examples lib python3 9 site packages dask dataframe core py 6335 futurewarning meta is not valid map_partitions expects output to be a pandas object try passing a pandas object as meta or a dict or tuple representing the name dtype of the columns in the future the meta you passed will not work warnings warn cpu times user 676 ms sys 46 7 ms total 723 ms wall time 4 25 s 11 name alice linearregression bob linearregression charlie linearregression dan linearregression edith linearregression frank linearregression george linearregression hannah linearregression ingrid linearregression jerry linearregression kevin linearregression laura linearregression michael linearregression norbert linearregression oliver linearregression patricia linearregression quinn linearregression ray linearregression sarah linearregression tim linearregression ursula linearregression victor linearregression wendy linearregression xavier linearregression yvonne linearregression zelda linearregression dtype object previous dataframes read and write data next gotcha s from pandas to dask by dask developers copyright 2018 dask developers