Meta tags:

Headings (most frequently used words):

the, dask, parallel, function, embarrassingly, workloads, client, define, to, info, scheduler, examples, documentation, contents, start, for, dashboard, your, computation, calling, set, of, input, parameters, call, use, delayed, make, our, lazy, run, in, using, futures, api, doing, some, analysis, on, results, handling, very, large, simulation, with, bags, cluster, localcluster, workers, worker,

Text of the page (most frequently used words):
dask (62), the (54), #results (26), for (26), this (25), function (23), parameters (23), #client (23), with (20), futures (19), our (18), computation (17), time (16), using (14), you (13), can (13), and (13), that (13), delayed (13), result (13), dashboard (12), run (12), costly_simulation (11), call (11), parallel (11), total (10), input_params (10), use (10), user (9), api (9), your (9), cluster (9), import (8), large (8), work (8), output (8), lazy_results (8), all (7), one (7), calling (7), compute (7), input (7), many (7), simulation (7), some (7), will (7), gather (7), status (7), want (7), embarrassingly (7), 127 (7), workloads (7), times (6), values (6), workers (6), lazy (6), random (6), define (6), examples (6), data (6), dataframes (6), here (5), only (5), cpu (5), sys (5), wall (5), list (5), bag (5), bags (5), could (5), very (5), just (5), 9497434903818136 (5), start (5), example (5), scale (5), when (5), make (5), 100 (4), tasks (4), map (4), above (4), scheduler (4), handling (4), also (4), then (4), pandas (4), get (4), are (4), doing (4), analysis (4), not (4), now (4), but (4), 245428056748925 (4), 1309710532899184 (4), 65913554017504 (4), 3695413820396394 (4), append (4), below (4), machine (4), lazy_result (4), these (4), python (4), parallelize (4), set (4), http (4), notebook (4), learning (4), web (3), see (3), 500 (3), each (3), more (3), into (3), which (3), already (3), methods (3), about (3), documentation (3), filtered_output (3), param_d (3), param_c (3), param_b (3), param_a (3), dataframe (3), have (3), from (3), need (3), object (3), watch (3), given (3), code (3), local (3), immediately (3), same (3), useful (3), different (3), than (3), simple (3), ways (3), worker (3), tcp (3), gib (3), memory (3), threads (3), 8787 (3), distributed (3), survey (3), datasets (3), developers (2), handle (2), evolving (2), workflows (2), asynchronous (2), servers (2), true (2), results_bag (2), looking (2), because (2), actually (2), partitions (2), fewer (2), 000 (2), apply (2), size (2), has (2), amount (2), task (2), batch (2), len (2), print (2), nice (2), locally (2), sample (2), index (2), series (2), view (2), there (2), write (2), page (2), later (2), trigger (2), wait (2), future (2), submit (2), notice (2), starts (2), cell (2), across (2), inputs (2), both (2), rather (2), being (2), ten (2), faster (2), add (2), running (2), computations (2), 5063310042417761 (2), 2444777543824386 (2), 531794781158632 (2), 3147033688719252 (2), 925532773375028 (2), started (2), normal (2), graph (2), hardware (2), even (2), like (2), good (2), concurrent (2), list_param (2), real (2), module (2), executable (2), applications (2), comm (2), info (2), localcluster (2), having (2), side (2), kind (2), often (2), cloud (2), live (2), contents (2), toggle (2), navigation (2), models (2), train (2), arrays (2), custom (2), copyright, 2018, next, previous, should, instead, taking, average, 926, 121, npartitions, from_sequence, hold, onto, sequences, few, convert, sequence, collection, asking, most, huge, every, item, well, trouble, schedule, solution, problem, bundle, single, either, making, new, operated, apis, described, avoiding, too, 244, tmp, simulation_result, csv, to_csv, 9844499213466507, mean, axessubplot, inline, plot, matplotlib, statistical, plots, save, interface, 647095, 195684, 864903, 228315, 358192, 322, 393723, 200821, 423060, 947608, 822234, 427, 827582, 631732, 309066, 814248, 072536, 342, 886125, 396112, 818088, 863210, 808715, 114, 060212, 822720, 366929, 472349, 398214, copy, expand, initial, interests, outside, simplicity, able, simulations, implement, complex, mechanism, individual, shared, file, system, store, encourage, going, shall, noted, performed, launching, won, any, retrieve, computed, right, lines, allowing, explicitly, until, done, process, implemented, itself, case, applying, equally, little, bit, completely, spreads, around, managing, load, balancing, dependencies, etc, ask, thread, although, still, working, practical, actual, additional, background, persist, was, sequentially, loop, 261, 304, may, during, contains, information, calls, yet, been, objects, ae2bb390, 5058, 4db9, a033, 6498fb822270, functions, almost, free, construct, funtion, its, records, what, relatively, straightforward, way, existing, base, isn, libraries, others, first, steps, second, step, especially, machines, joblib, multiprocessing, note, clever, easily, 123, 138, let, would, quite, long, otherwise, without, loops, 420719, 514555, 303275, 130992, 190979, 225035, 253090, 990032, 622723, 135610, 789435, 583202, 263298, 554442, 471111, 956577, 766664, 799919, 071036, 312124, head, columns, numpy, generate, lets, say, four, called, param_, 146, try, sum, return, sleep, def, does, operation, numbers, array, together, sleeps, simulate, cases, another, subprocess, home, runner, space, ii4pk6hj, directory, 46881, nanny, 45931, 33017, 46295, 859f8895, 4dc5, 45e4, 945c, 593b2390b28a, processes, 6c9279a2, type, connection, method, 6c01af35, 0de0, 11ed, 9e28, 000d3a8f7959, n_workers, threads_per_worker, progress, link, become, visible, create, recommend, open, screen, while, other, take, effort, arrange, windows, seeing, them, starting, provide, gain, insight, part, moreover, launched, super, computer, probably, end, connect, easy, achieve, respectively, hpc, big, infrastructure, yarn, kubernetes, jobqueue, place, further, independent, runs, heavy, focuses, building, seen, scientific, communities, high, performance, computing, facilities, monte, carlo, assume, following, shows, how, where, pieces, independently, show, three, github, session, pdf, ipynb, theme, book, project, 2019, 2020, 2021, surveys, forecasting, stencil, numba, etl, pipelines, prefect, image, processing, async, await, non, blocking, execution, analyze, hosted, json, singular, value, decomposition, generalized, linear, automate, tpot, voting, classifiers, xgboost, hyperparameter, optimization, text, vectorization, pipeline, incrementally, prediction, pytorch, score, predict, scikit, learn, small, problems, blockwise, ensemble, reading, messy, gotcha, groupby, read, resilience, against, failures, xarray, operating, sql, basic, sidebar, community, ecosystem,

Text of the page (random words):
embarrassingly parallel workloads dask examples documentation dask distributed dask ml examples ecosystem community toggle navigation sidebar dask examples documentation basic examples dask arrays dask bags dask dataframes custom workloads with dask delayed custom workloads with futures dask for machine learning operating on dask dataframes with sql xarray with dask arrays resilience against hardware failures dataframes dataframes read and write data dataframes groupby gotcha s from pandas to dask dataframes reading in messy data machine learning blockwise ensemble methods scale scikit learn for small data problems score and predict large datasets batch prediction with pytorch train models on large datasets incrementally train large datasets text vectorization pipeline hyperparameter optimization with dask scale xgboost use voting classifiers automate machine learning with tpot generalized linear models singular value decomposition applications analyze web hosted json data async await and non blocking execution asynchronous computation web servers dask embarrassingly parallel workloads handle evolving workflows image processing etl pipelines with prefect stencil computations with numba time series forecasting user surveys 2021 dask user survey results 2020 dask user survey results 2019 dask user survey results theme by the executable book project toggle navigation ipynb pdf contents start dask client for dashboard define your computation calling function define the set of input parameters to call the function use dask delayed to make our function lazy run in parallel using the futures api doing some analysis on the results handling very large simulation with bags embarrassingly parallel workloads contents start dask client for dashboard define your computation calling function define the set of input parameters to call the function use dask delayed to make our function lazy run in parallel using the futures api doing some analysis on the results handling very large simulation with bags live notebook you can run this notebook in a live session or view it on github embarrassingly parallel workloads this notebook shows how to use dask to parallelize embarrassingly parallel workloads where you want to apply one function to many pieces of data independently it will show three different ways of doing this with dask dask delayed concurrent futures dask bag this example focuses on using dask for building large embarrassingly parallel computation as often seen in scientific communities and on high performance computing facilities for example with monte carlo methods this kind of simulation assume the following we have a function that runs a heavy computation given some parameters we need to compute this function on many different input parameters each function call being independent we want to gather all the results in one place for further analysis start dask client for dashboard starting the dask client will provide a dashboard which is useful to gain insight on the computation we will also need it for the futures api part of this example moreover as this kind of computation is often launched on super computer or in the cloud you will probably end up having to start a cluster and connect a client to scale see dask jobqueue dask kubernetes or dask yarn for easy ways to achieve this on respectively an hpc cloud or big data infrastructure the link to the dashboard will become visible when you create the client below we recommend having it open on one side of your screen while using your notebook on the other side this can take some effort to arrange your windows but seeing them both at the same time is very useful when learning 1 from dask distributed import client progress client client threads_per_worker 4 n_workers 1 client 1 client client 6c01af35 0de0 11ed 9e28 000d3a8f7959 connection method cluster object cluster type distributed localcluster dashboard http 127 0 0 1 8787 status cluster info localcluster 6c9279a2 dashboard http 127 0 0 1 8787 status workers 1 total threads 4 total memory 6 78 gib status running using processes true scheduler info scheduler scheduler 859f8895 4dc5 45e4 945c 593b2390b28a comm tcp 127 0 0 1 46295 workers 1 dashboard http 127 0 0 1 8787 status total threads 4 started just now total memory 6 78 gib workers worker 0 comm tcp 127 0 0 1 33017 total threads 4 dashboard http 127 0 0 1 45931 status memory 6 78 gib nanny tcp 127 0 0 1 46881 local directory home runner work dask examples dask examples applications dask worker space worker ii4pk6hj define your computation calling function this function does a simple operation add all numbers of a list array together but it also sleeps for a random amount of time to simulate real work in real use cases this could call another python module or even run an executable using subprocess module 2 import time import random def costly_simulation list_param time sleep random random return sum list_param we try it locally below 3 time costly_simulation 1 2 3 4 cpu times user 2 24 ms sys 1 62 ms total 3 85 ms wall time 146 ms 3 10 define the set of input parameters to call the function we will generate a set of inputs on which we want to run our simulation function here we use pandas dataframe but we could also use a simple list lets say that our simulation is run with four parameters called param_ a d 4 import pandas as pd import numpy as np input_params pd dataframe np random random size 500 4 columns param_a param_b param_c param_d input_params head 4 param_a param_b param_c param_d 0 0 312124 0 071036 0 799919 0 766664 1 0 956577 0 471111 0 554442 0 263298 2 0 583202 0 789435 0 135610 0 622723 3 0 990032 0 253090 0 225035 0 190979 4 0 130992 0 303275 0 514555 0 420719 without using dask we could call our simulation on all of these parameters using normal python for loops let s only do this on a sample of our parameters as it would be quite long otherwise 5 results 6 time for parameters in input_params values 10 result costly_simulation parameters results append result cpu times user 123 ms sys 15 1 ms total 138 ms wall time 4 25 s 7 results 7 1 9497434903818136 2 245428056748925 2 1309710532899184 1 65913554017504 1 3695413820396394 1 5063310042417761 1 2444777543824386 1 531794781158632 1 3147033688719252 2 925532773375028 note that this is not very clever as we can easily parallelize code there are many ways to parallelize this function in python with libraries like multiprocessing concurrent futures joblib or others these are good first steps dask is a good second step especially when you want to scale across many machines use dask delayed to make our function lazy we can call dask delayed on our funtion to make it lazy rather than compute its result immediately it records what we want to compute as a task into a graph that we ll run later on parallel hardware using dask delayed is a relatively straightforward way to parallelize an existing code base even if the computation isn t embarrassingly parallel like this one calling these lazy functions is now almost free in the cell below we only construct a simple graph 8 import dask lazy_results 9 time for parameters in input_params values 10 lazy_result dask delayed costly_simulation parameters lazy_results append lazy_result cpu times user 1 43 ms sys 0 ns total 1 43 ms wall time 1 08 ms 10 lazy_results 0 10 delayed costly_simulation ae2bb390 5058 4db9 a033 6498fb822270 run in parallel the lazy_results list contains information about ten calls to costly_simulation that have not yet been run call compute when you want your result as normal python objects if you started client above then you may want to watch the status page during computation 11 time dask compute lazy_results cpu times user 261 ms sys 43 6 ms total 304 ms wall time 1 44 s 11 1 9497434903818136 2 245428056748925 2 1309710532899184 1 65913554017504 1 3695413820396394 1 5063310042417761 1 2444777543824386 1 531794781158632 1 3147033688719252 2 925532773375028 notice that this was faster than running these same computations sequentially with a for loop we can now run this on all of our input parameters 12 import dask lazy_results for parameters in input_params values lazy_result dask delayed costly_simulation parameters lazy_results append lazy_result futures dask persist lazy_results trigger computation in the background to make this go faster we can add additional workers although we re still only working on our local machine this is more practical when using an actual cluster 13 client cluster scale 10 ask for ten 4 thread workers by looking at the dask dashboard we can see that dask spreads this work around our cluster managing load balancing dependencies etc then get the result 14 results dask compute futures results 5 14 1 9497434903818136 2 245428056748925 2 1309710532899184 1 65913554017504 1 3695413820396394 using the futures api the same example can be implemented using dask s futures api by using the client object itself for our use case of applying a function across many inputs both dask delayed and dask futures are equally useful the futures api is a little bit different because it starts work immediately rather than being completely lazy for example notice that work starts immediately in the cell below as we submit work to the cluster 15 futures for parameters in input_params values future client submit costly_simulation parameters futures append future we can explicitly wait until this work is done and gather the results to our local process by calling client gather 16 results client gather futures results 5 16 1 9497434903818136 2 245428056748925 2 1309710532899184 1 65913554017504 1 3695413820396394 but the code above can be run in fewer lines with client map function allowing to call a given function on a list of parameters as for delayed we can only start the computation and not wait for results by not calling client gather right now it shall be noted that as dask cluster has already performed tasks launching costly_simulation with futures api on the given input parameters the call to client map won t actually trigger any computation and just retrieve already computed results 17 futures client map costly_simulation input_params values then just get the results later 18 results client gather futures len results 18 500 19 print results 0 1 9497434903818136 we encourage you to watch the dashboard s status page to watch on going computation doing some analysis on the results one of the interests of dask here outside from api simplicity is that you are able to gather the result for all your simulations in one call there is no need to implement a complex mechanism or to write individual results in a shared file system or object store just get your result and do some computation here we will just get the results and expand our initial dataframe to have a nice view of parameters vs results for our computation 20 output input_params copy output result pd series results index output index output sample 5 20 param_a param_b param_c param_d result 94 0 398214 0 472349 0 366929 0 822720 2 060212 114 0 808715 0 863210 0 818088 0 396112 2 886125 342 0 072536 0 814248 0 309066 0 631732 1 827582 427 0 822234 0 947608 0 423060 0 200821 2 393723 322 0 358192 0 228315 0 864903 0 195684 1 647095 then we can do some nice statistical plots or save result locally with pandas interface here 21 matplotlib inline output result plot 21 axessubplot 22 output result mean 22 1 9844499213466507 23 filtered_output output output result 2 print len filtered_output filtered_output to_csv tmp simulation_result csv 244 handling very large simulation with bags the methods above work well for a size of input parameters up to about 100 000 above that the dask scheduler has trouble handling the amount of tasks to schedule to workers the solution to this problem is to bundle many parameters into a single task you could do this either by making a new function that operated on a batch of parameters and using the delayed or futures apis on that function you could also use the dask bag api this is described more in the documentation about avoiding too many tasks dask bags hold onto large sequences in a few partitions we can convert our input_params sequence into a dask bag collection asking for fewer partitions so at most 100 000 which is already huge and apply our function on every item of the bag 24 import dask bag as db b db from_sequence list input_params values npartitions 100 b b map costly_simulation 25 time results_bag b compute cpu times user 926 ms sys 121 ms total 1 05 s wall time 7 79 s looking on dashboard here you should see only 100 tasks to run instead of 500 each taking 5x more time in average because each one is actually calling our function 5 times 26 np all results np all results_bag 26 true previous asynchronous computation web servers dask next handle evolving workflows by dask developers copyright 2018 dask developers