Meta tags:

Headings (most frequently used words):

operator, dask, kubernetes, what, the, contents, quickstart, is, resources, does, manage, worker, groups, clusters, jobs, autoscalers,

Text of the page (most frequently used words):
dask (42), the (40), kubernetes (34), operator (22), #resources (18), and (18), cluster (14), pod (13), #worker (12), resource (12), daskworkergroup (12), with (11), for (11), class (11), annotations (10), daskcluster (10), clusters (9), scheduler (9), that (9), daskjob (8), dashed (8), groups (7), kubecluster (7), will (7), can (7), kubectl (7), custom (7), what (6), workers (6), daskautoscaler (6), passed (6), kopf (6), documentation (6), you (5), service (5), install (5), manage (4), from (4), org (4), overview (4), creates (4), some (4), schedulerservice (4), ctrl (4), autoscalers (3), jobs (3), does (3), quickstart (3), name (3), job (3), all (3), also (3), set (3), created (3), starting (3), zalando (3), not (3), pods (3), https (3), perform (3), tasks (3), creating (3), different (3), group (3), schedulerpod (3), workerpoda (3), workerpodb (3), workerpodc (3), jobpod (3), stroke (3), api (3), your (3), deploying (3), search (3), developers (2), contents (2), installing (2), ghcr (2), latest (2), image (2), import (2), dask_kubernetes (2), scale (2), default (2), once (2), runs (2), like (2), distributed (2), runner (2), one (2), related (2), these (2), docs (2), concepts (2), working (2), objects (2), work (2), have (2), memory (2), intensive (2), onto (2), additional (2), are (2), settings (2), scaled (2), connect (2), via (2), order (2), which (2), native (2), manages (2), classdef (2), such (2), services (2), python (2), controller (2), create (2), done (2), helm (2), options (2), classic (2), 2025 (2), dev7 (2), g0b02f6317 (2), copyright, 2021, next, maximum, minimum, adapt, communicate, periodically, auto, desired, number, completion, removed, automatically, save, this, great, workflows, training, machine, learning, model, wants, spec, metadata, batch, style, specific, task, start, finish, alongside, leveraged, example, may, wish, smaller, pool, more, gpus, compute, well, has, support, extra, configuration, separately, then, use, schedule, exposed, user, clients, turn, represents, homogenous, similar, intelligence, around, lifecycle, must, attached, function, deployment, graph, fda061, width, 4px, dasharray, hierarchy, represent, primitives, run, processes, facilitate, communication, need, apply, definitions, allow, describe, itself, small, application, watches, events, our, other, accordingly, allows, either, pip, repo, namespace, generate, crds, needs, deploys, using, designed, dynamically, launch, hoc, deployments, package, provides, many, see, looking, high, level, new, users, should, head, page, note, welcome, dark, light, system, pdf, rst, history, releasing, testing, developer, migrating, troubleshooting, extending, advanced, getting, started, back, top, community, ecosystem, examples, skip, main, content,

Text of the page (random words):
dask kubernetes operator dask kubernetes 2025 7 1 dev7 g0b02f6317 documentation skip to main content dask distributed dask ml examples ecosystem community back to top ctrl k search ctrl k search ctrl k dask kubernetes 2025 7 1 dev7 g0b02f6317 documentation search ctrl k getting started overview installing operator kubecluster custom resources extending advanced troubleshooting classic migrating from classic developer testing releasing history rst pdf system settings light dark dask kubernetes operator contents quickstart what is the operator what resources does the operator manage worker groups clusters jobs autoscalers dask kubernetes operator welcome to the documentation for the dask kubernetes operator note if you are looking for high level documentation on deploying dask on kubernetes new users should head to the dask documentation page on kubernetes the package dask kubernetes provides a dask operator for kubernetes dask kubernetes is one of many options for deploying dask clusters see deploying dask in the dask documentation for an overview of additional options quickstart kubecluster deploys dask clusters on kubernetes clusters using custom kubernetes resources it is designed to dynamically launch ad hoc deployments install operator crds and controller needs to be done once on your kubernetes cluster helm install repo https helm dask org create namespace n dask operator generate name dask kubernetes operator install dask kubernetes pip install dask kubernetes from dask_kubernetes operator import kubecluster cluster kubecluster name my dask cluster image ghcr io dask dask latest cluster scale 10 what is the operator the dask operator is a set of custom resources and a controller that runs on your kubernetes cluster and allows you to create and manage your dask clusters as kubernetes resources creating clusters can either be done via the kubernetes api with kubectl or the python api with kubecluster to install the operator you need to apply some custom resource definitions that allow us to describe dask resources and the operator itself which is a small python application that watches the kubernetes api for events related to our custom resources and creates other resources such as pods and services accordingly what resources does the operator manage the operator manages a hierarchy of resources some custom resources to represent dask primitives like clusters and worker groups and native kubernetes resources such as pods and services to run the cluster processes and facilitate communication graph td daskjob daskjob daskcluster daskcluster daskautoscaler daskautoscaler schedulerservice scheduler service schedulerpod scheduler pod daskworkergroup daskworkergroup workerpoda worker pod a workerpodb worker pod b workerpodc worker pod c jobpod job runner pod daskjob daskcluster daskjob jobpod daskcluster schedulerservice daskcluster daskautoscaler schedulerservice schedulerpod daskcluster daskworkergroup daskworkergroup workerpoda daskworkergroup workerpodb daskworkergroup workerpodc classdef dask stroke fda061 stroke width 4px classdef dashed stroke dasharray 5 5 class daskjob dask class daskcluster dask class daskworkergroup dask class daskautoscaler dask class daskautoscaler dashed class schedulerservice dashed class schedulerpod dashed class workerpoda dashed class workerpodb dashed class workerpodc dashed class jobpod dashed worker groups a daskworkergroup represents a homogenous group of workers that can be scaled the resource is similar to a native kubernetes deployment in that it manages a group of workers with some intelligence around the pod lifecycle a worker group must be attached to a dask cluster resource in order to function all kubernetes annotations on the daskworkergroup resource will be passed onto worker pod resources annotations created by kopf or kubectl i e starting with kopf zalando org or kubectl kubernetes io will not be passed on clusters the daskcluster custom resource creates a dask cluster by creating a scheduler pod scheduler service and default daskworkergroup which in turn creates worker pod resources workers connect to the scheduler via the scheduler service and that service can also be exposed to the user in order to connect clients and perform work the operator also has support for creating additional worker groups these are extra groups of workers with different configuration settings and can be scaled separately you can then use resource annotations to schedule different tasks to different groups all kubernetes annotations https kubernetes io docs concepts overview working with objects annotations on the daskcluster resource will be passed onto the scheduler pod and service as well the daskworkergroup resources annotations created by kopf or kubectl i e starting with kopf zalando org or kubectl kubernetes io will not be passed on for example you may wish to have a smaller pool of workers that have more memory for memory intensive tasks or gpus for compute intensive tasks jobs a daskjob is a batch style resource that creates a pod to perform some specific task from start to finish alongside a daskcluster that can be leveraged to perform the work all kubernetes annotations https kubernetes io docs concepts overview working with objects annotations on the daskjob resource will be passed on to the job runner pod resource if one also wants to set kubernetes annotations on the cluster related resources scheduler and worker pods these can be set as spec cluster metadata in the daskjob resource annotations created by kopf or kubectl i e starting with kopf zalando org or kubectl kubernetes io will not be passed on once the job pod runs to completion the cluster is removed automatically to save resources this is great for workflows like training a distributed machine learning model with dask autoscalers a daskautoscaler resource will communicate with the scheduler periodically and auto scale the default daskworkergroup to the desired number of workers from dask_kubernetes operator import kubecluster cluster kubecluster name my dask cluster image ghcr io dask dask latest cluster adapt minimum 1 maximum 10 next installing contents quickstart what is the operator what resources does the operator manage worker groups clusters jobs autoscalers by dask kubernetes developers copyright 2021 dask kubernetes developers