Meta tags:
description=The Argo CD app for Storm had been red for two days, but the symptom that actually sent me down the... Tagged with devops, kubernetes, argocd.;
keywords=devops, kubernetes, argocd, software, coding, development, engineering, inclusive, community;

Headings (most frequently used words):

the, what, no, storm, we, that, and, official, helm, chart, how, put, apache, on, eks, anyway, dev, community, why, there, was, paved, road, cascade, wasn, obvious, at, first, argo, fighting, statefulsets, alb, couldn, find, port, validated, along, way, do, differently, tell, next, team, top, comments, more, from, mridul, tiwari,

Text of the page (most frequently used words):
the (90), and (51), storm (32), not (25), for (21), that (19), nimbus (19), was (19), chart (18), with (16), had (14), dev (13), cluster (13), prod (12), argo (12), one (10), alb (10), first (10), before (10), were (10), kubernetes (9), did (9), other (8), eks (8), app (8), you (8), supervisor (8), non (8), private (8), when (7), from (7), still (7), helm (7), zookeeper (7), shared (7), statefulset (7), images (7), same (7), share (6), community (6), never (6), this (6), init (6), containers (6), they (6), gitops (6), ingress (6), than (6), deployment (6), storage (6), karpenter (6), our (6), open (5), about (5), devops (5), every (5), are (5), will (5), but (5), official (5), entrypoint (5), three (5), port (5), target (5), two (5), upstream (5), docker (5), storageclass (5), what (5), failure (5), apache (5), statefulsets (5), pvcs (5), pattern (5), number (5), pods (5), phase (5), create (4), log (4), your (4), like (4), nodes (4), more (4), via (4), submit (4), templates (4), inbound (4), after (4), separate (4), already (4), logs (4), down (4), risk (4), instead (4), ports (4), named (4), mode (4), type (4), full (4), args (4), run (4), pvc (4), gp3 (4), does (4), how (4), could (4), logviewer (4), emptydir (4), rollout (4), everything (4), chain (4), account (3), made (3), software (3), use (3), organization (3), aws (3), ebs (3), workers (3), mridul (3), tiwari (3), being (3), abuse (3), hide (3), comments (3), there (3), image (3), waiting (3), check (3), cutover (3), local (3), based (3), apps (3), control (3), plane (3), workloads (3), validated (3), failures (3), sync (3), instance (3), emit (3), schema (3), week (3), csi (3), pending (3), been (3), four (3), bound (3), sat (3), yet (3), without (3), application (3), ecr (3), single (3), only (3), problem (3), actually (3), sidecar (3), own (3), class (3), looked (3), copy (3), grow (2), love (2), built (2), source (2), code (2), conduct (2), keep (2), kafka (2), disk (2), standing (2), terraform (2), work (2), also (2), new (2), comment (2), post (2), report (2), let (2), migration (2), guide (2), budget (2), time (2), archaeology (2), whether (2), start (2), layers (2), itself (2), has (2), quorum (2), topology (2), durable (2), shipping (2), legacy (2), migrating (2), all (2), tested (2), those (2), blocked (2), next (2), platform (2), ec2 (2), fleet (2), security (2), phased (2), prove (2), surprise (2), org (2), reference (2), means (2), arm (2), fear (2), committed (2), prevent (2), entrypoints (2), numeric (2), backend (2), empty (2), targetgroup (2), debugging (2), lbc (2), clusterip (2), hit (2), vct (2), drift (2), ignoredifferences (2), volumeclaimtemplates (2), exact (2), pass (2), examples (2), container (2), tree (2), silently (2), gp2 (2), argocd (2), team (2), showed (2), healthy (2), running (2), custom (2), corrected (2), behind (2), rest (2), tbd (2), topologies (2), worker (2), deployments (2), traditional (2), out (2), purpose (2), pod (2), fine (2), scary (2), arm64 (2), arch (2), have (2), done (2), worried (2), broke (2), way (2), cidr (2), load (2), balancer (2), just (2), added (2), rather (2), consumer (2), service (2), need (2), felt (2), outofsync (2), fixed (2), kind (2), fix (2), line (2), per (2), copied (2), which (2), hub (2), supervisors (2), used (2), finally (2), fullscreen (2), embedded (2), scream (2), ready (2), 6627 (2), exist (2), forever (2), right (2), wasn (2), reasonable (2), environment (2), architecture (2), any (2), choice (2), over (2), put (2), anyway (2), link (2), search (2), place, where, coders, stay, date, their, careers, 2016, 2026, ruby, rails, powers, inclusive, communities, forem, terms, privacy, policy, mlh, shop, free, postgres, database, contact, showcase, accounts, advertise, help, education, tracks, videos, challenges, home, space, discuss, development, manage, career, hits, 100, volume, won, greenfield, mistake, stepped, tagged, servers, prometheus, started, scraping, telegraf, feb, 2022, joined, engineer, paytm, india, location, coding, cheerful, dramatic, artist, contribution, follow, further, actions, may, consider, blocking, person, reporting, confirm, child, well, sure, want, become, hidden, visible, permalink, dismiss, preview, quickly, answer, faqs, store, snippets, template, trusted, user, personal, subscribe, top, diy, authorship, say, ever, chance, bug, might, below, questions, node, pipeline, horizontal, scale, slot, planning, adopt, dns, fully, none, stand, chapter, went, forward, good, reasons, direction, maintaining, alongside, doubles, operational, surface, patching, amis, groups, deploy, pipelines, bounded, blast, radius, reused, muscle, alloy, jenkins, delivery, someone, better, eyes, later, bigger, takeaway, assumption, models, infrastructure, actual, writing, inventory, treating, them, mysteries, knew, quirks, each, fired, recognized, spiraling, plus, gave, day, configure, upfront, especially, assume, behavior, cascades, through, exists, starting, again, pre, flight, checklist, forty, eight, hours, differently, tell, end, chains, closed, managed, ran, once, lifecycle, runs, jar, scaling, upgrades, interact, versus, hosts, haven, answered, kept, scope, fail, fast, dragging, into, live, restart, lost, exploration, noted, moved, negotiable, pool, amd64, seen, scheduling, runtime, crashes, graviton, verified, calling, stateful, nervous, expect, stable, identity, ephemeral, design, accepted, replica, question, deserved, explicit, along, separately, access, allowlists, group, affects, parallel, config, file, ingresses, coexist, during, testing, replacing, model, controller, weirdness, exposing, wanted, returned, effectively, combination, wired, backends, switched, came, 8080, familiar, under, got, perpetual, injects, expected, even, known, footgun, ways, volumeclaimtemplate, paths, immutable, fields, reconcile, away, respectignoredifferences, status, volumemode, apiversion, fighting, couldn, find, change, cost, bare, works, execs, argument, command, form, docs, apply, should, startup, wiring, exit, enter, exec, found, crashed, missing, land, then, immediately, louder, lesson, wrong, isn, sends, chasing, going, volumes, scheduled, listening, downstream, waited, health, marked, degraded, because, became, outside, requested, driver, flagged, feared, silent, things, break, loudly, propagated, cascade, obvious, completely, untested, hostname, data, datalog, component, paper, namespace, provisioned, tradeoff, prune, would, remove, avoided, another, planned, bind, stateless, five, decision, equal, measure, anywhere, region, putting, internal, runbook, networking, layout, experimental, wrote, afraid, list, turned, accurate, initial, plan, ship, maintained, project, publish, credible, migrate, options, research, bitnami, subchart, people, point, nothing, treat, supported, product, evaluated, thin, overlays, repo, values, opaque, dependency, gresearch, why, paved, road, part, sense, strategically, operationally, blank, page, trying, something, lift, shift, existing, lived, asg, elsewhere, ask, move, onto, whole, stack, streaming, sitting, else, pet, red, days, symptom, sent, rabbit, hole, quieter, crash, loop, stuck, printing, almost, deployed, originally, published, mriduliti, hashnode, posted, sep, mastodon, facebook, linkedin, clipboard, pick, gem, boost, save, jump, fire, raised, hands, exploding, head, unicorn, add, reaction, close, powered, algolia, navigation, menu, skip, content,

Text of the page (random words):
no official storm helm chart how we put apache storm on eks anyway dev community skip to content navigation menu search powered by algolia search log in create account dev community close add reaction like unicorn exploding head raised hands fire jump to comments save boost pick as gem more copy link copy link copied to clipboard share to x share to linkedin share to facebook share to mastodon share post via report abuse mridul tiwari posted on sep 19 originally published at mriduliti hashnode dev no official storm helm chart how we put apache storm on eks anyway devops kubernetes argocd the argo cd app for storm had been red for two days but the symptom that actually sent me down the rabbit hole was quieter than a crash loop init containers on the supervisor pods stuck forever printing the same line over and over waiting for nimbus 6627 no nimbus no workers no ui just a cluster that looked almost deployed and wasn t we were trying to do something the organization had never done before run apache storm on kubernetes not a lift and shift of an existing pattern there was no pattern storm still lived on traditional ec2 asg elsewhere the ask was to move it onto the same non prod eks cluster we were already standing up for the other application platform terraform karpenter argo gitops the whole stack streaming workloads sitting with everything else instead of a separate pet fleet that part made sense strategically operationally it was a blank page why there was no paved road apache storm does not ship a maintained official helm chart the project does not publish a credible migrate storm to kubernetes guide community options exist g research s gresearch storm chart with a bitnami zookeeper subchart is the one people point at but nothing we could treat as a supported product we evaluated it and still built a thin local chart instead same as other custom overlays in the repo own the templates own the values no opaque upstream dependency that decision felt right and scary in equal measure we were the first team anywhere in the org any account any region putting storm on kubernetes no internal reference architecture no runbook no copy what prod did every choice about storage networking image choice and gitops layout was experimental before we committed we wrote down what we were afraid of that list turned out to be more accurate than our initial rollout plan we planned a phased rollout on purpose phase one zookeeper and nimbus only prove the control plane and that pvcs actually bind phase two supervisors and storm ui as stateless deployments phase three a logviewer sidecar on supervisor pods not a separate deployment phase four private ecr images instead of docker hub upstream phase five storm ui on the shared private alb ingress pattern the rest of other already used one shared storm namespace on the non prod cluster not one cluster per environment karpenter provisioned arm64 nodes a storageclass for gp3 embedded in the storm chart itself rather than a separate argo application tradeoff being that chart prune would remove the class but we avoided yet another gitops app for a single consumer the architecture looked reasonable on paper component kind storage zookeeper statefulset pvc data datalog nimbus statefulset pvc supervisor deployment emptydir pod ip as storm local hostname storm ui deployment emptydir logviewer sidecar on supervisor shared emptydir logs reasonable and completely untested in our environment the cascade that wasn t obvious at first when things broke they did not break loudly failure chain number one was storage and it propagated silently our chart requested a storageclass named gp3 the ebs csi driver on the cluster was healthy the storageclass did not exist the cluster only had legacy in tree gp2 we had flagged this exact fear before rollout we feared silent pvc pending forever and we were right pvcs for zookeeper and nimbus sat in pending without bound volumes those statefulset pods never scheduled without nimbus listening on 6627 everything downstream waited supervisor and ui init containers blocked on the nimbus health check argo marked the app degraded the statefulsets showed outofsync because pods never became ready from the outside it looked like a storm problem it was a storage class problem three layers down that was the first lesson in how statefulset failures hide init containers do not scream your storageclass is wrong they scream upstream isn t ready yet which sends you chasing nimbus when nimbus was never going to start we fixed the missing class embedded gp3 via ebs csi in the chart and the control plane could finally land then failure chain number two hit immediately and this one was louder nimbus crashed with docker entrypoint sh exec nimbus not found enter fullscreen mode exit fullscreen mode storm ui did the same for ui we had copied helm examples that pass bare args like nimbus or ui which works for upstream docker hub images our private ecr images use a docker entrypoint sh that execs the first argument as a command they need storm nimbus and storm ui supervisors and the logviewer sidecar already used the storm form nimbus and ui did not official image docs do not apply when you own the entrypoint we should have tested container startup before wiring the full chart not after pvcs finally bound fix was a one line args change per deployment the archaeology cost more time than the fix argo fighting statefulsets and the alb that couldn t find a port with pods actually running we got failure chain number three perpetual argo sync drift on statefulset volumeclaimtemplates kubernetes injects apiversion kind volumemode and status after create our chart templates did not emit the full schema argo expected so the app sat outofsync even when the cluster was fine this is a known footgun with gitops and statefulsets we fixed it two ways corrected the chart templates to emit the full volumeclaimtemplate schema and added ignoredifferences jq paths with respectignoredifferences for the immutable vct fields argo will never reconcile away that one felt familiar if you ve run statefulsets under argo before the ingress problem did not failure chain number four was aws load balancer controller weirdness exposing storm ui we wanted the same private alb pattern as other other apps lbc returned effectively targetgroup port is empty we had instance target type with a clusterip service a combination that does not work the way we had it wired ingress backends need numeric ports with target type ip not named ports with instance mode we switched to target type ip and backend port 8080 as a number not a named service port storm ui came up behind the shared alb separately we were migrating inbound access on that alb from cidr allowlists to security group based inbound that affects every app on the load balancer not just storm we added a parallel sg based alb config file so cidr and sg ingresses could coexist during cutover testing rather than replacing the inbound model on a shared alb before every consumer was validated what we validated along the way not everything we worried about broke but everything we worried about deserved an explicit check stateful workloads on karpenter made us nervous zk and nimbus expect stable identity and disk karpenter nodes are ephemeral by design for non prod we accepted single replica zookeeper with no quorum ha in prod is still an open question arm64 was non negotiable on our karpenter pool private ecr images are single arch if they had been amd64 only we d have seen scheduling failures or runtime crashes on graviton we verified arch before calling the rollout done supervisor and logviewer logs live on emptydir pod restart means lost worker logs fine for non prod exploration scary for prod debugging we noted it and moved on durable log shipping is still tbd topology lifecycle is also tbd the chart runs the cluster it does not submit topologies how jar submit worker scaling and upgrades interact with deployments versus traditional supervisor hosts we haven t answered that yet we kept topologies out of scope on purpose so we could fail fast on pvcs and images without dragging application deployment into the first week by the end of the week all four failure chains were closed argo showed the storm app healthy we had the first apache storm cluster running on eks in the organization custom helm gitops managed on karpenter arm nodes zookeeper and nimbus statefulsets bound pvcs private images ran once we corrected entrypoints storm ui sat behind the same private alb pattern as the rest of other what i d do differently and what i d tell the next team if i were starting this again i d run a pre flight checklist before the first argocd app sync not after init containers had been waiting for forty eight hours storageclass exists on the target cluster gp3 via csi is not the same as in tree gp2 pvc pending cascades silently through init containers container entrypoint behavior docker run with the exact args your helm chart will pass especially for private images do not assume args nimbus from upstream examples argo statefulset volumeclaimtemplates emit the full schema or configure ignoredifferences upfront you will hit vct drift budget for it in the chart not as a week two surprise aws lbc clusterip target type ip and numeric backend ports named ports plus instance mode gave us an empty targetgroup port and a day of ingress debugging the bigger takeaway is about first in org workloads no reference deployment means every assumption arm storage ha logs ingress models on shared infrastructure has to be validated on the actual cluster writing down the fear inventory before we committed did not prevent the failures it did prevent us from treating them as mysteries we knew storage was a risk entrypoints were a risk argo statefulset sync was a risk alb quirks were a risk when each one fired we recognized it instead of spiraling we still went forward for good reasons platform direction is kubernetes maintaining a separate storm ec2 fleet alongside a new eks cluster doubles operational surface patching amis security groups deploy pipelines non prod first bounded blast radius we reused gitops muscle we already had with other apps alloy jenkins ingress phased delivery let us prove the control plane before workers and ui someone had to be first better on non prod with eyes open than a surprise prod migration later prod cutover still has open questions zookeeper ha with a three node quorum on eks a topology submit pipeline from ci supervisor horizontal scale and slot planning durable log shipping whether to adopt the community chart or keep the local one dns cutover from legacy storm ui and migrating the shared alb fully to sg based inbound after all apps are tested none of those blocked non prod stand up they are the next chapter storm on kubernetes is diy there is no official helm chart and no migration guide budget time for chart authorship and image entrypoint archaeology and when init containers say they re waiting for nimbus check whether nimbus ever had a chance to start the bug might be three layers below storm itself top comments 0 subscribe personal trusted user create template templates let you quickly answer faqs or store snippets for re use submit preview dismiss code of conduct report abuse are you sure you want to hide this comment it will become hidden in your post but will still be visible via the comment s permalink hide child comments as well confirm for further actions you may consider blocking this person and or reporting abuse mridul tiwari follow 19 love coding being cheerful and being dramatic also i am an artist new to open source contribution location india work devops engineer paytm joined feb 20 2022 more from mridul tiwari we tagged eks workers like app servers and prometheus started scraping telegraf on nodes that never had it devops kubernetes ai standing up a greenfield eks cluster and every mistake we stepped on devops kubernetes terraform when kafka hits 100 disk and the volume won t grow devops kafka ebs aws dev community a space to discuss and keep up software development and manage your software career home dev challenges dev videos dev education tracks dev help advertise on dev organization accounts dev showcase about contact free postgres database dev shop mlh code of conduct privacy policy terms of use built on forem the open source software that powers dev and other inclusive communities made with love and ruby on rails dev community 2016 2026 we re a place where coders share stay up to date and grow their careers log in create account