Meta tags:
description= My main storage array is 839 GB of btrfs sitting on a mdadm RAID5. Last month it was reading half... Tagged with storage, homelab, linux, sysadmin.;
keywords= storage, homelab, linux, sysadmin, software, coding, development, engineering, inclusive, community;
Headings (most frequently used words):
the, 333, gb, didn, know, was, keeping, dev, community, inventory, before, you, touch, anything, why, don, hand, edit, omv, config, sentinel, uuid, gotcha, what, it, actually, cost, top, comments, more, from, kingsley, kanu,
Text of the page (most frequently used words):
the (78), and (31), you (17), that (16), share (15), dev (13), omv (13), uuid (8), about (7), not (7), from (6), what (6), was (6), one (6), had (6), delete (6), #fullscreen (6), mode (6), rpc (6), homelab (5), storage (5), disk (5), can (5), 333 (5), export (5), folder (5), its (5), config (5), with (4), community (4), software (4), still (4), more (4), work (4), half (4), full (4), then (4), data (4), minutes (4), which (4), shared (4), definition (4), stale (4), file (4), went (4), where (3), space (3), than (3), kingsley (3), kanu (3), 2024 (3), write (3), iamkay (3), backups (3), for (3), look (3), year (3), folders (3), were (3), have (3), inventory (3), stays (3), find (3), back (3), first (3), nfs (3), smb (3), entry (3), sentinel (3), time (3), gone (3), hand (3), exit (3), enter (3), admin (3), orphan (3), are (3), same (3), anything (3), through (3), openmediavault (3), meaning (3), empty (3), create (2), account (2), log (2), open (2), source (2), code (2), conduct (2), keep (2), your (2), proxmox (2), btrfs (2), preview (2), why (2), sysadmin (2), monitoring (2), nothing (2), month (2), tells (2), did (2), founder (2), oak (2), techx (2), production (2), report (2), abuse (2), comments (2), originally (2), published (2), running (2), most (2), until (2), every (2), while (2), put (2), this (2), once (2), things (2), own (2), array (2), actually (2), purpose (2), seven (2), deletion (2), safe (2), when (2), want (2), referencing (2), service (2), got (2), pointing (2), reference (2), does (2), confirmed (2), directories (2), definitions (2), path (2), already (2), sharemgmt (2), apply (2), name (2), web (2), salt (2), gets (2), xml (2), save (2), almost (2), edit (2), output (2), line (2), three (2), like (2), but (2), they (2), part (2), later (2), deep (2), tell (2), deleted (2), srv (2), head (2), before (2), looking (2), didn (2), know (2), keeping (2), copy (2), link (2), search (2), place, coders, stay, date, grow, their, careers, made, love, 2016, 2026, ruby, rails, built, powers, other, inclusive, communities, forem, terms, use, privacy, policy, mlh, shop, free, postgres, database, contact, showcase, organization, accounts, advertise, help, education, tracks, videos, challenges, home, discuss, development, manage, career, flags, technology, guessed, wrong, stack, costs, paid, tier, prometheus, observability, may, joined, amsterdam, netherlands, location, fullstack, engineer, mern, automation, devops, ship, pipelines, reliable, infra, follow, dismiss, submit, subscribe, top, infrastructure, failures, fixes, tradeoffs, fills, ghosts, old, finished, migrations, shares, services, retired, never, complains, archaeology, harder, cheaper, remember, reminder, calendar, cron, job, deletes, automating, against, schedule, just, holding, decide, thirty, imbalance, whole, lesson, dangerous, fast, slow, correct, ratio, opposite, how, feels, moment, see, cost, order, avoids, remove, backwards, saw, left, walk, uses, special, set, points, stores, out, under, always, stop, fall, end, resolves, symptom, client, mount, hangs, 404s, rather, error, fa4b1c66, ef79, 11e5, 87a0, 0002b3a176b4, gotcha, actual, bytes, removed, plain, after, script, looped, over, wildcard, named, specific, tagged, modules, force, false, applychanges, per, target, reldirpath, tsv, enumeratesharedfolders, enumerate, working, uuids, names, supported, interface, drives, shell, goes, validation, destruction, scripted, truth, state, here, thing, catches, people, keeps, entire, configuration, right, there, readable, editable, block, will, next, regenerates, based, backend, whenever, change, either, clobbered, worse, leaves, system, disagreeing, exists, etc, don, text, tagging, each, active, something, reads, writes, real, nobody, needs, anymore, those, stub, has, underlying, directory, all, stubs, tiny, bites, list, too, gives, fat, ranked, size, levels, usually, enough, culprit, without, drowning, second, finds, ones, box, often, tale, whose, moved, stayed, behind, type, sort, max, depth, rule, hold, boring, down, two, commands, touch, twenty, answer, touched, since, plus, scatter, longer, existed, lot, interesting, number, been, invisible, makes, cleanup, only, way, wants, main, 839, sitting, mdadm, raid5, last, reading, couldn, bothered, would, because, least, linux, posted, jul, post, via, mastodon, facebook, linkedin, copied, clipboard, pick, gem, boost, jump, fire, raised, hands, exploding, unicorn, add, reaction, close, powered, algolia, navigation, menu, skip, content,
Text of the page (random words):
the 333 gb i didn t know i was keeping dev community skip to content navigation menu search powered by algolia search log in create account dev community close add reaction like unicorn exploding head raised hands fire jump to comments save boost pick as gem more copy link copy link copied to clipboard share to x share to linkedin share to facebook share to mastodon share post via report abuse kingsley kanu posted on jul 26 originally published at iamkay eu the 333 gb i didn t know i was keeping storage homelab linux sysadmin my main storage array is 839 gb of btrfs sitting on a mdadm raid5 last month it was reading half full and i couldn t tell you what half that bothered me more than a full disk would have because a full disk at least tells you where the space went so i went looking twenty minutes later i had my answer 333 gb of backups from 2024 that i had not touched since plus a scatter of empty share definitions pointing at folders that no longer existed i deleted the lot in about seven minutes the interesting part is not the number it s that the space had been invisible and that openmediavault makes the cleanup safe only if you do it the way omv wants inventory before you touch anything the rule i hold to on storage is boring on purpose look before you delete and write the looking down two commands did most of it du h max depth 3 srv dev disk by uuid sort rh head 40 find srv type d empty enter fullscreen mode exit fullscreen mode the first gives you the fat directories ranked by size three levels deep which is usually deep enough to name the culprit without drowning in output the second finds the empty ones which on an omv box are often the tell tale of a share whose data got moved or deleted while the share definition stayed behind i put the output in a text file and went through it line by line tagging each entry as one of three things active meaning something reads or writes it and it stays stale meaning it is real data nobody needs anymore like those 2024 backups orphan stub meaning omv still has a shared folder entry for it but the underlying directory is gone the 333 gb was almost all stale the orphan stubs were tiny on disk but they are the part that bites you later so they went on the list too why i don t hand edit omv s config here is the thing that catches people openmediavault keeps its entire configuration in one xml file at etc openmediavault config xml and it is right there readable editable you can open it find the share you want gone delete the block save it will work until the next time omv regenerates config from that file which it does through its salt based backend whenever you change almost anything in the web ui your hand edit either gets clobbered or worse leaves the running system and the config file disagreeing about what exists the supported path is omv rpc the same rpc interface the web ui drives anything the ui can do omv rpc can do from the shell and it goes through the same validation and the same salt apply so the destruction gets scripted and omv stays the source of truth about its own state enumerate the shared folders first so you are working from uuids not names omv rpc u admin sharemgmt enumeratesharedfolders jq r uuid name reldirpath tsv enter fullscreen mode exit fullscreen mode then per target i had already confirmed as stale or orphan delete the share definition by its uuid and apply omv rpc u admin sharemgmt delete uuid the uuid omv rpc u admin config applychanges modules force false enter fullscreen mode exit fullscreen mode the actual bytes i removed with a plain rm rf on the confirmed stale directories one at a time after the share definitions were gone no script looped over a wildcard every deletion named one specific path i had already tagged by hand the sentinel uuid gotcha omv uses one special uuid fa4b1c66 ef79 11e5 87a0 0002b3a176b4 as its not set reference when a service an nfs export or an smb share points at a shared folder it stores that folder s uuid delete the folder definition out from under a still referencing export and omv does not always stop you and the reference can fall back to that sentinel you end up with an nfs export or an smb share that resolves to nothing and the symptom is a client mount that hangs or 404s rather than an error at delete time the order that avoids it remove the referencing service first the nfs export the smb share then the shared folder definition then the data i got this backwards on one entry saw an export left pointing at the sentinel and had to walk it back what it actually cost thirty minutes of inventory seven minutes of deletion the imbalance is the whole lesson the dangerous work was fast and the safe work was slow which is the correct ratio and the opposite of how it feels in the moment when you can see 333 gb you want back i have put a reminder in the calendar to do this once a year not a cron job that deletes things i am not automating rm rf against my own data on a schedule just the inventory half the du and the find so once a year i have to look at what the array is actually holding and decide on purpose what stays most homelab storage fills with ghosts old backups half finished migrations shares for services you retired the disk never complains until it is full and by then the archaeology is harder cheaper to look every year while you still remember what the folders were for originally published on iamkay eu where i write about running production infrastructure the failures the fixes and the tradeoffs top comments 0 subscribe submit preview dismiss code of conduct report abuse kingsley kanu follow fullstack engineer js ts mern qa automation and devops i ship production software and the pipelines that keep it reliable founder oak techx i write about homelab and infra at iamkay eu location amsterdam netherlands work founder oak techx joined may 10 2024 more from kingsley kanu my monitoring stack costs nothing a month and tells me more than the paid tier did monitoring observability prometheus homelab proxmox still flags btrfs as a technology preview and i guessed wrong about why proxmox storage homelab sysadmin dev community a space to discuss and keep up software development and manage your software career home dev challenges dev videos dev education tracks dev help advertise on dev organization accounts dev showcase about contact free postgres database dev shop mlh code of conduct privacy policy terms of use built on forem the open source software that powers dev and other inclusive communities made with love and ruby on rails dev community 2016 2026 we re a place where coders share stay up to date and grow their careers log in create account
|
