Meta tags:
author=Daniel Nashed;

Headings (most frequently used words):

domino, on, ubuntu, and, phone, daniel, nashed, container, linux, for, backup, blog, running, nomad, web, touch, in, checking, cpu, state, power, consumtion, run, reproducible, builds, include, phased, updates, updating, free, acme, cas, bsi, rsa, requirements, having, plan, economically, right, sizing, server, memory, victorialogs, log, forwarder, goes, opentelemetry, veeam, 13, application, repositories, perfect, fit, unix, troubleshooting, best, practices, tips, more, recent, entries, feeds, links, archives,

Text of the page (most frequently used words):
the (145), #domino (86), and (74), rsa (61), for (44), server (31), ubuntu (31), acme (28), this (27), with (27), october (24), september (22), backup (22), container (21), certificate (21), phone (20), linux (19), that (19), january (18), april (18), 2026 (18), can (18), december (17), march (17), july (17), february (16), november (16), memory (16), bsi (16), from (16), there (16), ram (16), not (16), 4096 (16), may (15), august (15), opentelemetry (15), data (15), use (15), are (15), more (15), have (15), cache (15), wildcard (15), certificates (15), june (14), free (14), but (14), using (13), daniel (13), nashed (13), loki (13), let (13), encrypt (13), 2048 (13), 2009 (12), 2010 (12), 2013 (12), 2016 (12), 2018 (12), 2019 (12), 2020 (12), 2022 (12), 2024 (12), 2025 (12), forwarder (12), docker (12), was (12), zerossl (12), arm (12), 2014 (11), 2017 (11), 2021 (11), victorialogs (11), touch (11), into (11), filesystem (11), 2023 (10), comments (10), also (10), logs (10), interesting (10), one (10), additional (10), chain (10), 2012 (9), 2015 (9), veeam (9), application (9), provides (9), important (9), its (9), good (9), which (9), other (9), otelfwd (9), when (9), benefit (9), actalis (9), yes (9), 2011 (8), updates (8), running (8), repository (8), zfs (8), used (8), https (8), http (8), grafana (8), only (8), all (8), like (8), test (8), because (8), notes (8), log (7), native (7), windows (7), performance (7), well (7), new (7), you (7), makes (7), same (7), different (7), otlp (7), has (7), services (7), even (7), current (7), provide (7), ecdsa (7), support (7), blog (7), script (7), cpu (6), power (6), nomad (6), very (6), servers (6), root (6), supports (6), project (6), available (6), doesn (6), image (6), still (6), traveler (6), therefore (6), uses (6), first (6), might (6), another (6), provider (6), 3072 (6), google (6), trust (6), right (5), requirements (5), having (5), plan (5), include (5), web (5), infrastructure (5), will (5), get (5), nfs (5), currently (5), isn (5), benefits (5), particularly (5), useful (5), simply (5), hcl (5), events (5), much (5), out (5), json (5), applications (5), just (5), working (5), what (5), victoria (5), pretty (5), looking (5), size (5), required (5), easy (5), european (5), alternative (5), key (5), 3000 (5), cores (5), repositories (4), sizing (4), reproducible (4), phased (4), run (4), about (4), environments (4), part (4), normal (4), while (4), restore (4), directly (4), today (4), standard (4), based (4), option (4), github (4), com (4), environment (4), configured (4), again (4), them (4), point (4), example (4), unix (4), add (4), version (4), most (4), utility (4), how (4), always (4), than (4), marginal (4), smaller (4), does (4), production (4), idea (4), tested (4), service (4), hierarchy (4), complete (4), issuing (4), work (4), autoupdate (4), lts (4), release (4), packages (4), build (4), start (4), update (4), emulation (4), containers (4), feed (3), fit (3), goes (3), updating (3), builds (3), each (3), own (3), copy (3), works (3), daos (3), now (3), stack (3), any (3), easier (3), receiver (3), change (3), output (3), records (3), sends (3), before (3), nginx (3), through (3), itself (3), anymore (3), been (3), quite (3), bit (3), started (3), small (3), here (3), looks (3), 9428 (3), setup (3), don (3), actually (3), every (3), instead (3), workloads (3), large (3), little (3), mail (3), nsf (3), workload (3), result (3), look (3), kernel (3), next (3), adding (3), where (3), aren (3), customer (3), providers (3), certstore (3), certmgr (3), your (3), gives (3), public (3), throughout (3), length (3), certification (3), path (3), least (3), yr1 (3), intermediates (3), keys (3), 02102 (3), template (3), distribution (3), couple (3), once (3), registry (3), later (3), amd64 (3), came (3), chatgpt (3), usb (3), suse (3), openssl (3), dns (3), aws (3), apple (3), content (2), perfect (2), economically (2), cas (2), checking (2), state (2), consumtion (2), creates (2), consistent (2), resulting (2), clean (2), set (2), per (2), difference (2), compared (2), integration (2), becomes (2), via (2), should (2), testing (2), provided (2), deduplication (2), way (2), access (2), fast (2), need (2), configuration (2), file (2), databases (2), they (2), contains (2), event (2), monitoring (2), allows (2), push (2), backend (2), case (2), format (2), perspective (2), original (2), longer (2), api (2), automatically (2), resources (2), send (2), interface (2), structured (2), socket (2), local (2), console (2), several (2), multiple (2), depend (2), week (2), volumes (2), command (2), victoriametrics (2), simple (2), goal (2), enough (2), valuable (2), headroom (2), providing (2), could (2), 640 (2), physical (2), argument (2), configurations (2), become (2), expensive (2), cheap (2), smtp (2), busy (2), many (2), substantially (2), jvm (2), around (2), actual (2), high (2), significant (2), lower (2), frequently (2), accessed (2), less (2), diminishing (2), being (2), machine (2), unused (2), applies (2), eventually (2), automation (2), primary (2), avoids (2), single (2), configure (2), straightforward (2), particular (2), individual (2), availability (2), make (2), without (2), issue (2), www (2), recommendation (2), documentation (2), ssl (2), returned (2), solve (2), requirement (2), wr1 (2), letsencrypt (2), intermediate (2), includes (2), pdf (2), tls (2), practice (2), means (2), post (2), certs (2), upgrade (2), built (2), fixpack (2), verse (2), apt (2), security (2), some (2), nice (2), then (2), things (2), system (2), challenging (2), getting (2), device (2), mobile (2), efficiency (2), off (2), desktop (2), yubikey (2), box (2), rdp (2), workshop (2), vss (2), v12 (2), v10 (2), kvm (2), clamav (2), tips (2), timedate (2), mime (2), rancher (2), client (2), lotus (2), lab (2), k3s (2), ibm (2), champion (2), download (2), flexnet (2), feature (2), pack (2), dnug (2), dbmt (2), centos (2), cli (2), alpine (2), 2008, archives, links, comment, feeds, recent, entries, contact, remarkably, natural, combination, manages, integrates, overall, separation, responsibilities, pool, permissions, arbitrary, remains, responsible, creating, functionality, workflows, mount, scenario, nfsv3, configurable, gui, enable, dedup, sets, nfsv4, special, agent, side, mode, matches, architecture, transaction, target, flow, storage, backups, nashcom, see, updated, enhanced, observability, platforms, backends, turned, ingestion, representation, endpoint, probably, biggest, architectural, implementation, reachable, kept, locally, replayed, write, ahead, wal, converts, groups, incoming, scopes, sending, batched, efficient, transmission, importantly, reliable, delivery, those, inputs, forwarding, error, datagram, syslog, 127, tcp, connection, bound, step, further, reads, queue, domfwd, simplest, stdout, piped, ways, input, methods, receivers, changes, specific, over, last, forward, evolved, generic, retentionperiod, 30d, storagedatapath, ports, restart, hostname, container_name, latest, setting, compose, localhost, insert, entry, protobuf, docs, formats, hit, radar, favorit, fits, remaining, metrics, knows, according, value, economic, beyond, helpsâ, decreases, largest, possible, capture, reasonable, operational, max, progressively, capacity, reduce, cost, consider, vms, their, meaningfully, years, common, nearly, convincing, virtualized, overprovisioning, accumulates, quickly, huge, amounts, create, directory, accessing, consumers, process, external, database, suggests, availableâ, tests, reducing, did, corresponding, dramatic, increase, disk, continued, behave, doubling, double, numbers, illustrative, curve, depends, principle, conceptually, gbs, extremely, contain, grows, increasingly, retained, incremental, gets, happily, whenever, conclude, distinction, processes, remainder, usage, normally, remain, approximately, meaningful, question, concept, economics, surprisingly, helps, usually, generally, investment, matters, building, significantly, regardless, additionally, unnecessary, dependency, contractual, relationship, both, multi, strategy, requests, tied, depending, discussed, accounts, centrally, account, reverse, typically, contract, sla, policies, hierarchies, rate, limits, conditions, protocol, reason, unnecessarily, dependent, lesson, exercise, sizes, fairly, practical, clients, aspx, customers, italy, meets, limitation, nevertheless, anywhere, usertrust, authority, sectigo, authentication, r46, pki, goog, relevant, globalsign, gts, including, org, address, introducing, moving, issuance, yr2, yr3, would, require, increasing, though, isrg, generation, results, prompted, bund, shareddocs, downloads, publikationen, technischerich, tlinien, tr02102, __blob, publicationfile, recommendations, nothing, wrong, webpki, browser, however, german, recommends, minimum, signing, larger, bits, raised, concern, spend, time, breaking, nerves, text, copied, somewhere, else, reformatted, disclaimer, formatted, gave, after, minutes, find, details, glibc, upgraded, optional, commercial, firewall, coverage, extra, favorite, deploying, installation, shutdown, remove, done, distribute, domain, wide, harbor, ons, seconds, interim, fix, potentially, etc, introduced, clicks, never, base, cleaning, older, versions, known, distributions, debian11, debian10, along, soon, usr, bin, true, manually, sense, included, perfectly, phasing, fractions, machines, progression, continue, successful, startup, finally, short, section, really, eight, leaves, plenty, experimenting, why, x64, had, installed, snap, stage, installing, software, operating, mostly, read, switching, adventure, devices, spacewar, picked, refurbished, got, mine, limited, number, models, needs, specifically, adapted, hardware, emergency, form, factor, tablet, during, discussion, projects, already, yet, platform, types, general, similar, raspberry, second, shows, information, interval, cool, spelunking, hooked, bank, turning, cpus, big, effective, tune, maximum, frequency, scripts, found, far, turn, idle, consumption, wrote, battery, status, prime, core, pulled, down, inside, parts, deploy, components, room, mow, firefox, level, crafted, window, added, forumlas, domino2025, 509, x509, wsl2, wsl, wireguard, windowsserver, sandbox, win64, widget, writer, vmware, virustotal, replication, utm, truenas, scale, trivy, tpm2, toubleshooting, totp, tools, tool, tip, tika, tar, systemd, symphony, harvester, sudo, statistics, ssh, squid, spr, splunk, spf, spamgeek, spam, solaris, snmp, sles, sigpipe, sftp, index, selinux, scalability, saml, sametime, sai, safelinx, rspamd, rnug, rhel, restic, redhat, rapidjson, rag, proxmox, prometheus, private, privacy, pqc, poodle, podman, photonos, photon, passthru, ots, osx, openssh, openshift, openntf, onetouchsetup, configuation, ollama, oidc, odd, ocr, nvme, nvidia, nsl, nshrun, nshmailx, nshdellog, nshcertool, nshback, nsd, ini, notes9, nomab, nivia, nifnsf, news, technologies, network, nashcertool, minio, microsoft, microca, mhs, meetings, meeting, mastodon, markdown, mailto, macos, macbook, mac, lxc, ls2capi, lotusscript, loadtest, llm, llama, livetext, libssh, leap, le4d, kyrtool, kyr, kubernetes, keyring, k8s, traefik, k11, jwt, jdbc, java, iphone, ios, intel, install, inotes, imap, idvault, idma, vault, icap, connected, hyper, hetzner, ambassador, hashicorp, git, gdb, fun, fritz, fritzbox, fp9, fp10, formulas, forgejo, licensing, fido2, fail2ban, exchange, engage, encryption, enagage, eap, domio, perfomance, dominoiq, dominodownload, dominoctl, dominobackup, dominoautoupdate, domino8, domino10, dnachnug, dkim, discord, digital, ocean, debian, das, daostune, dachnug, cve, curl, cuda, cscan, crowdsec, critical, problem, crash, codeberg, clustering, cluster, clamd, citrix, ciphers, stream, buypass, bug, borgbackup, borg, bash, base64, backujp, attachments, astra, asta, as16, silicon, silcon, appdevpack, appdev, anti, virus, ansible, alloy, aix64, aix, admincentral, adfs, 64bit, tags, troubleshooting, best, practices, imprint,

Text of the page (random words):
102 2 pdf __blob publicationfile this prompted me to test several free acme providers with rsa 4096 certificates and look at the complete certificate chain not just the server certificate acme ca results let s encrypt free acme yes wildcard yes rsa ca chain includes rsa 2048 bsi rsa 3000 no google trust services free acme yes wildcard yes rsa ca chain includes rsa 2048 bsi rsa 3000 no zerossl free acme yes wildcard yes rsa ca chain rsa 3072 4096 throughout bsi rsa 3000 yes actalis free acme yes wildcard no not free at least rsa ca chain rsa 4096 throughout bsi rsa 3000 yes based on the certificate chain returned in my test let s encrypt let s encrypt s current generation y rsa hierarchy uses rsa 2048 issuing intermediates with an rsa 4096 server certificate the chain looks like this rsa 4096 server certificate rsa 2048 let s encrypt yr1 rsa 4096 root yr rsa 4096 isrg root x1 the important part is yr1 even though the server certificate itself uses rsa 4096 the certificate chain contains an rsa 2048 issuing ca therefore increasing the rsa key size of the server certificate doesn t solve the bsi requirement let s encrypt could address this by introducing new rsa 3072 or rsa 4096 issuing intermediates and moving rsa issuance to them the current yr1 yr2 yr3 intermediates use rsa 2048 keys so this would require new intermediate ca keys and certificates let s encrypt certificate hierarchy https letsencrypt org certificates google trust services google trust services also provides free acme certificates including wildcard certificates i tested an rsa 4096 wildcard certificate the resulting chain was rsa 4096 server certificate rsa 2048 google trust services wr1 rsa 4096 gts root r1 rsa 2048 globalsign root ca again the relevant issuing ca wr1 uses rsa 2048 google trust services therefore doesn t solve this particular requirement when using its current rsa certificate hierarchy google trust services certificate repository https pki goog repository zerossl â rsa with wildcard support zerossl is the interesting result when rsa certificates are required zerossl supports free acme wildcard certificates and the complete rsa certification path returned in my test uses at least rsa 3072 rsa 4096 server certificate rsa 3072 zerossl rsa dv ssl ca 2 rsa 4096 sectigo public server authentication root r46 rsa 4096 usertrust rsa certification authority there is therefore no rsa 2048 certificate anywhere in this certification path this makes zerossl a good option when rsa wildcard certificates and the bsi rsa key length recommendation are all requirements zerossl acme documentation https zerossl com documentation acme actalis â a european alternative actalis is another interesting acme provider particularly for european customers actalis is based in italy and provides free acme certificates in my test the complete rsa chain used rsa 4096 so it also meets the bsi rsa key length recommendation the important limitation is that actalis currently doesn t provide wildcard certificates through its free acme service it is nevertheless an interesting european alternative for environments where wildcard certificates aren t required actalis acme https www actalis com acme certificates aspx what about ecdsa there is another straightforward option use ecdsa instead of rsa let s encrypt s current ecdsa hierarchy avoids the rsa 2048 issue if all clients and applications in your environment support ecdsa certificates this is a good option this gives us a fairly simple practical result ecdsa wildcard â let s encrypt rsa â 3072 throughout wildcard â zerossl european ca without wildcard â actalis have a plan b there is another lesson from this exercise which might be even more important than the rsa key sizes don t depend on a single acme ca when using a free public acme service a customer typically doesn t have an individual contract or sla with that ca availability policies certificate hierarchies rate limits or service conditions can change acme is a standard protocol so there is little reason to make certificate infrastructure unnecessarily dependent on one provider for example even if let s encrypt is your normal ca having a zerossl acme account configured and tested gives you an alternative the reverse applies when zerossl is your primary provider the important point is to test the alternative before you actually need it domino certmgr makes this easy hcl domino certmgr supports the acme providers discussed here and makes it easy to configure multiple acme accounts centrally in certstore nsf certificate requests aren t tied to let s encrypt different acme providers can be configured and used depending on the requirements of a particular certificate that makes a multi provider strategy straightforward ecdsa â let s encrypt rsa â zerossl plan b â configure and test both actalis can additionally be useful as a european alternative where wildcard certificates aren t required having more than one acme provider configured and tested avoids unnecessary dependency on a single free service with which the customer might not have any contractual relationship for production certificate automation having a working plan b is a good idea regardless of which ca is the primary provider comments 0 domino linux economically right sizing domino server memory on linux daniel nashed 23 september 2026 10 18 37 more ram is generally good for a domino server but more ram is not automatically a good investment this matters more today because ram has become significantly more expensive again particularly when building servers with large memory configurations a useful concept from economics applies surprisingly well to server sizing diminishing marginal utility the first additional gb of ram can provide significant performance benefits the next gb still helps but usually a little less eventually adding another 32 gb might provide only a small additional benefit the interesting question therefore isn t how much ram can domino use it is how much ram provides a meaningful benefit for a workload on linux unused ram normally doesn t remain unused linux uses available memory as filesystem cache a 64 gb domino server might therefore look approximately like this 64 gb ram 8â 10 gb domino other processes 50 gb filesystem cache remainder kernel other usage it is easy to look at this and conclude that the machine benefits from all 64 gb but there is an important distinction memory being used is not the same as memory being required linux will happily use additional ram to cache filesystem data whenever memory is available diminishing marginal utility the first gbs of filesystem cache are extremely valuable because they contain the most frequently accessed data as the cache grows increasingly less frequently accessed data is retained additional ram still provides a benefit but the incremental benefit gets smaller conceptually filesystem cache marginal benefit 8 gb very high 16 gb high 24 gb significant 32 gb useful 48 gb lower 64 gb lower still the numbers are only illustrative the actual curve depends on the workload the important principle is doubling the filesystem cache does not double its performance benefit in my tests on one busy domino mail server for example linux was using around 55 gb of filesystem cache reducing the available cache substantially did not result in a corresponding dramatic increase in physical disk i o even around 24 gb of cache the server continued to behave very well that suggests that much of the additional cache was useful because it was availableâ not because the workload actually required it different domino workloads are different a mail server traveler server directory server and smtp server do not have the same memory requirements a busy mail server accessing many nsf databases can benefit substantially from filesystem cache traveler is quite different its important memory consumers include the jvm domino http and native process memory while an ha configuration uses an external database for traveler data providing huge amounts of ram to a traveler vm simply to create a large filesystem cache may therefore provide little additional benefit ram isn t cheap anymore for years a common argument was ram is cheap just add more that argument isn t nearly as convincing anymore large server memory configurations have become expensive again and in virtualized environments overprovisioning accumulates quickly consider 20 domino vms with 32 gb more ram each than their workloads meaningfully benefit from 20 ã 32 gb 640 gb that s 640 gb of physical infrastructure memory providing progressively smaller benefits the same memory could instead provide capacity for additional workloads ha headroom or simply reduce the cost of the infrastructure right size instead of max size the goal should not be to provide every domino server with the largest possible filesystem cache the goal is to provide enough application memory enough filesystem cache to capture the valuable part of the working set and reasonable operational headroom beyond that point additional ram still helpsâ but its marginal utility decreases that s the economic part of right sizing don t size memory simply because linux knows how to use it size it according to the value that additional memory actually provides comments 0 opentelemetry victorialogs for domino daniel nashed 21 september 2026 16 25 15 grafana loki is my current favorit and fits well to the the remaining grafana stack we have for domino metrics when looking into opentelemetry formats victorialogs hit my radar it s pretty simple to setup in a docker container and works pretty well even on a local setup https docs victoriametrics com victorialogs the entry point for opentelemetry is different than for loki and the format is protobuf http localhost 9428 insert opentelemetry v1 logs setting up a container via docker compose services victorialogs image victoriametrics victoria logs latest container_name victorialogs hostname victorialogs restart always ports 9428 9428 command storagedatapath victoria logs data retentionperiod 30d volumes victoria logs data victoria logs data volumes victoria logs data here is how it looks like comments 0 opentelemetry grafana loki victorialogs domino domino log forwarder goes opentelemetry daniel nashed 21 september 2026 08 43 26 i have been working quite a bit on my domino log forwarder project over the last week what started as a small utility to forward domino console output to grafana loki has evolved into a more generic opentelemetry log forwarder one of the most important changes is that there is no loki specific output anymore otelfwd now sends opentelemetry logs using otlp http json grafana loki is one of the receivers i use and test with but the forwarder itself doesn t depend on loki one forwarder multiple input methods there are several ways to feed data into otelfwd the domino console is the simplest one domino stdout is piped into otelfwd just like with the original version of the project the new domfwd domino server add in goes a step further it reads structured events directly from the domino event monitoring queue and sends them to otelfwd through a local unix socket other applications can use the same structured interface through a unix socket or a tcp connection bound to 127 0 0 1 there is a unix datagram syslog interface i use this for example to send nginx access and error logs directly into otelfwd all those inputs use the same forwarding infrastructure from that point on reliable otlp delivery otelfwd converts and groups the incoming records into opentelemetry resources and scopes and sends them using otlp http json before sending records are batched for efficient transmission more importantly otelfwd has its own write ahead log wal if the configured otlp receiver isn t reachable records are kept locally and automatically replayed when the receiver becomes available again the backend is no longer loki only this is probably the biggest architectural change compared with the original domino log forwarder there is no longer a loki push api implementation in the forwarder the output is opentelemetry only for my own environment i currently use grafana loki which provides an otlp endpoint but from the perspective of otelfwd loki is simply an otlp receiver this also makes it much easier to use the same forwarder with other observability platforms while testing other backends victorialogs turned out to be an interesting case victorialogs also supports opentelemetry logs but its native opentelemetry json ingestion format is different from the standard otlp http json representation used by otelfwd native domino events forwarder the project now contains also a domino events forwarder it s a native integration into the domino event monitoring stack and allows you to push enhanced events to any opentelemetry backend see the updated project on github https github com nashcom domino log forwarder comments 0 veeam zfs veeam backup 13 1 application repositories perfect fit for domino backup daniel nashed 19 september 2026 18 11 03 veeam backup application repositories are an interesting new option for application backups â and they are a particularly good fit for hcl domino backup restore the repository is based on zfs and provides nfs storage that application servers can use directly this matches the domino backup architecture very well domino backup restore can simply copy databases transaction logs daos data and other backup data to an nfs target using its standard file copy flow there is no need for a special backup agent or configuration on domino backup side the standard file copy mode works for backup and restore the repository supports nfsv4 and benefits from zfs fast deduplication which can be particularly useful for domino backup data deduplication isn t configurable in the gui today but there is a way to get root access to the repository to enable fast dedup on the zfs data sets linux domino servers can mount the repository directly via nfs on windows nfs should currently also be used for this scenario in my testing nfsv3 from windows provided good performance the important difference compared with using an arbitrary nfs server is the integration with veeam the application repository becomes part of the normal veeam backup workflows while domino remains responsible for creating an application consistent backup using its native backup restore functionality each server will get it s own zfs data set in zfs pool with permissions per server this creates a very clean separation of responsibilities domino creates the application consistent backup veeam provides and manages the backup repository and integrates the resulting data into the overall backup infrastructure for domino environments that is a remarkably natural combination comments 0 about contact recent entries running nomad web on ubuntu touch phone in a container ubuntu phone checking cpu state and power consumtion run domino on a phone reproducib...