Meta tags:
author=Jean Niklas hyPiRion L orange;
description=I could ensure my OCaml builds are 100% reproducible, but what s the cost?;

Headings (most frequently used words):

nixing, builds, so, then, hypirion, why, not, my, ocaml, yet, opam, switches, esy, broken, learning, nix, flakes, and, unstable, ground, one, thing, at, time, what, the, solution,

Text of the page (most frequently used words):
the (68), nix (41), that (35), and (35), for (19), with (18), this (17), you (13), not (13), there (12), have (12), but (12), build (11), use (11), ocaml (11), esy (10), opam (10), will (9), from (9), reproducible (9), flakes (9), about (9), one (9), when (8), are (8), probably (8), project (8), glibc (7), would (7), like (7), learn (7), #builds (7), get (7), now (6), ensure (6), dependencies (6), what (6), though (6), all (6), can (6), while (6), time (6), could (6), repl (6), bash (6), running (6), post (5), out (5), seems (5), overlays (5), just (5), only (5), same (5), how (5), through (5), switch (5), comes (4), work (4), blog (4), was (4), then (4), help (4), some (4), great (4), being (4), using (4), means (4), know (4), which (4), after (4), nixpkgs (4), avoid (4), usually (4), something (3), your (3), well (3), much (3), documentation (3), pain (3), manage (3), right (3), problem (3), isn (3), development (3), need (3), other (3), version (3), lot (3), sounds (3), variables (3), unstable (3), future (3), wanted (3), why (3), little (3), derivation (3), pill (3), store (3), try (3), pills (3), someone (3), setup (3), learnt (3), stuff (3), myself (3), possible (3), install (3), even (3), nixing (3), environments (3), per (3), yet (3), switches (3), new (3), 2024 (2), clearly (2), also (2), had (2), better (2), than (2), first (2), already (2), mentioned (2), example (2), going (2), should (2), without (2), any (2), solution (2), initial (2), system (2), having (2), available (2), tiny (2), used (2), managed (2), were (2), find (2), library (2), again (2), agenda (2), nixos (2), projects (2), where (2), hard (2), certainly (2), over (2), quite (2), more (2), thing (2), reproducibility (2), yeah (2), experimental (2), they (2), book (2), two (2), point (2), around (2), information (2), properly (2), enough (2), bit (2), weird (2), under (2), added (2), thought (2), working (2), here (2), nice (2), else (2), tricks (2), things (2), port (2), packages (2), make (2), sort (2), either (2), fewer (2), tools (2), installed (2), very (2), cost (2), rather (2), similar (2), others (2), big (2), dynamic (2), set (2), image (2), really (2), linux (2), guarantee (2), encountered (2), recommended (2), broken (2), fix (2), case (2), until (2), dependency (2), won (2), compiler (2), itself (2), last (2), editor (2), getting (2), automatically (2), merlin (2), lsp (2), server (2), most (2), old (2), copyright, jean, niklas, orange, caveat, architectures, say, machine, ways, both, architecture, targets, understand, mostly, core, package, maintainers, fuzz, given, share, fun, wouldn, surprised, been, exposed, aarch64, x86_64, note, situation, wrote, fixed, sync, its, unless, box, pinning, ppx_rapper, dream, silver, bullet, combination, think, fine, aiming, recall, pains, always, compile, part, main, git, revisions, nor, directly, such, issue, resolve, depends, later, still, tool, pet, servers, able, leverage, small, don, kubernetes, cluster, cattle, wonderful, mind, spread, place, wanting, focus, skipped, reasonml, frontend, begin, changing, too, many, recipe, disaster, kinda, defeats, purpose, definition, necessarily, versions, current, heat, death, universe, exactly, viable, long, term, strategy, feature, according, features, considered, changed, removed, docs, entire, reliable, toy, reasons, stopped, looking, best, way, ground, anyway, understanding, move, sure, makes, sense, dig, deeper, initially, greeted, nothing, telling, moving, target, pulling, rug, bumping, says, sugar, expression, import, 21422, h3bhzvz9ipglcybbcvkxvm4vg9lwvqg4, 2p26, reproduce, behaviour, 3950, ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6, p45, said, impression, wasn, consider, author, finds, location, calls, made, antsy, compress, binaries, exclude, gcc, reached, tired, skimmed, rest, felt, gathered, earlier, navigate, likely, end, copying, hack, till, worked, introduction, nitty, gritty, details, tips, want, existing, hand, conceptually, osmosis, kinds, places, since, may, start, went, bottom, learning, results, worse, performance, debugging, options, negative, portability, easier, limiting, language, hidden, dns, tcp, implementation, force, key, value, implemented, battle, tested, leveldb, rocksdb, java, runtime, jni, reason, sqlite, perhaps, extent, nonzero, startup, see, trivial, importantly, annoyance, docker, container, images, libraries, standalone, executables, poor, man, suppose, deal, copy, steps, based, template, because, basically, mess, ends, wrong, practice, ymmv, reflecting, actually, haven, anger, few, team, leads, debug, document, stupid, differences, between, macos, never, gotten, claims, thus, giving, our, different, machines, suggestion, who, read, mismatch, pointing, towards, seem, covers, approach, write, file, flake, unlikely, impossible, imagine, happen, immediate, roll, down, lockfile, fact, buildable, above, issues, related, pin, newer, pg_query, difference, however, forces, handle, turn, investigate, fair, solve, theory, natural, choice, has, isolated, lockfiles, installs, concern, once, knew, didn, pressing, ensuring, exact, off, perfect, terminal, setups, calling, default, global, setting, afraid, multiple, confuse, eval, env, change, back, create, recent, release, continue, environment, familiar, independent, those, called, whatever, depend, accessible, within, got, good, feedback, community, offering, solutions, points, faced, during, talked, exploring, web, posted, jun, posts, home, hypirion,

Text of the page (random words):
why i m not nixing my ocaml builds yet hypirion home posts about why i m not nixing my ocaml builds yet posted 21 jun 2024 after my last blog post about ocaml and web development i got a lot of good feedback from the ocaml community offering solutions to the pain points i faced during my project setup some talked about opam tricks while others recommended exploring nix overlays opam switches in case you re not familiar you can have independent environments with opam those environments are called switches and whatever you depend on or install will only be accessible within that switch if i were to continue using opam i d probably have an environment per project to build a new one i d do opam switch create my new project 5 2 0 5 2 0 being the most recent ocaml compiler release and install my dependencies and tools there if i wanted to change back to my old project i could do opam switch set my old project this isn t perfect though while it is possible to have per terminal opam switch setups by calling eval opam env switch my new project they are by default global and setting this up with my editor is probably possible but i m afraid working on multiple projects at the same time will confuse merlin the lsp server i already knew about switches but it didn t help me with my most pressing problem ensuring that my build is reproducible to ensure that the exact same dependencies are used i m probably better off using esy so esy then in theory the natural choice for development would be esy esy has isolated environments per project automatically lockfiles and installs not yet installed dependencies automatically my only concern would once again be how i m getting merlin the lsp server to work with esy itself the difference from opam however is that esy forces you to handle this which means there should be documentation on how to get your editor up and running with it which in turn means i probably won t have to investigate as much though to be fair to opam the solution for getting esy up and running is probably similar to how you d solve it there but even with a lockfile this will not ensure my build is reproducible in fact it won t even guarantee that it will be buildable in the blog post mentioned above i encountered two issues related to glibc one in the ocaml compiler itself and one in the dependency pg_query i d have to pin myself out of the last one to get it to work with my newer glibc while unlikely it s not impossible to imagine that this could happen in the future and that there s no immediate fix available in that case my build will be broken until either i or someone else fix the dependency or until i roll down my glibc version nixing broken builds a suggestion from someone who read my blog post was to use nix overlays to avoid the glibc mismatch i encountered pointing me towards nix ocaml nix overlays nix overlays seem to use nix flakes under the covers as the recommended approach is to write a flake nix file one of the things i ve never really gotten around to is to try out nix nix sounds great to me it claims to ensure builds are reproducible thus giving us a guarantee that our stuff will build in the future even if we re on different machines 1 reflecting on it it s actually quite weird that i haven t learnt it in anger i m usually one of the few running linux in a team which usually leads me to debug and document stupid differences between linux and macos clearly nix could help me out here right perhaps to some extent but there s a nonzero startup cost for one i d have to get the others to install use and learn nix as we ll see that s not trivial more importantly through my annoyance with big docker container images i ve learnt to avoid dynamic libraries and rather make standalone executables and for ci cd i ve usually managed to set up a reproducible image that builds stuff sort of like a poor man s nix i suppose when that s up and running usually there s not really that big of a deal to have a nix setup you can copy the build image s steps and get your system up and running based on that template because there s basically no dynamic library you have to mess with there s very little that ends up going wrong in practice for me ymmv but this comes with a hidden cost if i m using go i use go s dns tcp ip implementation force myself to use e g the go port of sqlite or just use a key value store implemented in go rather than the battle tested leveldb rocksdb or similar when running on the java runtime i avoid jni when possible for the same reason this probably results in me either having worse performance fewer debugging tools and fewer options it s not all negative though reproducibility and portability is much easier if you manage to avoid os installed packages but if i can get the same through nix without limiting myself to the language i use then that would be very nice learning nix i ve sort of learnt nix conceptually through osmosis from all kinds of places and since it is on my agenda to properly learn it i thought i may as well try it now it s a little hard to know where to start so i went to nix pills to learn the stuff bottom up nix pills seems like a great introduction to all the nitty gritty details tips and tricks and all the other things you probably need to know if you want to port existing packages over to nix i on the other hand wanted to make my ocaml builds reproducible that made me a bit antsy after a while while it s certainly nice to know how to compress binaries and exclude gcc from the derivation that s not what i m after by the time i reached nix pill 12 i was tired and skimmed through the rest i felt i had gathered enough information from the earlier pills to navigate flakes as i would likely end up copying someone else s setup and hack on it till it worked that being said my initial impression of nix wasn t as reproducible as i thought it would be consider the example of a working derivation in pill 7 here the author finds the location of bash 4 2 through some repl calls nix repl l nixpkgs added 3950 variables nix repl bash nix store ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6 bash 4 2 p45 when i try to reproduce this behaviour i get bash 5 2 nix repl l nixpkgs added 21422 variables nix repl bash nix store h3bhzvz9ipglcybbcvkxvm4vg9lwvqg4 bash 5 2p26 and from what the pill says this is repl sugar for the expression import nixpkgs in nix i m sure this makes sense if i dig a bit deeper but it s a little weird to me that this is what i m initially greeted with nothing is telling me that nixpkgs is a moving target how can my derivation be reproducible if nix could be pulling the rug under me by bumping my dependencies anyway the point was not to learn nix properly but to get enough understanding that i could move on to learn about nix flakes flakes and unstable ground after looking around for information about nix flakes it seems like the nixos flakes book is the best way to learn about it this sounds great but there are two reasons why i stopped at this point an entire book i just wanted reliable builds for my toy ocaml project flakes is an experimental feature and according to the docs experimental features are considered unstable which means that they can be changed or removed at any time using flakes for reproducibility kinda defeats the purpose then unstable means it s by definition not necessarily reproducible in future versions of nix yeah yeah i know i could just use the current nix version to the heat death of the universe but that s not exactly a viable long term strategy one thing at a time mind you this isn t about nix being hard though i certainly find the documentation spread all over the place and it seems like there is quite a lot of it this is more about me wanting to focus and learn one thing at a time just as i skipped reasonml ocaml on the frontend to begin with as well changing too many variables at the same time is a recipe for disaster nix is still on the agenda for me though it sounds like nixos is a great tool for pet servers so being able to leverage it for small projects where i don t need a kubernetes cluster to manage the cattle seems wonderful but recall my initial pains from the first post i d like to have a reproducible ocaml build system while having the right glibc available will ensure that my build will always compile that s only a tiny part of the problem my main problem is that i have to ensure the right git revisions are used for my dependencies and that isn t something nix nor nix flakes directly help me with as such for development nix would only help me with a tiny issue i ve now managed to resolve and if i were to use nix only to find out i need to use some other library that depends on a later glibc version i d probably be in a lot of pain again so what s the solution then the silver bullet for me seems to be a combination of esy and nix overlays i think i can manage fine with just esy for now so that s what i m aiming for i d also like to note that the build situation is much better now than when i wrote the first blog post i already mentioned that the esy example was fixed and dream is now in sync with its documentation unless you re going to use ppx_rapper all should work out of the box without any pinning pain now there s a caveat when it comes to architectures clearly you ll have to do something to ensure that your x86_64 build will work on say an aarch64 machine there are ways to do that both when it comes to build dependencies but also when it comes to architecture targets from what i understand this is mostly something core package maintainers have to fuzz with though given i ve had my share of fun with glibc i wouldn t be surprised if i would ve been exposed to this as well copyright 2024 jean niklas l orange