Dear Café,
In the nice bar mood setting, I thought I would share my impressions when pondering changing from Linux-originated container technology to FreeBSD jails. Please treat what I say here as you would treat some fellow meandering about his stories about his great errands - with a bit of interest, and a bit of reservation. In the end, you can never know how much of it is true or accurate, and how much of it is just made up or self-believed misconception.
Reading up forums and hearing people, Container vs Jails seems to be about technical merits or security stance. Won't comment on those (I believe in the wake of modern discoveries these would need a lot deeper and nuanced analysis, than could be hand-waved away as easily done), but highlight a more selfish divide: portability and reproducibility of configuration.
You see, I worked at places, and often heard "cattle not pets". Means you should not grow to like and treat a given system with utmost devotion (like you would a pet), but rather treat it as dispensable, replaceable (like... ok.. maybe the anglophone metaphor breaks down a bit, since you would still not be that crude to cattle, but anyway). In retrospect, one driving force behind this might be the many kinds of Linux systems - each shipping a different version of (spare me for uttering the name) systemd, having slightly different conventions of location or structuring of configuration files, naming and packaging utilities up slightly differently etc. So you incline not to grow too attached with a flavor of Linux, and try to abstract.
(Truth be told, the luxury of abstraction is more the privilege of the application developers.. the more you are an infra developer or sysadmin kind, the less you can distance yourself from the pits of configuration and system specifics)
Thus, Containers (packaging up a userland into a neat layered tarball) are used not because we (app developers with some sysadmin blood flowing in the veins too) couldn't configure a system to run some service directly. It is because we don't want to, because the next day on an other distro it would work and break very slightly differently. We don't want to care, we want uniformity.
So we package the userland, and expose some minimal surface to configure (ah, the diversification there too.. ENV vars, command-line flags... or, if you really need it, mounting some specific config files into the running container), and thus run our little services.
Jails... a very interesting topic. Giving you all the isolation you want, yet not sparing from the usual burdens of configuring. But, this is where the uniformity of FreeBSD might be a savior. If you get those etc files right once, you can keep reusing them forever (hey, anyway, I said mounting some config files into a container every now and then fine, so setting up some config files to a jail must fly as well).
(Just between us, there's still a gap between "can run this service with passing some flags to the container config" and "need to create jail, install some packages inside, copy some config files inside". For a linux app dev, the former sounds reproducible/minimal. The latter gives an Ansible vibe, which is tolerable, but never raises fond memories. Aside, with Podman coming to FreeBSD, maybe this gap would close as well)
Then there's the lingering self-doubt. If it is a jail, will I be able to resist to apply a one-off fix inside, that I forget to record in whatever manual or automation if I want to set a jail up the next time? The userland of running containers is immutable - well there's the mutable overlay, but when you kill and restart the service, you expect it to just run with only the immutable layers reconstructed.
An other factor might be backups. Don't think it is standard expectation that in event of failure, there would be meaningful backups of the running system state. Not of the system you treat as cattle, anyway. So your best bet is to have all these minimal specifications of what containers you run with what parameters, and better they work if you need to spin them up on a quickly provisioned new node!
But, if you have a great backup culture, and can restore jail contents quickly and at will (and can bridge that time over somehow for your service), then maaaaybe you don't need to have all cattle. Maybe you can have pets. Maybe it is ok to fix things in-jail, since you have an immutable historic backup record you can compare against.
The promise of FreeBSD with jails is that we can again be the sysadmins and small kings of our own kingdoms. The reality of running services as Containers is the demanding expectation to herd a cattle without mercy. And here we are, torn between words.
Thank you for your attention, now let's get an other beer.
