Hi all,
I’ve been running a bunch of services in docker containers using Docker Compose for a while now, with data storage on ZRAID mirrored NVME and/or ZRAID2 HDDs.
I’ve been thinking about moving from my single server setup to three micro-servers (Intel N150s), both for redundancy, learning, and fun.
Choosing Kubernetes was easy, but I’d like to get some outside opinions on storage. Some examples of how I’m using storage:
- Media and large data storage: Currently on the ZRAID2 HDDs, will stay here but be migrated to a dedicated NAS
- High IO workloads like Postgresql and email: Currently running on the NVMEs
- General low-volume storage: Also currently on NVMEs, but different use case. These are lower IO, like data storage for Nextcloud, Immich, etc
I’m a huge fan of being able to snapshot with ZFS, as I mirror all my data off-site with hourly pushes for some container data, and daily for the rest. I’d like to be able to continue this kind of block-level backups if possible.
Assume I’m a noob at Kubernetes storage (have been reading, but still fresh to me). I’d love to know how others would set up their storage interfaces for this.
I’m trying to understand if there’s a way to have the storage “RAIDed” across the drives in the three micro-servers, or if things work differently than I expect. Thanks!
For starting out I would suggest Longhorn, I’ve had several issues with it but maybe they’ve been fixed. I also used Rook Ceph which is overkill but neat to play with. Techno Tim has several videos on Kubernetes including setting up a cluster (he has an ansible playbook for it) as well as how to set up Longhorn.
That would be block storage like glusterfs or ceph, or object storage like minio or rook.
You could also use ZFS to provide PVCs for your Pods, with openebs.
If the mini-servers don’t have hardware redundancy, I’d stick to Replicated Volumes only…
If you go the openebs+ZFS route, you can make a kubernetes service (DaemonSet because it should run on every node) that makes and sends/exposes ZFS snapshots.
I’d recommend Ceph (in the form of Rook ) if you’re willing to put in the time to learn it. For a simpler solution, check out Longhorn. Ceph is more mature, and Rook is just a solution that almost fully automates its deployment on kubernetes, while Longhorn is built from scratch as a kubernetes native storage solution. The people who built Longhorn (Rancher Labs) also make a FOSS kubernetes management service called Rancher, so if you prefer a more intuitive web UI for K8s, be sure to check that out too
Rook is the 2nd most used container storage solution I’ve encountered or set up at my job, with legacy storage appliances like IBM FlashSystem and NetApp being the first
Glusterfs?
It’ll let you turn the storage on each server into a single volume and you can set raid-like mirroring/striping.
