mysql credentials
  * Put secrets/mysql_root_password into a Nomad var named secrets/mysql.root_password

postgres credentials
  * Put secrets/postgres_password into a Nomad var named secrets/postgresql.postgres_password

kopia repository server setup (on a non-NixOS host at the time):
  * kopia repository create filesystem --path /backup/persist
  * kopia repository connect filesystem --path=/backup/persist
  * kopia server user add root@zippy
    then, add the password to secrets/zippy.yaml -- the key needs to be "kopia"
  * kopia server start --address 0.0.0.0:51515 --tls-cert-file ~/kopia-certs/kopia.cert --tls-key-file ~/kopia-certs/kopia.key --tls-generate-cert (first time)
  * kopia server start --address 0.0.0.0:51515 --tls-cert-file ~/kopia-certs/kopia.cert --tls-key-file ~/kopia-certs/kopia.key (subsequent)
[TLS is mandatory for this]

NFS services server setup (one-time on the NFS server host, e.g. zippy):
  * sudo btrfs subvolume create /persist/services
  * sudo mkdir -p /persist/root/.ssh
  * sudo ssh-keygen -t ed25519 -f /persist/root/.ssh/btrfs-replication -N "" -C "root@$(hostname)-replication"
  * Get the public key: sudo cat /persist/root/.ssh/btrfs-replication.pub
    Then add this public key to each standby's nfsServicesStandby.replicationKeys option

NFS services standby setup (one-time on each standby host, e.g. c1):
  * sudo btrfs subvolume create /persist/services-standby

Moving NFS server role between hosts (e.g. from zippy to c1):
  See docs/NFS_FAILOVER.md for detailed procedure
  Summary:
  1. On current primary: create final snapshot and send to new primary
  2. On new primary: promote snapshot to /persist/services
  3. Update configs: remove nfs-services-server.nix from old primary, add to new primary
  4. Update configs: add nfs-services-standby.nix to old primary (with replication keys)
  5. Deploy old primary first (to demote), then new primary (to promote)