More docs.

docs/MIGRATION_TODO.md

@@ -0,0 +1,153 @@
+# Cluster Revamp Migration TODO
+
+Track migration progress from GlusterFS to NFS-based architecture.
+See [CLUSTER_REVAMP.md](./CLUSTER_REVAMP.md) for detailed procedures.
+
+## Phase 0: Preparation
+- [x] Review cluster revamp plan
+- [ ] Backup everything (kopia snapshots current)
+- [ ] Document current state (nomad jobs, consul services)
+
+## Phase 1: Convert fractal to NixOS (DEFERRED - do after GlusterFS migration)
+- [ ] Document fractal's current ZFS layout
+- [ ] Install NixOS on fractal
+- [ ] Import ZFS pools (double1, double2, double3)
+- [ ] Create fractal NixOS configuration
+- [ ] Configure Samba server for media/shared/homes
+- [ ] Configure Kopia backup server
+- [ ] Deploy and verify fractal base config
+- [ ] Join fractal to cluster (5-server quorum)
+- [ ] Update all cluster configs for 5-server quorum
+- [ ] Verify fractal fully operational
+
+## Phase 2: Setup zippy storage layer
+- [x] Create btrfs subvolume `/persist/services` on zippy
+- [x] Configure NFS server on zippy (nfs-services-server.nix)
+- [x] Configure Consul service registration for NFS
+- [x] Setup btrfs replication to c1 (incremental, 5min intervals)
+- [x] Fix replication script to handle SSH command restrictions
+- [x] Setup standby storage on c1 (`/persist/services-standby`)
+- [x] Configure c1 as standby (nfs-services-standby.nix)
+- [x] Configure Kopia to exclude replication snapshots
+- [x] Deploy and verify NFS server on zippy
+- [x] Verify replication working to c1
+- [ ] Setup standby storage on c2 (if desired)
+- [ ] Configure replication to c2 (if desired)
+
+## Phase 3: Migrate from GlusterFS to NFS
+- [x] Update all nodes to mount NFS at `/data/services`
+- [x] Deploy updated configs (NFS client on all nodes)
+- [ ] Stop all Nomad jobs temporarily
+- [ ] Copy data from GlusterFS to zippy NFS
+  - [ ] Copy `/data/compute/appdata/*` → `/persist/services/appdata/`
+  - [ ] Copy `/data/compute/config/*` → `/persist/services/config/`
+  - [ ] Copy `/data/sync/wordpress` → `/persist/services/appdata/wordpress`
+  - [ ] Verify data integrity
+- [ ] Verify NFS mounts working on all nodes
+- [ ] Stop GlusterFS volume
+- [ ] Delete GlusterFS volume
+- [ ] Remove GlusterFS from NixOS configs
+- [ ] Remove syncthing wordpress sync configuration
+
+## Phase 4: Update and redeploy Nomad jobs
+
+### Core Infrastructure (CRITICAL)
+- [x] mysql.hcl - moved to zippy, using `/data/services`
+- [ ] postgres.hcl - update paths, add affinity for zippy
+- [ ] redis.hcl - update paths, add affinity for zippy
+- [ ] traefik.hcl - update paths (already floating)
+- [ ] authentik.hcl - verify (stateless, no changes needed)
+
+### Monitoring Stack (HIGH)
+- [ ] prometheus.hcl - update paths
+- [ ] grafana.hcl - update paths
+- [ ] loki.hcl - update paths
+- [ ] vector.hcl - remove glusterfs log collection
+
+### Databases (HIGH)
+- [ ] clickhouse.hcl - update paths, add affinity for zippy
+- [ ] unifi.hcl - update paths (includes mongodb)
+
+### Web Applications (HIGH-MEDIUM)
+- [ ] wordpress.hcl - update from `/data/sync/wordpress` to `/data/services/appdata/wordpress`
+- [ ] gitea.hcl - update paths
+- [ ] wiki.hcl - update paths, verify with exec driver
+- [ ] plausible.hcl - verify (stateless)
+
+### Web Applications (LOW, may be deprecated)
+- [ ] ghost.hcl - update paths or remove (no longer used?)
+- [ ] vikunja.hcl - update paths or remove (no longer used?)
+- [ ] leantime.hcl - update paths or remove (no longer used?)
+
+### Network Infrastructure (HIGH)
+- [ ] unifi.hcl - update paths (already listed above)
+
+### Media Stack (MEDIUM)
+- [ ] media.hcl - update paths, add constraint for fractal
+  - [ ] radarr, sonarr, bazarr, plex, qbittorrent
+
+### Utility Services (MEDIUM-LOW)
+- [ ] evcc.hcl - update paths
+- [ ] weewx.hcl - update paths
+- [ ] code-server.hcl - update paths
+- [ ] beancount.hcl - update paths
+- [ ] adminer.hcl - verify (stateless)
+- [ ] maps.hcl - update paths
+- [ ] netbox.hcl - update paths
+- [ ] farmos.hcl - update paths
+- [ ] urbit.hcl - update paths
+- [ ] webodm.hcl - update paths
+- [ ] velutrack.hcl - verify paths
+- [ ] resol-gateway.hcl - verify paths
+- [ ] igsync.hcl - update paths
+- [ ] jupyter.hcl - verify paths
+- [ ] whoami.hcl - verify (stateless test service)
+- [ ] tiddlywiki.hcl - update paths (if separate from wiki.hcl)
+
+### Backup Jobs (HIGH)
+- [x] mysql-backup - moved to zippy, verified
+- [ ] postgres-backup.hcl - verify destination
+- [ ] wordpress-backup.hcl - verify destination
+
+### Verification
+- [ ] All services healthy in Nomad
+- [ ] All services registered in Consul
+- [ ] Traefik routes working
+- [ ] Database jobs running on zippy (verify via nomad alloc status)
+- [ ] Media jobs running on fractal (verify via nomad alloc status)
+
+## Phase 5: Convert sunny to NixOS (OPTIONAL - can defer)
+- [ ] Document current sunny setup (ethereum containers/VMs)
+- [ ] Backup ethereum data
+- [ ] Install NixOS on sunny
+- [ ] Restore ethereum data to `/persist/ethereum`
+- [ ] Create sunny container-based config (besu, lighthouse, rocketpool)
+- [ ] Deploy and verify ethereum stack
+- [ ] Monitor sync status and validation
+
+## Phase 6: Verification and cleanup
+- [ ] Test NFS failover procedure (zippy → c1)
+- [ ] Verify backups include `/persist/services` data
+- [ ] Verify backups exclude replication snapshots
+- [ ] Update documentation (README.md, architecture diagrams)
+- [ ] Clean up old GlusterFS data (only after everything verified!)
+- [ ] Remove old glusterfs directories from all nodes
+
+## Post-Migration Checklist
+- [ ] All 5 servers in quorum (consul members)
+- [ ] NFS mounts working on all nodes
+- [ ] Btrfs replication running (check systemd timers on zippy)
+- [ ] Critical services up (mysql, postgres, redis, traefik, authentik)
+- [ ] Monitoring working (prometheus, grafana, loki)
+- [ ] Media stack on fractal
+- [ ] Database jobs on zippy
+- [ ] Consul DNS working (dig @localhost -p 8600 data-services.service.consul)
+- [ ] Backups running (kopia snapshots include /persist/services)
+- [ ] GlusterFS removed (no processes, volumes deleted)
+- [ ] Documentation updated
+
+---
+
+**Last updated**: 2025-10-22
+**Current phase**: Phase 2 complete (zippy storage setup done), ready for Phase 3 (GlusterFS → NFS migration)
+**Note**: Phase 1 (fractal NixOS conversion) deferred until after GlusterFS migration is complete