Remove glusterfs.

This commit is contained in:
2025-10-25 08:51:29 +01:00
parent bd5988dfbc
commit ccf6154ba0
9 changed files with 19 additions and 72 deletions

View File

@@ -30,10 +30,7 @@ NixOS cluster configuration using flakes. Homelab infrastructure with Nomad/Cons
└── services/ # Nomad job specs (.hcl files) └── services/ # Nomad job specs (.hcl files)
``` ```
## Current Architecture (transitioning) ## Current Architecture
**OLD**: GlusterFS on c1/c2/c3 at `/data/compute` (being phased out)
**NEW**: NFS from zippy at `/data/services` (current target)
### Storage Mounts ### Storage Mounts
- `/data/services` - NFS from `data-services.service.consul` (zippy primary, c1 standby) - `/data/services` - NFS from `data-services.service.consul` (zippy primary, c1 standby)
@@ -86,26 +83,18 @@ NixOS cluster configuration using flakes. Homelab infrastructure with Nomad/Cons
## Migration Status ## Migration Status
**Phase**: 4 in progress (20/35 services migrated) **Phase 3 & 4**: COMPLETE! GlusterFS removed, all services on NFS
**Current**: Migrating services from GlusterFS → NFS **Next**: Convert fractal to NixOS (deferred)
**Next**: Finish migrating remaining services, update host volumes, remove GlusterFS
**Later**: Convert fractal to NixOS (deferred)
See `docs/MIGRATION_TODO.md` for detailed checklist. See `docs/MIGRATION_TODO.md` for detailed checklist.
**IMPORTANT**: When working on migration tasks:
1. Always update `docs/MIGRATION_TODO.md` after completing each service migration
2. Update both the individual service checklist AND the summary counts at the bottom
3. Pattern: `/data/compute/appdata/foo``/data/services/foo` (NOT `/data/services/appdata/foo`!)
4. Migration workflow per service: stop → copy data → edit config → start → update MIGRATION_TODO.md
## Common Tasks ## Common Tasks
**Deploy a host**: `deploy -s '.#hostname'` **Deploy a host**: `deploy -s '.#hostname'`
**Deploy all**: `deploy` **Deploy all**: `deploy`
**Check replication**: `ssh zippy journalctl -u replicate-services-to-c1.service -f` **Check replication**: `ssh zippy journalctl -u replicate-services-to-c1.service -f`
**NFS failover**: See `docs/NFS_FAILOVER.md` **NFS failover**: See `docs/NFS_FAILOVER.md`
**Nomad jobs**: `services/*.hcl` - update paths: `/data/compute/appdata/foo``/data/services/foo` (NOT `/data/services/appdata/foo`!) **Nomad jobs**: `services/*.hcl` - service data stored at `/data/services/<service-name>`
## Troubleshooting Hints ## Troubleshooting Hints

View File

@@ -8,7 +8,6 @@
./unattended-encryption.nix ./unattended-encryption.nix
./cifs-client.nix ./cifs-client.nix
./consul.nix ./consul.nix
./glusterfs-client.nix # Keep during migration, will be removed in Phase 3
./nfs-services-client.nix # New: NFS client for /data/services ./nfs-services-client.nix # New: NFS client for /data/services
]; ];

View File

@@ -1,13 +0,0 @@
{ pkgs, ... }:
{
environment.systemPackages = [ pkgs.glusterfs ];
fileSystems."/data/compute" = {
device = "192.168.1.71:/compute";
fsType = "glusterfs";
options = [
"backup-volfile-servers=192.168.1.72:192.168.1.73"
"_netdev"
];
};
}

View File

@@ -1,24 +0,0 @@
{
pkgs,
config,
lib,
...
}:
{
services.glusterfs = {
enable = true;
};
environment.persistence."/persist".directories = [ "/var/lib/glusterd" ];
# TODO: each volume needs its own port starting at 49152
networking.firewall.allowedTCPPorts = [
24007
24008
24009
49152
49153
49154
49155
];
}

View File

@@ -37,17 +37,17 @@ See [CLUSTER_REVAMP.md](./CLUSTER_REVAMP.md) for detailed procedures.
## Phase 3: Migrate from GlusterFS to NFS ## Phase 3: Migrate from GlusterFS to NFS
- [x] Update all nodes to mount NFS at `/data/services` - [x] Update all nodes to mount NFS at `/data/services`
- [x] Deploy updated configs (NFS client on all nodes) - [x] Deploy updated configs (NFS client on all nodes)
- [ ] Stop all Nomad jobs temporarily - [x] Stop all Nomad jobs temporarily
- [ ] Copy data from GlusterFS to zippy NFS - [x] Copy data from GlusterFS to zippy NFS
- [ ] Copy `/data/compute/appdata/*``/persist/services/appdata/` - [x] Copy `/data/compute/appdata/*``/persist/services/appdata/`
- [ ] Copy `/data/compute/config/*``/persist/services/config/` - [x] Copy `/data/compute/config/*``/persist/services/config/`
- [ ] Copy `/data/sync/wordpress``/persist/services/appdata/wordpress` - [x] Copy `/data/sync/wordpress``/persist/services/appdata/wordpress`
- [ ] Verify data integrity - [x] Verify data integrity
- [ ] Verify NFS mounts working on all nodes - [x] Verify NFS mounts working on all nodes
- [ ] Stop GlusterFS volume - [x] Stop GlusterFS volume
- [ ] Delete GlusterFS volume - [x] Delete GlusterFS volume
- [ ] Remove GlusterFS from NixOS configs - [x] Remove GlusterFS from NixOS configs
- [ ] Remove syncthing wordpress sync configuration - [x] Remove syncthing wordpress sync configuration (no longer used)
## Phase 4: Update and redeploy Nomad jobs ## Phase 4: Update and redeploy Nomad jobs
@@ -125,8 +125,8 @@ See [CLUSTER_REVAMP.md](./CLUSTER_REVAMP.md) for detailed procedures.
- [ ] Verify backups include `/persist/services` data - [ ] Verify backups include `/persist/services` data
- [ ] Verify backups exclude replication snapshots - [ ] Verify backups exclude replication snapshots
- [ ] Update documentation (README.md, architecture diagrams) - [ ] Update documentation (README.md, architecture diagrams)
- [ ] Clean up old GlusterFS data (only after everything verified!) - [x] Clean up old GlusterFS data (only after everything verified!)
- [ ] Remove old glusterfs directories from all nodes - [x] Remove old glusterfs directories from all nodes
## Post-Migration Checklist ## Post-Migration Checklist
- [ ] All 5 servers in quorum (consul members) - [ ] All 5 servers in quorum (consul members)
@@ -143,8 +143,8 @@ See [CLUSTER_REVAMP.md](./CLUSTER_REVAMP.md) for detailed procedures.
--- ---
**Last updated**: 2025-10-23 22:30 **Last updated**: 2025-10-25
**Current phase**: Phase 4 complete! All services migrated to NFS **Current phase**: Phase 3 & 4 complete! GlusterFS removed, all services on NFS
**Note**: Phase 1 (fractal NixOS conversion) deferred until after GlusterFS migration is complete **Note**: Phase 1 (fractal NixOS conversion) deferred until after GlusterFS migration is complete
## Migration Summary ## Migration Summary

View File

@@ -6,7 +6,6 @@
../../common/cluster-member.nix # Consul + storage clients ../../common/cluster-member.nix # Consul + storage clients
../../common/nomad-worker.nix # Nomad client (runs jobs) ../../common/nomad-worker.nix # Nomad client (runs jobs)
../../common/nomad-server.nix # Consul + Nomad server mode ../../common/nomad-server.nix # Consul + Nomad server mode
../../common/glusterfs.nix # GlusterFS server (temp during migration)
../../common/nfs-services-standby.nix # NFS standby for /data/services ../../common/nfs-services-standby.nix # NFS standby for /data/services
# To promote to NFS server (during failover): # To promote to NFS server (during failover):
# 1. Follow procedure in docs/NFS_FAILOVER.md # 1. Follow procedure in docs/NFS_FAILOVER.md

View File

@@ -6,7 +6,6 @@
../../common/cluster-member.nix # Consul + storage clients ../../common/cluster-member.nix # Consul + storage clients
../../common/nomad-worker.nix # Nomad client (runs jobs) ../../common/nomad-worker.nix # Nomad client (runs jobs)
../../common/nomad-server.nix # Consul + Nomad server mode ../../common/nomad-server.nix # Consul + Nomad server mode
../../common/glusterfs.nix # GlusterFS server (temp during migration)
./hardware.nix ./hardware.nix
]; ];

View File

@@ -6,7 +6,6 @@
../../common/cluster-member.nix # Consul + storage clients ../../common/cluster-member.nix # Consul + storage clients
../../common/nomad-worker.nix # Nomad client (runs jobs) ../../common/nomad-worker.nix # Nomad client (runs jobs)
../../common/nomad-server.nix # Consul + Nomad server mode ../../common/nomad-server.nix # Consul + Nomad server mode
../../common/glusterfs.nix # GlusterFS server (temp during migration)
../../common/binary-cache-server.nix ../../common/binary-cache-server.nix
./hardware.nix ./hardware.nix
]; ];

View File

@@ -6,7 +6,6 @@
../../common/cluster-member.nix # Consul + storage clients ../../common/cluster-member.nix # Consul + storage clients
../../common/nomad-worker.nix # Nomad client (runs jobs) ../../common/nomad-worker.nix # Nomad client (runs jobs)
# NOTE: zippy is NOT a server - no nomad-server.nix import # NOTE: zippy is NOT a server - no nomad-server.nix import
../../common/glusterfs.nix # GlusterFS server (temp during migration)
# ../../common/ethereum.nix # ../../common/ethereum.nix
../../common/nfs-services-server.nix # NFS server for /data/services ../../common/nfs-services-server.nix # NFS server for /data/services
# To move NFS server role to another host: # To move NFS server role to another host: