alo-cluster/README.md

alo-cluster NixOS Configuration

This repository contains the NixOS configuration for a distributed cluster of machines managed as a unified flake.

Architecture Overview

The configuration uses a layered profile system that enables code reuse while maintaining clear separation of concerns:

minimal-node          # Base system (SSH, users, boot, impermanence)
    ↓
cluster-node          # Cluster services (Consul, GlusterFS, CIFS, encryption)
    ↓
server-node           # Server workloads (future: MySQL, PostgreSQL)
    ↓
workstation-node      # Development tools (Docker, deploy-rs, emulation)
    ↓
desktop-node          # GUI environment (Hyprland, Pipewire, fonts)

Each layer extends the previous one, inheriting all configurations. Hosts select a profile level that matches their role.

Special Node Types

• cloud-node: Minimal + Consul only (cloud VPS deployments)
• compute-node: Cluster + Nomad worker (container orchestration)

Directory Structure

.
├── flake.nix                 # Main flake definition with all hosts
├── common/
│   ├── global/               # Global configs applied to all systems
│   │   ├── console.nix       # Linux console colors (Solarized Dark)
│   │   ├── locale.nix        # Timezone and locale settings
│   │   └── nix.nix           # Nix daemon and flake configuration
│   ├── minimal-node.nix      # Base layer: SSH, users, boot, impermanence
│   ├── cluster-node.nix      # Cluster layer: Consul, GlusterFS, CIFS
│   ├── server-node.nix       # Server layer: bare metal services (future)
│   ├── workstation-node.nix  # Workstation layer: dev tools
│   ├── desktop-node.nix      # Desktop layer: GUI environment
│   ├── cloud-node.nix        # Cloud VPS profile
│   ├── compute-node.nix      # Nomad worker profile
│   └── [feature modules]     # Individual feature configs
├── hosts/
│   ├── c1/                   # Compute node 1
│   ├── c2/                   # Compute node 2
│   ├── c3/                   # Compute node 3
│   ├── alo-cloud-1/          # Cloud VPS
│   ├── chilly/               # Server node
│   ├── zippy/                # Workstation node
│   └── sparky/               # Desktop node
├── home/
│   ├── default.nix           # Home-manager entry point
│   ├── profiles/             # Per-profile package sets
│   │   ├── server.nix
│   │   ├── workstation.nix
│   │   └── desktop.nix
│   ├── programs/             # Per-profile program configurations
│   │   ├── server.nix        # CLI tools (fish, tmux, git, nixvim)
│   │   ├── workstation.nix   # + dev tools
│   │   └── desktop.nix       # + Hyprland, wofi
│   └── common/               # Shared home-manager configs
└── services/                 # Nomad job definitions (not NixOS)

Profile System

System Profiles

Profiles are automatically applied based on the mkHost call in flake.nix:

# Example: Desktop profile includes all layers up to desktop-node
mkHost "x86_64-linux" "desktop" [
  ./hosts/sparky
];

Available profiles:

• "server" → minimal + cluster + server
• "workstation" → minimal + cluster + server + workstation
• "desktop" → minimal + cluster + server + workstation + desktop

Home-Manager Profiles

Home-manager automatically inherits the same profile as the system, configured in home/default.nix:

imports = [ ./programs/${profile}.nix ];
home.packages = profilePkgs.${profile};

This ensures system and user configurations stay synchronized.

Host Definitions

Current Hosts

Host	Profile	Role	Hardware
c1, c2, c3	compute-node	Nomad workers	Bare metal servers
alo-cloud-1	cloud-node	Reverse proxy	Cloud VPS
chilly	server	Home Assistant in a VM	Bare metal server
zippy	workstation	Development machine, server	Bare metal server
sparky	desktop	Desktop environment	Bare metal desktop

Adding a New Host

1. Create host directory:

   mkdir -p hosts/newhost
   

2. Create hosts/newhost/default.nix:

   { config, pkgs, ... }:
   {
     imports = [
       ../../common/encrypted-btrfs-layout.nix  # or your layout
       ../../common/global
       ./hardware.nix
     ];
   
     networking.hostName = "newhost";
     # Host-specific configs here
   }
   

3. Generate hardware config:

   nixos-generate-config --show-hardware-config > hosts/newhost/hardware.nix
   

4. Add to flake.nix:

   newhost = mkHost "x86_64-linux" "workstation" [
     ./hosts/newhost
   ];
   

Deployment

Using deploy-rs

Deploy to specific host:

deploy -s '.#sparky'

Deploy to all hosts:

deploy

Deploy with detailed logging:

deploy -s '.#sparky' -- --show-trace

Manual Deployment

nixos-rebuild switch --flake .#sparky --target-host sparky

Key Features

Impermanence

All hosts use tmpfs root with selective persistence. Persistent paths configured per-host in persistence.directories and persistence.files.

Unattended Encryption

Cluster nodes support automatic unlocking via Tailscale network using common/unattended-encryption.nix.

Cluster Services

• Consul: Service discovery and distributed KV store
• GlusterFS: Distributed filesystem client
• CIFS/Samba: Network file sharing

Desktop Environment (sparky only)

• Hyprland: Wayland compositor with CapsLock→Super remapping
• wofi: Application launcher (Super+D)
• foot: Terminal emulator (Super+Q)
• greetd/tuigreet: Login manager with console option

Development Tools (workstation/desktop)

• Docker with rootless mode
• deploy-rs for NixOS deployments
• ARM emulation via binfmt
• Full NixVim configuration

Future Work

• Migrate Nomad services (MySQL, PostgreSQL) to bare NixOS services under server-node.nix
• Add monitoring stack (Prometheus, Grafana)
• Document Tailscale key rotation process
• Add automated testing for configuration changes