alo-cluster/docs/DIFF_CONFIGS.md

# Configuration Diff Tool

Tool to compare all NixOS host configurations between current working tree and HEAD commit.

## Purpose

Before committing changes (especially refactors), verify that you haven't accidentally broken existing host configurations. This tool:
- Builds all host configurations in current state (with uncommitted changes)
- Builds all host configurations at HEAD (last commit)
- Uses `nvd` to show readable diffs for each host
- Highlights which hosts changed and which didn't

## Usage

### Prerequisites

The script requires `nvd` to be in PATH. Use either:

**Option 1: direnv (recommended)**
```bash
# Allow direnv in the repository (one-time setup)
direnv allow

# direnv will automatically load the dev shell when you cd into the directory
cd /home/ppetru/projects/alo-cluster
# nvd is now in PATH
```

**Option 2: nix develop**
```bash
# Enter dev shell manually
nix develop

# Now run the script
./scripts/diff-configs.sh
```

### Quick Start

```bash
# Compare all hosts (summary)
./scripts/diff-configs.sh

# Compare with detailed path listing
./scripts/diff-configs.sh -v c1

# Compare with content diffs of changed files (deep mode)
./scripts/diff-configs.sh --deep c1

# Compare only x86_64 hosts (avoid slow ARM cross-compilation)
./scripts/diff-configs.sh c1 c2 c3 zippy chilly sparky

# Verbose mode with multiple hosts
./scripts/diff-configs.sh --verbose c1 c2 c3

# Via flake app
nix run .#diff-configs

# Show help
./scripts/diff-configs.sh --help
```

### Typical Workflow

```bash
# 1. Make changes to configurations
vim common/impermanence.nix

# 2. Stage changes (required for flake to see them)
git add common/impermanence.nix

# 3. Check what would change if you committed now
# For quick feedback, compare only x86_64 hosts first:
./scripts/diff-configs.sh c1 c2 c3 zippy chilly sparky

# 4. Review output, make adjustments if needed

# 5. If changes look good and affect ARM hosts, check those too:
./scripts/diff-configs.sh stinky alo-cloud-1

# 6. Commit when satisfied
git commit -m "Refactor impermanence config"
```

## Output Explanation

### No Changes
```
━━━ c1 ━━━
  Building current... done
  Building HEAD...    done
  ✓ No changes
```
This host's configuration is identical between current and HEAD.

### Changes Detected
```
━━━ stinky ━━━
  Building current... done
  Building HEAD...    done
  ⚠ Configuration changed

<<< /nix/store/abc-nixos-system-stinky-25.05 (HEAD)
>>> /nix/store/xyz-nixos-system-stinky-25.05 (current)

Version changes:
[C]  octoprint: 1.9.3 -> 1.10.0
[A+] libcamera: ∅ -> 0.1.0
Closure size: 1500 -> 1520 (5 paths added, 2 paths removed, +3, +15.2 MB)
```

Legend:
- `[C]` - Changed package version
- `[A+]` - Added package
- `[R-]` - Removed package
- `[U.]` - Updated (same version, rebuilt)

### Verbose Mode (--verbose)

With `-v` or `--verbose`, also shows the actual store paths that changed:

```
━━━ c1 ━━━
  Building current... done
  Building HEAD...    done
  ⚠ Configuration changed

[nvd summary as above]

  Changed store paths:
    Removed (17 paths):
    - config.fish
    - system-units
    - home-manager-generation
    - etc-fuse.conf
    ... and 13 more

    Added (17 paths):
    - config.fish
    - system-units
    - home-manager-generation
    - etc-fuse.conf
    ... and 13 more
```

This is useful when nvd shows "No version changes" but paths still changed (e.g., refactors that rebuild config files).

### Deep Mode (--deep)

With `-d` or `--deep`, shows actual content diffs of changed files within store paths (implies verbose):

```
━━━ c1 ━━━
  Building current... done
  Building HEAD...    done
  ⚠ Configuration changed

[nvd summary and path listing as above]

  Content diffs of changed files:

    ▸ etc-fuse.conf
      @@ -1,2 +1,2 @@
      -user_allow_other
      +#user_allow_other
       mount_max = 1000

    ▸ nixos-system-c1-25.05
      activate:
        @@ -108,7 +108,7 @@
         echo "setting up /etc..."
        -/nix/store/...-perl/bin/perl /nix/store/...-setup-etc.pl /nix/store/abc-etc/etc
        +/nix/store/...-perl/bin/perl /nix/store/...-setup-etc.pl /nix/store/xyz-etc/etc

    ▸ unit-dbus.service
      dbus.service:
        @@ -1,5 +1,5 @@
         [Service]
        +Environment="LD_LIBRARY_PATH=/nix/store/.../systemd/lib"
         Environment="LOCALE_ARCHIVE=..."
```

**What it shows**:
- Matches changed paths by basename (e.g., both have "config.fish")
- Diffs important files: activate scripts, etc/*, *.conf, *.fish, *.service, *.nix
- Shows unified diff format (lines added/removed)
- Limits to first 50 lines per file

**When to use**:
- When you need to know **what exactly changed** in config files
- Debugging unexpected configuration changes
- Reviewing refactors that don't change package versions
- Understanding why a host rebuilt despite "No version changes"

### Build Failures
```
━━━ broken-host ━━━
  Building current... FAILED
  Error: attribute 'foo' missing
```
If a host fails to build, the error is shown and the script continues with other hosts.

## How It Works

1. **Discovers hosts**: Queries `deploy.nodes` from flake to get all configured hosts
2. **Creates worktree**: Uses `git worktree` to check out HEAD in a temporary directory
3. **Builds configurations**: Builds `config.system.build.toplevel` for each host in both locations
4. **Compares with nvd**: Runs `nvd diff` to show package-level changes
5. **Cleans up**: Removes temporary worktree automatically

## Important Notes

### Git Staging Required

Flakes only evaluate files that are tracked by git. To make changes visible:
```bash
# Stage new files
git add new-file.nix

# Stage changes to existing files
git add modified-file.nix

# Or stage everything
git add .
```

Unstaged changes to tracked files **are** visible (flake uses working tree content).

### Performance

- First run may be slow (building all configurations)
- Subsequent runs benefit from Nix evaluation cache
- Typical runtime: 1-5 minutes depending on changes
- **ARM cross-compilation is slow**: Use host filtering to avoid building ARM hosts when not needed
  - Example: `./scripts/diff-configs.sh c1 c2 c3` (x86_64 only, fast)
  - vs `./scripts/diff-configs.sh` (includes stinky/alo-cloud-1, slow)

### When to Use

**Good use cases**:
- Refactoring shared modules (like impermanence)
- Updating common configurations
- Before committing significant changes
- Verifying deploy target consistency

**Not needed for**:
- Adding a single new host
- Trivial one-host changes
- Documentation updates

## Troubleshooting

### "Not in a git repository"
```bash
cd /home/ppetru/projects/alo-cluster
./scripts/diff-configs.sh
```

### "No changes detected"
All changes are already committed. Stage some changes first:
```bash
git add .
```

### Build failures for all hosts
Check flake syntax:
```bash
nix flake check
```

### nvd not found
Install nvd:
```bash
nix profile install nixpkgs#nvd
```
(Already included in workstation-node.nix packages)

## Related Tools

- `nvd` - Package diff tool (used internally)
- `nix diff-closures` - Low-level closure diff
- `nix store diff-closures` - Alternative diff command
- `deploy-rs` - Actual deployment tool

## See Also

- `common/global/show-changelog.nix` - Shows changes during system activation
- `docs/RASPBERRY_PI_SD_IMAGE.md` - SD image building process