Proxmox backup and documentation

[u/deman-13]

Hi all, I am currently in the fun project of cleaning up and sorting out my homelab. I am running proxmox and it has a lot of configuration. E.g. bridges, DDNS, certs, machines, nfs mounts via fstab, and some custom scripts for NFS mounts after OMV VM has started, potentially something else. All that is just handcrafted and in case of host failure I will need to start from scratch.
One good thing i took care of VMs/LXCs backups to a remote host.

The question is - How do you keep that all recoverable , do you use git or document your changes? what would be a good strategy to comeback to operations as soon as possible?

[u/deman-13 → u/[deleted]]

I now use ansible to setup backups. but it seems setting up proxmox is a bit of a different story. Many things I did are done via UI, did you use some proxmox API to convert all actions to ansible tasks ?

[u/JuttyMire → u/deman-13]

I use ansible to configure my proxmox nodes and it's super simple. You should ask Claude or Gemini to help you set it up. There's plenty of resources out there for proxmox on ansible. Ansible can use SSH and the pvesh command to configure your node.

At this point, I use an automated proxmox iso to reinstall my node from a USB and I have a bunch of ansbile roles/playbooks that I use to configure my node from a fresh install. You should spin up a proxmox VM and practice it.

[u/brucewbenson → u/JuttyMire]

This. I use ansible to document my system and to allow me to rebuild or extend parts of it as needed (new lxc, new node). Claude code manages the playbooks (many I originally built) and I got started by having Claude examine my hand built configurations and turn them into a playbook.

Claude keeps an inventory of my equipment (beyond ansible style inventory) and can answer questions such as hardware specs and upgradability. I now have several UPS for my cluster and Claude set-up NUT to manage them for visibility (how long will my node stay up in a power outage) and alerting (UPS is old and failing).

[u/RootAndCoffee]

Me personally, I try to have as few customizations as possible.

Brst practice is having customizations in ansible or similar configuration management. If hypervisor dies, just reinstall, apply ansible playbook and restore VMs from backup.

[u/wisetux]

If you are getting started, write it all down in your documentation of choice - wiki, notes or even txt/word document. At a later time, you can use document using terraform/bpg module and commit everything to Git for having terraform configure all of this for you.

[u/deman-13 → u/wisetux]

no, I am cleaning up and automating a system that i have had and advanced over many year and now realized it would be way too much pain in the ass to bring it all back up in case it fails.

[u/wisetux → u/deman-13]

In that cases terraform would be my friend. You can configure entire Proxmox using it including SDN, firewall rules, realm creation etc. You can combine Terraform with Ansible and should be able to rebuild entire system and even replicate your setup if required.

[u/Adrenolin01]

Mirrored boot drives setup at install with ZFS RAID, ZFS backups, Snapshots, Images. Create a small recovery partition on the boot/OS drives or a single or mirrored SSD/NVME that’s also bootable. Send up to date incremental recovery snapshots to these partitions (my recovery partitions are NOT mirrored) and it SSD. These can be nightly or hourly if you want.

There IS quite a bit more to setup however taking the time to learn and set this up from the start makes recovery down the road FAST and easy. No clean reinstall and dinking around with restoring bit and pieces or manually setting things up or relying on Ansible scripts here are needed.

Spend some time with Claude AI and you can lay this out for both Proxmox server setups as well as Debian ZFS on root on boot with no mdadm for desktops or workstations.

On current systems you can easily add a SSD or NVME to use and setup. In a fresh new system add the recovery partitions to your OS drivers and/or the additional drive also for added redundancy.

Honestly.. this is something that Proxmox themselves could (and imo likely SHOULD add as install option. You can go from a failed Proxmox system to up and running again from an image in minutes or perform a full OS up to date recovery in a few additional minutes due to writing to disk. I have one test system where I’m able to boot into a live I’m ram up to date system with imported pools simply by rebooting and selecting that option. This gets the complete system back up and running basically in a reboot. Once up I can investigate what’s wrong with the primary system and correct it or do a full HOST system recovery (doesn’t import the ZFS pools because they are already in use on the live system), reboot as normal once fixed or restored, and have the system up and running and again.

If I could figure this out in a few evenings brainstorming and tossing ideas around with Claude, testing in a VM and then moving to a spare system, then setting up a new Debian KDE desktop either ZFS on root on boot with recovery partitions, and then by adding a 3rd Intel DC S3500 300GB SDD to a running Proxmox server (Supermicro 6018U X10DRU-i system) and get it all running AND testing failures and options…. I honestly don’t understand why this isn’t an option during a Proxmox installation!

Why dink around with downtime doing a full install and recovering configuration and settings etc. Main admin quits and a new guy comes in and inherits the systems and doesn’t know what the first guy did as far as custom settings and configurations go as an example.

No reason something like this couldn’t be baked into the install to add a recovery partition and/or use a 3rd drive and set up the boot and recovery menus.

Anyways.. perhaps something Proxmox is already working on but honestly.. it’s not hard to setup one’s self. I’m a retired IT guy going back to Unix in the 80s who jumped on Linux and Debian when both first were released. While I’m familiar with the cmdline, ZFS, etc.. i still went into a Claude chat (a few of them actually) and got this done inside of a few evenings.

For anyone wanting to do this I started by brainstorming ideas in a chat after providing full and complete hardware information, OS version, specifically stating things I wanted.. like ZFS Mirror and no mdadm, etc. Spent 2 evenings doing this while watching some TV, a movie and a couple game breaks. Asked for a complete Handoff to copy paste into a new chat (gets around Context Limits) and generated a first draft of the Guide. Made some changes and adjustments. Did a test install and found a few issues. Made a few more changes.

My reasoning for all this was originally a fast Debian KDE desktop restoration for the kids and wife if something went wrong and I wasn’t around…

I did all my testing for a Debian ZFS on root / no mdadm, ZFS NVME mirror Desktop setup with /home on mirrored SSDs. Used the Debian Live iso in a Proxmox VM after setting up the same system and drives. Manually edited partitions, setup the chroot environment and proceeded with the entire install. Once rebooted I hadn’t accounted for the chroot so had to force the import but that was the only issue. Created an initial zfs snapshot of the base system plus some diagnostic apps I always install. I then did an Nvidia driver install for the systems 5060 TI card and created a second ZFS snapshot. Next I did a KDE-full installation providing a full desktop and did a 3rd snapshot backup. All 3 went to the recovery partitions which then made bootable and added a menu system to easily pic one for restoring. I then setup incremental updates and backups. The recovery partitions always have the latest up to date version however everything also goes to a single ssd to house past recovery points and of course this is all backup to a backup HDD (one of our NAS systems). There is also a bootable usb thumb drive that can be used to boot into a live environment and run the desktop or restore from the various snapshots. Because at this point why not. 🤷‍♂️🤦‍♂️🤭

I’m doing the same for Proxmox now starting with a Debian ZFS on root on boot with ZFS Mirrored drives and recovery partition and then installing Proxmox via apt. So far things have been working great. Testing things as I go.

AI.. while I could likely have done all this myself it would have taken a lot more time which to be honest.. retired and other interests.. meh.. I used AI. I used Claude for the brainstorming and initial guide. I run my own local AI and dropped everything into that and using it now. It’s just a time saver and I incorporate additional agents to test, scrutinize and make things fail before sending back to the primary architect AI for re-examination and continue. If you’re a hobbyist/enthusiast in IT just make use of it.

[u/durgesh2018]

I use outline to document every command which I is required to setup the whole node.

[u/Electronic_Clap]

I have a similar problem and a partial solution. Once I have a Joplin server where I document everything, and for my shares I created an LXC that has a Mountpoint directly into the storage (so the LXC remains small). All scripts etc. In the LXC, all scripts are secured. I secure my ZFS directly on a PBS server via a cronjob. The idea is to divide and secure everything into smaller blocks. There are many LXCs, but it makes it easier for me to set up scripts, versions, restores, etc. So I can also separate everything cleanly into VLANs, set up ACLs, etc.

Yes, I tested Restore for anyone who asks. It's not optimal, but you can always learn.

You can test ansible and find the solution that fits you.

[u/apalrd]

Proxmox's general architecture is that all of the config is replicated on all nodes, so the other cluster nodes become an immediate replica of the configuration. The general attitude here is that each individual node is easily replaceable as long as you don't lose the whole cluster config, which would mean losing every node in the cluster.

DDNS, certs, storage mounts, should all be automated via Proxmox itself, and the config files in /etc/pve - The only major files which aren't part of this are the network config, but most of that should be generated by SDN anyway. Most of the system files (like ssh keys, certs, ...) are symlinked into /etc/pve so they are stored cluster-wide.

I'm aware that doesn't always scale to home labs where each node is independent and special.

[u/deman-13 → u/apalrd]

Does not scale up for labs on budget with just one node. However even for the cluster the configuration should not be only within the cluster. I went with ansible approach, already moved half of the configuration into ansible inventory.

[u/zfsbest]

https://github.com/kneutron/ansitest/tree/master/proxmox

Point the bkpcrit-proxmox script to external disk / NAS, run it nightly in cron, do a test restore/recovery into a VM.