Wow, reading some of the posts makes me feel very grateful for the resources and ability to have a pretty extensive storage and backup infrastructure. I'd like to share how an IT Network/Sys Admin stores/sort of organizes their collection and general data(granted almost 0 JAV, 90% idol, 10% mixed VR). I've been in IT for 15 years, managed several enterprise backup/DR systems, and I treat my home lab/network pretty similarly. I'm a big fan of linux and open source software so I utilize that as much as possible. I also utilize ZFS for all my server file systems! ZFS has the following benefits, checksums with period scrubbing for data integrity (ECC RAM comes into play with this as well), ZFS Snapshots and send/receive for backing up data, on the fly compression with deduplication when doing a ZFS Snapshot. Overall, ZFS is just amazing, and well worth the time/effort to implement
- Server 1
- OS - Proxmox VE (Debian based linux hypervisor)
- Hardware
- AMD Ryzen 7 2700 8c/16t on an Asrock Rack x470-2x10gb motherboard with IPMI
- 64GB DDR4 ECC (error correcting very important for data integrity IMO)
- 2x 240GB SSD ZFS RAID1 for the OS
- 1x 1TB NVME SSD ZFS for VMs (Virtual Machines)
- 6x 4TB NAS 7.2k drives in ZFS RAID10 (main storage for all data)
- LSI-3008i SAS HBA
- Services
- Plex container
- File shares via SAMBA
- Wireguard container
- Unifi Controller
- Several VMs for different purposes (one of them dedicated to video encoding)
- Server 2
- OS - Proxmox Backup Server
- Hardware
- Junk old 4th gen i3 with 16gb ram and cheap mobo
- 2x 240gb ssd in ZFS RAID1 for OS
- 1x500GB SSD ZFS for VM Backup
- 1x 8TB Toshiba X300 ZFS for backing up RAID10 array - will need to upgrade at some point
- Desktop 1
- OS - Windows 10 Pro
- Hardware
- Ryzen 7 5800x - 32GB RAM - RTX 3080
- 1x 1TB NVME - main storage and also used as a scratch drive when doing IO heavy video editing
- 1x 8TB Toshiba - secondary scratch drive, and 3rd tier of backing up RAID10 array
- Services
- Dropbox Pro 2TB for ultra-important documents and certain videos...
My main strategy for storage/backups is this
My main Proxmox server hosts a decently sized 12TB volume shared out with SAMBA (SMB on windows), that I connect to via standard mapped drive on Desktop1. All my downloads go straight to this network share. This is protected against bit-rot and other errors via the ECC memory on the server, as well as ZFS protections. It's also capable of losing up to 3 drives and still have all the data available (I have a drive failure right now that I'm working on replacing). That storage then gets ZFS snapshots sent to the 8TB on Server2 (Proxmox BS). Finally, I do a regular file-level backup periodically and often using Total Commander on my desktop that does a synch between the network share, and my local 8TB HDD.
I've thought a lot, and spent a lot of time configuring this, but it should protect against bitrot, data corruption, even ransomware as long as it's caught quick enough. I'm not sure how many snapshots I can get on the PBS server, but its several weeks worth at least. SSDs are getting cheap enough that I plan on upgrading the 8TB HDD to an 8TB SSD in my main workstation as well.