Mine would be when I had a 48 bay disk array / JBOD fail on me… badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.
These were all running bcache on top of mdraid… One single mount.
Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.
I just HAD to solve this one though.
In come several spare chassis / mobo etc… get a bunch of drives powered and on /dev/ , move to the next.
A couple spare gigabit switches…
several gigabit NICs…
two explicit paths for each machine…
a bit of iSCSI magic, and one machine now had the physical disks all exposed to it… mdadm --assemble blah blah, bit of UUID chaos…
It’s surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.
Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of “I can lose this and don’t care” vs “this box gets RAID10 and/or offsite backups nightly”.
back up your critical stuff people!
Power isn’t so bad really. These were all 2tb disks, so ended up with about 40tb total after shenanigans.
Cabling was atrocious. I’d originally been using several pulled 4 disk SAS backplanes from servers thrown out of my work, just screwed down to some pine boards in a stack.
Zip ties and sata/sas cables were the rule.
Overall the drives had 4 small power supplies initially, 300watt iirc. Cheap.
Somewhere I have an old photo from the wiring during the “rescue” but I have to look around.
Disk failure was actually not an issue for ages before. A specific thunderstorm with 2 nearby tornadoes is what caught me this time. Before that, the occasional single disk every 6 months or so.
I still have running arrays using 15 year old disks, many with more than 10 years of that powered on. If you find good models, they can last ages.