It's never a good sign when fsck leaves 55 Gbytes worth of files in /lost+found
I guess it doesn't help to have the file system on RAID when the logical file system structures break. Seems mostly to be backups that are affected, though, those should repair themselves on the next backup run. That and a couple of Git repositories that lost some objects and needed to be recloned from checkouts/backups.
@nafmo
Sounds like you might find zfs useful?
@adrinux Is that somehow less resilient to driver bugs/memory corruption/cosmic radiation or whatever happened (I have no idea) than mdraid+ext4?
This is the first time I have had something like this happening after running mdraid+ext4 for well over a decade on multiple machines. I have not tested zfs, so I have no point of comparison.
@nafmo More resilient I'd say, it's designed for data integrity. I've had unrecoverable blocks on disks that damaged data and had zfs just automatically repair that data (this is in a simple two disk mirror).
I've also had memory go bad, had long running disk corruption as a result and zfs apparently still maintained integrity. Though in this latter case it was just backups so I did redo the backups from fresh just in case.
Have switched all three of my mdadm/ext4 pools over now. Yeah, I'm no power user. I haven't gotten into all the fancier zfs stuff like parity disks and snapshots yet.
I really like the zfs automated scrubs and associated alerts (via zed daemon), it feels like it's keeping watch on my data. With mdadm it always felt like I found out something was wrong when i felt other symptoms, like a corrupted file crashing an app. Maybe I could have been using mdadm better? I don't remember it doing anything like that.
@nafmo If you have zfs questions https://discourse.practicalzfs.com/latest is a good place to ask. People far more knowledgeable than me there.
(Was set up as an exodus from r/zfs when Reddit started behaving badly. But tends not to come up in searches yet unless you look for it specifically.)
@adrinux Thanks for the pointers.
I think I will stick with something I know how to work with (ext4 or xfs over mdraid) as I have use that setup extensively both at home and at work. But if I get around to it, I might make me a toy VM to play with it a bit to learn more.