3

I have a mdadm raid 6 file system, which only has 3 disks out of 4 running. I have 4x2tb disks, whenever I add the 4th disk (I tried all week) and do a ls it gives me some file system errors:

$ ll /mnt/downloads/downloads ... d????????? ? ? ? ? ? drivers/ ...

But whenever I remove the newly added disk it shows the file system correctly:

$ sudo mdadm /dev/md0 --fail /dev/sde1 mdadm: set /dev/sde1 faulty in /dev/md0 $ ll /mnt/downloads/downloads (correct contents)

I have tried zeroing the superblock, doing a sudo wipefs -a /dev/sde1 to wipe the raid related blocks, and all have resulted in the same failure.

Checking the mdadm array with just 3 disks shows no errors, by doing echo check > /sys/block/md0/md/sync_action.

I have tried reading all sectors of the disk to see if it will say there's a bad block but nothing of this sort has occured.

I'm running a sudo badblocks -wsv /dev/sde1 on the disk now, but I doubt any errors will show up.

This has left me very confused, is my disk just bad in some way and disk checks just don't work for some reason?

Or is it something related to me not adding the disk correctly? I ran:

sudo mdadm /dev/md0 -a /dev/sde1

I think I always ran this command while the file system was still mounted, and unmounted it during adding the disk, I don't think this would cause an issue would it?

Improve this question

2 Answers 2

Reset to default
4

If the drive itself isn't bad then it's most likely a kernel bug.

For example, there was a RAID6 resync related data corruption bug recently, and depending on which kernel version you are running, you might be affected:

BUG: RAID6 recovery broken by commit 4f4fd7c5798bbdd5a03a60f6269cf1177fbd11ef

Otherwise, also check for RAID inconsistencies ( mdadm --action=check /dev/mdX, watch head /sys/block/md*/md/mismatch_cnt ) which could exist in the remaining parity of the RAID6.

Also check all other angles, memtest, smartctl, etc. as well as dmesg for error messages that might have popped up during resync.

Improve this answer
2
  • 1
    Hmm thanks I'm running version 5.1.4-arch1-1-ARCH, which is affected as stated in your link. Would've never thought that it was a kernel bug and not my own fault. I'm looking to see if 5.1.5.arch1-2, the latest on arch linux has it fixed, I'm not sure how to see that if it is fixed there? Commented May 31, 2019 at 8:24
  • 1
    @simonzack according to ChangeLog-5.1.5 the two patches mentioned in that mailing list thread are included, so it might work... good luck Commented May 31, 2019 at 8:41
0

Spotted the issue while playing with btrfs, and updating vmware, turns out:

Hard disk 0 & Hard disk 4, the physical hard disks I added to my Arch Linux vmware guest, were one and the same.

No wonder it was breaking.

frostschutz's answer might have still mattered, as my kernel version was in the affected versions.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.