Brins mad mumblings

Create Loop devices

cd /tmp
truncate -s 1G L{1,2,3}.img
losetup --show -f L1.img
losetup --show -f L2.img
losetup --show -f L3.img

Create the Array

mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/loop{0,1}
mkfs.ext4 /dev/md0

Add 3^rd Disk as Spare

mdadm --add /dev/md0 /dev/loop2
mdadm --detail /dev/md0

Convert Raid1 to Raid5 (reshape).

mdadm --grow /dev/md0 --level=5 --raid-devices=3
cat /proc/mdstat

Grow Filesystem to suit. Notice the Array and Used size changes after conversion.

mdadm --detail /dev/md0
resize2fs /dev/md0

Cleanup Loopback interfaces / md devices

mdadm --stop /dev/md0
losetup -d /dev/loop{0,1,2}
rm -fv /tmp/L{1,2,3}.img

Error happens when /boot/grub/i386-pc/ is empty

• grub rescue> ls
• grub rescue> set root=(md/0)
• grub rescue> set prefix=(md/0)/usr/lib/grub
• grub rescue> insmod normal
• grub rescue> normal

• grub> set root=(md/0)
• grub> linux /boot/vmlinuz-4.19.0-20-amd64 root=/dev/md0
• grub> initrd /boot/initrd.img-4.19.0-20-amd64
• grub> boot

• blockdev –flushbufs /dev/*
• grub-install –no-floppy /dev/md0 –force
• grub-install –no-floppy /dev/sd[abc]
• grub-update

How to resolve grub error: file ‘/grub/i386-pc/normal.mod’ not found

cat /proc/mdstat
sudo fdisk -l /dev/sdb
sudo fdisk /dev/sda
sudo fdisk -l /dev/sda
sudo mdadm --manage /dev/md0 --remove /dev/sda1
sudo mdadm --manage /dev/md0 --add /dev/sda1
sudo mdadm --detail /dev/md0
cat /proc/mdstat
sudo grub-install /dev/sda

sudo zdb
ls -l /dev/disk/by-id/
sudo zpool status
sudo zpool offline rpool /dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F31K832N-part2
sudo zpool replace rpool /dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F31K832N-part2 /dev/disk/by-id/ata-Hitachi_HDS722020ALA330_JK11A8B9HZAPMF-part2
sudo zpool status

I’m migrating my data around between my different raid groups (Mirror and RaidZ1). I Found a good post on how to do it with zfs send/receive and snapshots.

• Use Pipe Viewer (pv) to monitor progress
• Create a snapshot on source use -r for child subsystems
• zfs send -R / receive -F  (may need -d  to remove first node name on destination)
• Remove any snapshots and reconfigure Samba and mountpoints.
• check zfs set has the correct settings

Create snapshot and copy to destination
zfs snapshot -r sourcepool@moving
zfs send -R sourcepool@moving | pv | zfs receive -F destpool
# OR
zfs send -R sourcepool@moving | pv | zfs receive -dF destpool


Stop any locks on system (Samba / NFS / FTP etc)
zfs snapshot -r sourcepool@moving2
zfs send -Ri sourcepool@moving sourcepool@moving2 | pv | zfs receive -F destpool
# OR
zfs send -Ri sourcepool@moving sourcepool@moving2 | pv | zfs receive -dF destpool


I’ve been looking at ZFS on Linux for some time now, mainly due to silent corruption on my EXT4 file-system. One of the big benefits for me is data/meta checksums, on data corruption is fetched from parity, and a data scrub picks up large numbers of errors. One of implementation problems is that people recommend ECC memory. This is due to the chance that bad memory may corrupt the disk when doing a scrub of the data and the Checksums fail to match.

Another feature that I hope to use is Autogrow, by replacing the existing disks with larger disks one-by-one, then resilvering the data across all the disks, when the last one is replaced, the new space becomes available to the pool. While its a lot to get your head around, ZFS (now owned by Oracle) is very similar to WAFL on NetApp. So that makes it far easier for me to grasp.

• Use /dev/disk/by-id/ata-* convention when adding to the zpool
• Use Whole Disk and ashift=12 for 4k blocks
• Use disk quantity’s of power of 2 plus parity
• set parameters such as autoexpand/autoreplace on before they are needed
• create subvolumes on the pool for space and or quota management
• unmount any previous mountpoint before doing set mountpoint (its immediate)
• Align Partitions on 4k boundaries ie: python ((314574848*512)%4096) = 0 remainder

For my new setup I plan to use 2 x 2TB, partitioned as follows

• Part1 – 100G – linux mirror (md0)
  • / (root/boot) – 30G ext4
  • /var  – 80G riserfs
  • /home/media/Videos/PchMenu – 10G riserfs
  • swap – 20G swap
• Part2 – Remainder – ZFS mirror
  • rpool/home
  • rpool/home/photos
  • rpool/home/camera
  • rpool/music

For my media I plan to use 3x4TB in raidz1 as follows

• Whole of disk
  • storage
  • storage/media
  • storage/backups

zpool create -o ashift=12 backup /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK2334PBHE5JHR

zpool create -o ashift=12 rpool mirror sda sdb
zpool set autoexpand=on rpool
zfs create rpool/home
zfs create rpool/home/photos
zfs create rpool/home/camera
zfs create rpool/music
zfs set canmount=noauto rpool
zfs set atime=off rpool
zfs set atime=on rpool/home
zfs set relatime=on rpool/home
zfs set atime=off rpool/home/photos
zfs set atime=off rpool/home/camera
zfs set mountpoint=/home rpool/home
zfs set quota=10G rpool/home/camera
#zfs set copies=2 rpool/home/photos

zpool create -o ashift=12 storage raidz sdc sdd sde
zpool set autoexpand=on storage
zfs create storage/media
zfs create storage/backups
zfs create storage/backups/dad
zfs create storage/backups/dean
zfs set canmount=noauto storage
zfs set canmount=noauto storage/backups
zfs set atime=off storage
#zpool set autoreplace=on storage
#zfs set sync=disabled storage
#zfs set compression=on documents

You can put swap onto zfs, not sure if its wise though

#sudo zfs create tank/swap -V 2G -b 4K
#sudo mkswap -f /dev/tank/swap
#sudo swapon /dev/tank/swap

I’ve used iptraf for years, but never been able to get the filter to work properly to exclude the ssh sessions. I found a good post about filtering ssh on iptraf, and when used with ip traffic on all interfaces it shows traffic through the Linux box.

• Create a new filter, add a new entry and set the dest port to “22 to 0 ” and Y for TCP, E for Exclude and Y to match opposite, Enter to save
• Add a second entry with ALL IP protocols set to Y and I for Include, Enter to save
• Press X to save the filter (CTRL+X Cancels!)
• Apply the Filter.

Also a few other tools for checking interfaces and Bandwidth

• bmon
• slurm
• iftop -nNbP -i ppp0
• tcptrack -i ppp0
• pktstat -n -i ppp0
• vnstat
• bwm-ng
• cbm
• speedometer
• sktstat
• netwatch