Jan '09 26

Lets Blog

I’ve finally decided to start a Blog, a collection of notes to myself.
BF4 Stats YDraig's Gamercard
Click to Visit Geelong Weather.com

Aug '14 27

Move ZFS filesystem between pools

I’m migrating my data around between my different raid groups (Mirror and RaidZ1). I Found a good post on how to do it with zfs send/receive and snapshots.

  • Use Pipe Viewer (pv) to monitor progress
  • Create a snapshot on source use -r for child subsystems
  • zfs send -R / receive -F  (may need -d  to remove first node name on destination)
  • Remove any snapshots and reconfigure Samba and mountpoints.
  • check zfs set has the correct settings

Create snapshot and copy to destination
zfs snapshot -r sourcepool@moving
zfs send -R sourcepool@moving | pv | zfs receive -F destpool
# OR
zfs send -R sourcepool@moving | pv | zfs receive -dF destpool

Stop any locks on system (Samba / NFS / FTP etc)
zfs snapshot -r sourcepool@moving2
zfs send -Ri sourcepool@moving sourcepool@moving2 | pv | zfs receive -F destpool
# OR
zfs send -Ri sourcepool@moving sourcepool@moving2 | pv | zfs receive -dF destpool

Jul '14 15

ZFS on Linux

I’ve been looking at ZFS on Linux for some time now, mainly due to silent corruption on my EXT4 file-system. One of the big benefits for me is data/meta checksums, on data corruption is fetched from parity, and a data scrub picks up large numbers of errors. One of implementation problems is that people recommend ECC memory. This is due to the chance that bad memory may corrupt the disk when doing a scrub of the data and the Checksums fail to match.

Another feature that I hope to use is Autogrow, by replacing the existing disks with larger disks one-by-one, then resilvering the data across all the disks, when the last one is replaced, the new space becomes available to the pool. While its a lot to get your head around, ZFS (now owned by Oracle) is very similar to WAFL on NetApp. So that makes it far easier for me to grasp.

  • Use /dev/disk/by-id/ata-* convention when adding to the zpool
  • Use Whole Disk and ashift=12 for 4k blocks
  • Use disk quantity’s of power of 2 plus parity
  • set parameters such as autoexpand/autoreplace on before they are needed
  • create subvolumes on the pool for space and or quota management
  • unmount any previous mountpoint before doing set mountpoint (its immediate)
  • Align Partitions on 4k boundaries ie: python ((314574848*512)%4096) = 0 remainder

For my new setup I plan to use 2 x 2TB, partitioned as follows

  • Part1 – 100G – linux mirror (md0)
    • / (root/boot) – 30G ext4
    • /var  – 80G riserfs
    • /home/media/Videos/PchMenu – 10G riserfs
    • swap – 20G swap
  • Part2 – Remainder – ZFS mirror
    • rpool/home
    • rpool/home/photos
    • rpool/home/camera
    • rpool/music

For my media I plan to use 3x4TB in raidz1 as follows

  • Whole of disk
    • storage
    • storage/media
    • storage/backups

zpool create -o ashift=12 backup /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK2334PBHE5JHR

zpool create -o ashift=12 rpool mirror sda sdb
zpool set autoexpand=on rpool
zfs create rpool/home
zfs create rpool/home/photos
zfs create rpool/home/camera
zfs create rpool/music
zfs set canmount=noauto rpool
zfs set atime=off rpool
zfs set atime=on rpool/home
zfs set relatime=on rpool/home
zfs set atime=off rpool/home/photos
zfs set atime=off rpool/home/camera
zfs set mountpoint=/home rpool/home
zfs set quota=10G rpool/home/camera
#zfs set copies=2 rpool/home/photos

zpool create -o ashift=12 storage raidz sdc sdd sde
zpool set autoexpand=on storage
zfs create storage/media
zfs create storage/backups
zfs create storage/backups/dad
zfs create storage/backups/dean
zfs set canmount=noauto storage
zfs set canmount=noauto storage/backups
zfs set atime=off storage
#zpool set autoreplace=on storage
#zfs set sync=disabled storage
#zfs set compression=on documents

You can put swap onto zfs, not sure if its wise though

#sudo zfs create tank/swap -V 2G -b 4K
#sudo mkswap -f /dev/tank/swap
#sudo swapon /dev/tank/swap

Feb '14 20

Top Talkers

Use tcpdump to determine the top talkers on an interface (not conversations)

tcpdump -tnn -c 5000 -i eth1 | awk -F “.” ‘{print $1″.”$2″.”$3″.”$4}’ | sort | uniq -c | sort -nr | awk ‘ $1 > 100 ‘

Sep '13 09

iptraf for Monitoring Bandwidth

I’ve used iptraf for years, but never been able to get the filter to work properly to exclude the ssh sessions. I found a good post about filtering ssh on iptraf, and when used with ip traffic on all interfaces it shows traffic through the Linux box.

  • Create a new filter, add a new entry and set the dest port to “22 to 0 ” and Y for TCP, E for Exclude and Y to match opposite, Enter to save
  • Add a second entry with ALL IP protocols set to Y and I for Include, Enter to save
  • Press X to save the filter (CTRL+X Cancels!)
  • Apply the Filter.

Also a few other tools for checking interfaces and Bandwidth

  • bmon
  • slurm
  • iftop -nNbP -i ppp0
  • tcptrack -i ppp0
  • pktstat -n -i ppp0
  • vnstat
  • bwm-ng
  • cbm
  • speedometer
  • sktstat
  • netwatch
Jul '13 14

Video Editing

I was having problems with Vegas Pro 12.0 today (as usual). The render would just randomly stop. After reading a few forum and blogs, looks like the settings to change are:

  • Max Threads = max cores (8)
  • GPU accel = OFF
  • Dynamic Ram, I’ve set this to 1000, but others report 0 is better
  • Disabled display fames in video preview with external display

I got so frustrated I was looking around at other programs, and have downloaded trials for Adobe Premiere Elements 11 and Magix Movie Edit Pro Plus, while Cyberlink Power Director is also interesting.

May '13 13

Serial via Network Bridge

I had a requirement for a Serial Bridge, eg: serial->Network->serial, I found a nice page with some options,  socat to be good enough for what I needed.

Server (Listening Side connected to end device)

socat -v -d -d tcp-l:54321,reuseaddr,fork file:/dev/ttyS0,nonblock,raw,echo=0,waitlock=/tmp/ttyS0.lock

Client (connecting to server)

socat -v -d -d tcp: file:/dev/ttyS0,nonblock,raw,echo=0,lockfile=/tmp/ttyS0.lock

Alternative using Virtual PTY and SSH remote exec

socat PTY,link=$HOME/dev/vmodem0,raw,echo=0,wait-slave EXEC:'"ssh modemserver.us.org socat - /dev/ttyS0,nonblock,raw,echo=0"'
Nov '12 08

BSOD analyze

I always forget how to get the driver info out of windbg

!analyze -v;r;kv;lmtn;lmtsmn;.bugcheck;!sysinfo cpuinfo;!sysinfo machineid; !sysinfo cpuspeed; !sysinfo smbios .logclose

If your getting errors about symbols, use !symfix remember to restart windbg afterwards.

Oct '12 03

Verify Linux MD device

Yet another faulty disk with bad blocks has me wondering how to check the MD array for consistancy, found a good post about it.

sudo bash -c "echo check >  /sys/block/md0/md/sync_action"
sudo bash -c "echo repair >/sys/block/md0/md/sync_action"
cat /sys/block/md1/md/mismatch_cnt