October 08, 2013

First of all, many thanks to Jesus Alvarez for providing ZFS packages for Arch Linux.


  • We aren't booting directly from ZFS, as we will have a MD RAID 1 array for the kernel and initramfs, however everything else is on ZFS. I understand it's possible to boot directly from ZFS with some version of GRUB however I've not tried it.
  • Arch Linux isn't for the faint of heart...
  • There are many things that can go wrong along the way during setup.
    • MD seeing old arrays that you have to delete first.
    • ZFS not seeing the drives on reboot because drive letters changed
    • ZFS not seeing the drives on reboot because the disklabel is on too many partitions of the disk
    • Partition kernel cache being a pain
    • zpool labelclear and/or zeroing out a drive are best bets if/when you run into issues
  • Arch ZFS is tied to a specifc Linux Kernel version, so there might be days you can't update to a new kernel unless you build Arch ZFS from source.
  • Same with Arch Linux ISO, tied to specific versions so can't use a previous month ISO once the Arch ZFS repo has been updated.
  • Booting EFI? Well, good luck. I had a hell of a time trying to get it working on a Mac Pro. Much cleaner to use GPT partitions and just boot with MBR into syslinux. Very clean and none of the mess that is GRUB. Tried to get kernel EFI stub booting working but either my hardware, or myself, was lacking. Also, FAT32... really? That's a huge pain to try and keep synced across a multi-disk array.

Add Arch Linux ISO compattible repo

$ cat >> /etc/pacman.conf <<DELIM
Server = http://demizerone.com/\$repo/\$arch
SigLevel = Never

Install Arch ZFS

pacman -Sy archzfs

Partition Disk

$ gdisk -l /dev/sda
GPT fdisk (gdisk) version 0.8.7

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): D9C3DBBE-5C00-4FAD-8708-A80BEA6261FA
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          264191   128.0 MiB   FD00  Linux RAID
   2          264192      1953525134   931.4 GiB   BF00  Solaris root

Copy partition table to other disks

sgdisk -R=/dev/sdx /dev/sda
sgdisk -G /dev/sdx

Create ZFS Pool

4 Drive RAIDZ2

zpool create -f -o ashift=12 -O compression=lz4 -m none -R /mnt rpool raidz2 /dev/sd[bcde]2

5 Drive RAIDZ1

zpool create -f -o ashift=12 -O compression=lz4 -m none -R /mnt backup raidz1 /dev/sd[cdefg]

6 Drive RAIDZ2

zpool create -f -o ashift=12 -O compression=lz4 -m none -R /mnt rpool raidz2 /dev/sd[bcdefg]2


I've had issues lately with zpool creation writing a label to the full drive which prevents it from importing by disk ID. In order to fix it, I have to clear the label and recreate all the partitions on each drive. I'm unsure if this is a bug with ZFS, Arch ZFS, or with how I am creating the pool. Either way, it didn't happen a few months ago but has happened multiple times recently.

zpool labelclear -f /dev/sdx

Create root filesystem and home directory for root

zfs create rpool/ROOT -o mountpoint=none
zfs create rpool/ROOT/arch -o mountpoint=/
zfs create rpool/HOME -o mountpoint=/home
zfs create rpool/HOME/root -o mountpoint=/root

Specify which filesystem is the root filesystem

Despite being named "bootfs", this isn't really the boot filesystem, just the filesystem that we want to use when our kernel/initramfs is loaded. bash zpool set bootfs=rpool/ROOT/arch rpool

Export and re-import to make sure we pick up the by-id changes

zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

Create boot

  • Must use metadata <1.0 with syslinux 4.
  • Use RAID 1 so that we can replace almost all drives even if we forget to update md

4 Drive RAID 1

mdadm --create /dev/md0 --level=1 --raid-devices=4 --metadata=1.0 /dev/sd[abcd]1

5 Drive RAID 1

mdadm --create /dev/md0 --level=1 --raid-devices=5 --metadata=1.0 /dev/sd[abcde]1

6 Drive RAID 1

mdadm --create /dev/md0 --level=1 --raid-devices=6 --metadata=1.0 /dev/sd[abcdef]1

Create and mount boot filesystem

mkfs.ext4 /dev/md0
mkdir -p /mnt/boot
mount /dev/md0 /mnt/boot

Install the Arch ZFS repo used for the actual install

cat >> /etc/pacman.conf <<DELIM
[demz-repo-core]  # For zfs packages
Server = http://demizerone.com/\$repo/\$arch
SigLevel = Never

Install base system

pacstrap -i /mnt base base-devel archzfs syslinux gptfdisk openssh git jshon

Install mdadm config

mdadm --examine --scan > /mnt/etc/mdadm.conf


genfstab -U -p /mnt >> /mnt/etc/fstab

Edit fstab, remove all ZFS file systems leaving only boot

nano /mnt/etc/fstab

chroot into the install

arch-chroot /mnt /bin/bash

Modify mkinitcpio

Add zfs before filesystems and remove fsck

nano /etc/mkinitcpio.conf

Regenerate with mkinitcpio

mkinitcpio -p linux

Install syslinux

/usr/sbin/syslinux-install_update -i -a -m

Configure syslinux

nano /boot/syslinux/syslinux.cfg

Modify APPEND as follows. Maybe you won't need zfs_force=1 however I've found that my pools won't import without it most times.

LABEL arch
  MENU LABEL Arch Linux
  LINUX ../vmlinuz-linux
  APPEND zfs=bootfs zfs_force=1
  INITRD ../initramfs-linux.img

LABEL archfallback
  MENU LABEL Arch Linux Fallback
  LINUX ../vmlinuz-linux
  APPEND zfs=bootfs zfs_force=1
  INITRD ../initramfs-linux-fallback.img

Standard Arch Linux Config

locale | cat > /etc/locale.conf
nano /etc/locale.gen # uncomment your languages in this file
ln -s /usr/share/zoneinfo/UTC /etc/localtime
nano /etc/hostname # set your hostname here
nano /etc/hosts # add your hostname entry here

Make a copy of the ZFS service to mount all the filesystems on boot

cp /usr/lib/systemd/system/zfs.service /etc/systemd/system/
nano /etc/systemd/system/zfs.service # I comment out the import command as it gives me trouble.

Install ZFS snapshot scripts

mkdir -p /root/git-local
git clone https://github.com/zfsonlinux/zfs-auto-snapshot.git /root/git-local/zfs-auto-snapshot
cd /root/git-local/zfs-auto-snapshot && make install

Enable Services

systemctl enable zfs.service
systemctl enable dhcpcd.service
systemctl enable sshd.service
systemctl enable cronie.service

Take first snapshot

zfs snapshot -r rpool@fresh-install

Exit from the chroot, unmount and export

umount /mnt/boot
umount /mnt
zpool export rpool
February 18, 2013

Spent a couple hours trying to figure out how OpenLDAP can be used as an user authentication source for pfSense since I blew away the working configuration I previously had. Whenever I'm about to wipe a machine or delete a VM I always think "is there anything else I need off of here" and without fail, I end up forgetting about some obscure config that I previously spent hours on. Oh well. At least I might as well attempt to document it here in case I do something so foolish again.

What I needed was a guide, but for OpenLDAP -> User Manager instead of a guide for Active Directory -> User Manager. Unfortunately I couldn't find one. It's not like these guides have a ton of configuration info, there are very few settings. So I imagine that most people either know what they are doing, or use Active Directory, or perhaps just have much better luck and memory than I.

In any case, here's how I've got it working.


To be honest, I've not a clue how I setup my OpenLDAP server. It was a mix of various Ubuntu guides applied to an Arch Linux system. And the whole "we store the configuration in LDAP" makes my head hurt. I really hope there are some massive benefits to storing the configuration that way.

Also, I'm using OpenLDAP here instead of just LDAP because there seems to be quite a number of different ways of making things happen in OpenLDAP vs other LDAP implementations.

My user account

dn: uid=jasonrm,ou=People,dc=example,dc=com
cn: Jason McNeil
gidnumber: 10002
givenname: Jason
homedirectory: /home/jasonrm
loginshell: /bin/zsh
objectclass: posixAccount
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: hostObject
ou: cn=Developers,ou=Groups,dc=example,dc=com
ou: cn=Administrators,ou=Groups,dc=example,dc=com
sn: McNeil
uid: jasonrm
uidnumber: 10000

As far as I am aware, how I'm using ou is incorrect. However, that weirdness is exactly what makes this work. This our way of telling pfSense what groups my user is a part of. I do have correct groups elsewhere, but it would appear that pfSense has no clue on how to look those up, and either I don't know how or OpenLDAP lacks the ability to do the memberOf lookup that Active Directory is capable of. It appears that pfSense 2.1 will have the ability to require that the user be part of a specific group via extended query, however that's not what I'm needed, at least, if I understand that feature correctly.


System : User Manager : Group Manager

Create a group with a name that exactly matches the CN of the OU your user(s) will be a part of. In my case I might create the following groups.

  • Administrators
  • Developers

The member count will show 0 on these groups, unless you add a local user to the group. Be sure to add permissions for this group, however leaving the permissions blank won't have any ill effects until after everything is working and you try and login with an LDAP user.

System : User Manager : Servers : LDAP Server Settings

  • Search Scope:
    • Level: Entire Subtree
    • Base DN: [empty]
  • Authentication containers: ou=People,dc=example,dc=com
  • Bind credentials: Use anonymous binds to resolve distinguished names
  • User naming attribute: uid
  • Group naming attribute: cn
  • Group member attribute: ou

You might have to set bind credentials, depends on how you setup your LDAP server. The attributes were the tricky parts for me. When you test the authentication frmo the Diagnostics: Authentication page, if everything except for the attributes are correct you should still see the User: [username] authenticated successfully. message.

Is it working?

From the Diagnostics : Authentication page you should see a message similar to the following when LDAP search and group membership (via our OU abuse) is working.

User: jasonrm authenticated successfully.
This user is a member of these groups: 

In my case, Developers isn't shown as a group I'm a member of because I haven't created that group on the pfSense side despite me having the OU entry.

I hope this is helpful to someone, even if it ends up being myself at a later date after I've blown away yet another test machine...

December 03, 2012

Lots of talk about how jobs of all kinds are being "shipped overseas" over the past few decades ramping up around an election and mostly ignored thereafter. I'm certainly no economist, however I do understand the core principal of any company. Make profit. It's self-interest to make as much profit. Any company that chooses to not maximize profit for the long term will not survive.

Maximizing profit however does not entail any particular strategy. It's not a given that maximizing profit means using the cheapest materials and selling them to a customer at high price. Maybe some will choose that path, but short term gains usually end in the company being acquired by one more stable. Does this always happen? I imagine not.

What happens when I find the best product in a category that aligns with my income? I buy it. Country of origin is rarely, if ever, something I have cared about. I look for the best product to spend my money on, as in investment in that item for the future use of it. I know Apple isn't going to be the cheapest hardware maker, however I've come to trust their design and support. I've experienced PC manufacturer support before and somehow I lived to tell the tales. Sure it's not as cheap, but I allow Apple to maximize their profit and I obtain what I consider to be a superior product. If Apple needs to make in the USA so that a better product is produced, then I'll be willing to pay that premium.

Why do some expect companies to operate differently than a large number of Americans? Most people don't bid out a project and try and find the one that uses mostly USA products. They look for quality of people and materials used. Where it's from doesn't matter so much unless one country of origin is of superior quality than another. It's not about country vs. country. It's about maximizing profit. Aside from earning more at work, how can the average employee maximize profit? By spending their money wisely and finding the optimal product for their needs and budget.

If America wants to produce all the products we've shipping overseas (or to our North and South America neighbors) then America will need to produce a higher level of quality for the additional cost.

Although blatant theft of designs like I've seen a certain audio gear maker produce makes me sick, I understand why some people might buy them. To me I'd rather support the original designers because I want them to produce more gear that I can use in the future. I invest in their future so that when I need something down the road, I know they'll still be around to provide it. Cheap knock-off mills are never going to have the same level of support, it's not what their goal is.

Well, this has been a bit of mixed rambling, but here we are, a slightly technical post instead of me whining about a girl I love. Whoops.

July 17, 2012

Almost made the mistake of going down the "I'm going to build everything" rabbit hole again.

I got to thinking, what if in this crazy new world where I write a lot more I get a lot of traffic from some other site? What if GitHub Pages where this is currently hosted can't handle the traffic? So I started looking at Amazon CloundFront again and started the process of setting up the various pieces. Somehow I managed to stop myself before I wasted more than a few minutes on it. So maybe there is hope for me yet.

July 16, 2012

Why are some companies so retarded?

Your stupid inline "onpaste" override is not going to stop me from pasting my account number or password.

In fact, most people are going to pick eaiser passwords because of that. Sure you don't have to deal with as many people forgetting thier password. And maybe thats just it, it's your way of saying that you don't care about really keeping your customers accounts safe. You just like the illusion.


And I expect it from a company like Capital One, not such much from the Apple Developer login.

July 15, 2012

I'm sick of wasting time on tools instead of getting things done.

Thus, I'm going to try and move as much as possible of my personal projects over to GitHub through the end of the year starting with my website and this blog.

I had been using nanoc and the LESS/CoffeeScript to CSS/JS compiler. It works, however the actual process of updating just feels like a pain. Where as now that I have a Jekyll repo in GitHub Pages I can directly create and edit posts from prose.io with ease. I'm sure there are going to be some issues and leaving behind LESS and CoffeeScript was painful, however I shouldn't be screwing around with my blog much, rather I should be focusing on the real projects that I want to get done.

It wasn't easy, I even started a repo for a new "jason's custom web server and pre-processor and compressor"... it would have been awesome. An awesome waste of time and for what? I've already built all those things, no reason to rebuild it for a website/blog.

Maybe I'm finally maturing.

Not likely, as one thing I did make sure my template supports is a way to quickly make snarky posts about random things or links. Basicly, twitter-light.

Anyhow. I got this all working in a day, where as the previous things took a few, so already my "move everything to GitHub" plan seems to be paying off.