Skip to content

February 8, 2012

33

ZFS: FreeBSD ZFS root install with bsdinstall

As anyone who’s tried out the new FreeBSD 9.0-RELEASE will know,  with it came the new installer, bsdinstall. Love it or hate it, it’s there, but it does have some pretty cool features that were not available on the old sysinstall. One of the features a lot of people were hoping for with the release of bsdinstall was the ability to do a full zfs root install of FreeBSD from the installer. This did not make the release and I can understand why, dialog is cool but creating a menu system for zfs with it is a nightmare. I am working on a zfs installer myself for a separate project and believe me, it will give you a head ache to think about it with all the features to write in.

The installer (pc-bsdinstall?) from the PC-BSD project is a full X GUI and already has a full ZFS install option, I believe the plan is to use the backend functionality from this installer to implement the features in bsdinstall.

The usual method for installing a full zfs root before FreeBSD 9.0-RELEASE was to use the very cool (as featured in bsdmag) mfsbsd which can be found at http://mfsbsd.vx.sk, included in the special addition (SE) of mfsbsd was a zfs install script, which is the inspiration for the commands used here. Since the release of FreeBSD 9.0 and the addition of the ‘Live CD’ option on the ISO some howtos have cropped up showing how to setup full zfs root from the bsdinstall live cd. Sounds good at first, but no real advantage over mfsbsd, in fact you have not zfsinstall script so it’s more of a pain, also you will have to do *all* of the FreeBSD install config by hand once extracted.

So until we get a ZFS root install option in bsdinstall, here is a way to do it by dropping out to shell at certain points of the bsdinstall and create a zpool to install to. See the video above to see how its done.

You may notice close to the end the installer jumps back to the first screen and goes to ‘Live CD’. This was a bad edit on my part, normally at this point you would be asked if you want to enter shell to do any final changes. You should chose No at this point and the next dialog box will give you the option to reboot or enter ‘Live CD’. Enter the ‘Live CD’ at this point to follow along with the video.



  • Frederique

    This tutorial is very nice. Works like a charm.

    What about a raidz1 pool?

    • http://www.mebsd.com/ Jake Smith

      This method will work with any zpool configuration you can think of, as long as you have enough disks :)

      Once you have all your disks setup with gpart just change the zpool create line, e.g. zpool create -f -m none -o altroot=/mnt -o cachefile=/tmp/zpool.cache tank raidz gpt/disk00 gpt/disk01 gpt/disk02 gpt/disk03

      The rest of the setup is the same, recently I used this method to install full zfs on a raidz3 pool.

  • http://www.youtube.com/profile?user=backless88 backless88

    [..YouTube..] Hi guys thank a lots for your share.That’s very simple,quickly and clean.
    Some question for swap disk size.Your setting is 1/5 size for swap disk to use ZFS mirror.
    I am not sure that’s correct or not.If you have enough memory,maybe your swap disk could be down to 1/10 or 1/20. However we know ZFS designers recommend 2 GB of RAM for every 1 TB of storage,
    Anyhow thank your show again,and now I try to setting ZFS raid z3 Root from bsdinstall.
    Maybe this time will success.^_^
    regard,
    Senmao Chang.

  • backless88

    Hi guys thank a lots for your share.That’s very simple,quickly and clean.
    Some question for swap disk size.Your setting is 1/5 size for swap disk to use ZFS mirror.
    I am not sure that’s correct or not.If you have enough memory,maybe your swap disk could be down to 1/10 or 1/20. However we know ZFS designers recommend 2 GB of RAM for every 1 TB of storage,
    Anyhow thank your show again,and now I try to setting ZFS raid z3 Root from bsdinstall.
    Maybe this time will success.^_^
    regard,
    Senmao Chang.

  • mebsd

    Hi, yeah the amount of swap I used here might not be accurate to a production environment. The rule of thumb is normally to make the swap partition at least the total size of RAM. This is so you can dump the entire memory into a crash file if you have a kernel panic.

    2GB of RAM for every 1TB of storage is a fair estimate for a well performing system. You could get away with 1GB for each 1TB. It depends a lot on your setup, if you use dedup you will find you need a lot of memory.

  • gkontos

    Nice tutorial, it would be better to use gpart add -b 34 -s 94 -t freebsd-boot ada(x) in order for your drives to be 4K aligned.

  • Roby Sadeli

    Hi Jake!
    A really nice tutorial.
    A question: If one of the disks in a zfs mirrored tank fails, what should be done? Is it as simple as replacing the disk and do a “zpool replace” or you have to re-create the boot, swap and data partition by hand?
    Thanks.

    • http://www.mebsd.com/ Jake Smith

      Hi Roby, thanks for visiting :)

      Once you have physically replaced the failed disk, you will need to perform the gpart commands on the new disk before you can replace the disk in the zpool. Unfortunately zfs can not do the partitions for you, this is true even on Solaris.
      N.B. The replacement disk must be the same size or bigger than the disk that has failed, zfs will not allow you to replace a drive with a smaller one. If you don’t remember the gpart setting you originally used you can issue the gpart show command against one of the working disks.

      Also, have a look at the hot spare option in zfs, FreeBSD can not auto replace the disk for you yet, (awaiting a zfs daemon). However you can gpart a replacement disk ready and reserve it in the zpool. Then when you have a drive fail it would be as simple as just issuing a zpool replace. Check out https://blogs.oracle.com/eschrock/entry/zfs_hot_spares

  • OrlandoPCRepair

    The best FreeBSB tutorial videos on the web are available @ my channel OrlandoPCRepair

  • Formatierer

    Hey dude, I tried like 5 written tutorials and ran in like 5 different problems! Then I watched your video and managed to crack this bitch in like 30 minutes! Thx a lot man!!! I just have a few questions for better understanding:

    1. What did you do with that dd command at 1:15? I left it out but ran into problems later so it must be important. But what does it mean?
    2. Why is it so important to export the tank and reimport it at 4:45?

  • Formatierer

    Would be very kind if you would take a few minutes to answer these questions.

    Again Thx alot!

  • mebsd

    Hey, glad this video has helped you out! In response to your questions:

    1. The dd command ensures any old ZFS meta data is cleared from the partition.

    2. Exporting and reimporting is very important, when you export a ZFS pool any out standing writes are synced to the disks. It basically forces ZFS to process the ZIL.

    Hope this answers your questions.

  • Vadim Ippolitov

    Hi! Great video. I’m wondering now if it’s possible to use the real MBR-style partitions (named slices in FreeBSD) as the basis for my zpool. As a matter of fact, I already have a zpool (created in Linux), and now I just want to install FreeBSD on it. Does anyone know if that’s possible without moving around all the data already on the pool? P.S. I am still new to FreeBSD.

  • Shahriyar Guliyev

    Thanks you mate, keep up good work!

  • ale0122

    2:44 $ exit

    then the installer continues :)

  • ale0122

    excelent video thanks!!! :D

  • Kapil Patil

    Excellent Video.. Works for ME :)

  • asdf90091

    Nice presentation! Any idea if this will work on a raidz2 array of flash drives?

  • Blackgentoo

    thanks a lot! your guide worked perfectly for me!

  • ptyll123

    I see your movie and I raise you one mfsbsd.vx.sk – thanks for playing.

  • mebsd

    Yes mfsbsd is a cool project, and with its zfsinstall script, probably the best one to go for if you’re just starting out with ZFS root on FreeBSD, or maybe PC-BSD as this has a GUI installer for ZFS.

    However what this movie is demonstrating is how to do it using the standard FreeBSD installer, bsdinstall. Which hopfully one day will have a ZFS root install option in the dialog menu. After all mfsbsd doesn’t just exist to help you install FreeBSD,

  • mebsd

    You may also be interested to see the blog post I made this video for, mebsd.com/configure-freebsd-servers/video-howto-freebsd-full-zfs-root-from-bsdinstall.html – I talk a little about mfsbsd on there also.

  • Carlos Porter
  • Curt Dox
  • RWY32L

    Very simple and easy tutorial. Thank you very much.

  • abw

    Thanks for the tutorial it was most helpful. I found the video a little too fast to follow so I took the effort to transcribe the commands with the timings in the video. My minor mods at 0:15-16, Hopefully this may be of help to someone else… please feel free to edit.

    # FreeBSD full ZFS Root from bsdinstall
    # Video Source: http://www.youtube.com/watch?v=9HKz77Ka_Xo
    # Website : https://mebsd.com/configure-freebsd-servers/video-howto-freebsd-full-zfs-root-from-bsdinstall.html
    #
    # Time in video given in minutes and seconds m:ss

    0:00. Boot from CD (not boot only version!)
    0:01. Install
    0:02. Keymap Selection – (No) Yes
    0:04. United Kingdom
    0:06. Set Hostname – freebsd-zfs # use FQDN! pc.lan
    0:10. Distribution Select – Remove all components – games lib32 ports
    0:11. Partitioning – select Shell option
    0:12. dmesg | grep ada # show both hdd’s: ada0 ada1
    0:15. gpart destroy -F ada0 # added by me – destroy existing format
    0:16. gpart destroy -F ada1 # added by me – destroy existing format
    0:18. gpart create -s gpt ada0
    0:20. gpart create -s gpt ada1
    0:21. gpart add -b 34 -s 94 -t freebsd-boot ada0 # use for 4K alignment
    0:29. gpart add -b 34 -s 94 -t freebsd-boot ada1 # use for 4K alignment
    0:32. gpart add -t freebsd-swap -s 4g ada0
    0:42. gpart add -t freebsd-swap -s 4g ada1
    0:46. gpart add -t freebsd-zfs -l disk00 ada0
    0:52. gpart add -t freebsd-zfs -l disk01 ada1
    0:56. gpart show
    1:00. dd if=/dev/zero of=/dev/ada0p3 count=560 bs=512 # ensure old ZFS meta data is cleared
    1:10. dd if=/dev/zero of=/dev/ada1p3 count=560 bs=512 # ensure old ZFS meta data is cleared
    1:14. gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 # make partiton bootable
    1:26. gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 # make partiton bootable
    1:28. zpool create -f -m none -o altroot=/mnt -o cachefile=/tmp/zpool.cache tank mirror gpt/disk00 gpt/disk01
    1:50. zpool status
    1:54. zfs create -o mountpoint=/ tank/root
    2:02. zfs create -o mountpoint=/usr tank/usr
    2:06. zfs create -o mountpoint=/var tank/var
    2:10. zfs create -o mountpoint=/tmp tank/tmp
    2:16. zfs create -o mountpoint=/usr/home tank/usr/home
    2:20. zfs list
    2:25. zpool set bootfs=tank/root tank
    2:35. zpool get all tank
    2:44. exit # Return to FreeBSD Installer GUI
    2:45. Checksum Verification – base.txz and kernel.txz
    2:48. Archive Extraction – base.txz and kernel.txz
    3:00. Set the root password
    3:02. Network Configuration – em0
    3:04. Network configuration – IPv4 – DHCP Yes
    3:08. Network Configuration IPv6 – No
    3:10. Network Configuration – Resolver Configuration – take default. (FQDN of this network!)
    3:11. Select UTC or local for CMOS clock – No
    3:12. Time Zone Selector – 8 Europe
    3:14. Countries in Europe – 49 United Kingdom – GMT, Yes
    3:16. System Configuration – ssh, ntpd
    3:17. Dumpdev Configuration – Yes (Use no – zfs can’t save crash dumps!)
    3:19. Add users
    3:36. Final Configuration – Exit; Apply configuration and exit installer
    3:40. Manual Configuration – Open a shell to make final manual modifications? – No
    3:40. ABORT!!!!! is this a bad video edit? <======== Should be –
    – Complete – (Installation of BSD complete) choose "Live CD"

    3:40. Welcome – (Start of freeBSD Installer!) "Live CD"
    3:44. Log in as root # Note system has the name 'freebsd-zfs'
    3:48. df
    3:49. vi /mnt/etc/fstab
    3:52. add the lines (new file):-
    /dev/ada0p2 none swap sw 0 0
    /dev/ada1p2 none swap sw 0 0 #{for mirror}
    4:09. vi /mnt/etc/rc.conf
    4:10. add the line:-
    zfs_enable="YES"
    4:20. vi /mnt/boot/loader.conf
    4:24. add the lines (new file):-
    zfs_load="YES"
    vfs.root.mountfrom="zfs:tank/root" # note: vfs NOT zfs
    4:38. pwd # Should be /root
    4:40. zpool export tank
    4:48. zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache tank
    5:08. cp /tmp/zpool.cache /mnt/boot/zfs/
    5:12. zpool get all tank
    5:20. reboot # System should be come up as full zfs
    5:24. END

  • Marcio Cruz

    excelent !
    works on FreeBSD 9.2

  • Reginald Dancy

    Great video! Very helpful!

  • Alex Xela

    Сам пока zfs не занимался. Видео понравилось, быстро и красиво. Молодец.
    Было бы не плохо, если бы оформили статью на эту темы. Я бы почитал.

  • Richard Kearsley

    Thank you for the guide, it’s very helpful
    Could you explain about the cachefile? I would really like to understand
    what’s happening :)

  • Reginald Dancy

    Once again great video! That said I ran into an issue since I had to rebuild my system. I had to use zpool export -f zroot because the device “/mnt” was busy. Afterwards when I tried to import, I got the following message “more than one matching pool, import by ID instead. Problem is I don’t the matching pool. My zpool doesn’t appear when I typed zpool status. Any ideas?