flashrd frequently asked

If you need assistance, you may find it here. You may also wish to join the mailing list or read its archive for more information.

What are the minimum system requirements for flashrd?

flashrd boots within 64MB RAM and 1GB flash hosting a default OpenBSD/i386 or Open BSD/amd64 installation. At least 128MB RAM is recommended for the default flashrd setup. To hold more than one version of a system on the flash at any given time, a 4GB or larger flash is recommended. A system with at least 256MB of RAM is recommended for basic network appliance use, and 1GB or more of RAM for conventional workstation, server, or full internet BGP table use. Flashrd does not create a swap space in the default configuration, and some uses can demand significant amount of RAM to be available. Memory based filesystems can also be sized up and demand significant amounts of RAM. For small OpenBSD flash installations (perhaps flash as small as 8MB), see flashboot.

How else can this tool be used?

flashrd allows the disk filesystem to be used in a read-write manner as necessary or convenient. All memory filesystems can be easily eliminated from the flashrd configuration by changing parameters in the etc/rc.flashrd.sub script. The fstab can be used to permanently configure certain partitions as read-write if desired. They must be removed from the rc.flashrd.sub script on the running image if they are not to move between read-write and read-only status (but do not remove the vnddirs from rc.flashrd.sub prior to image creation, as it is used by flashrd itself).

A flashrd image can be used in an application where many copies of the same machine run concurrently. If your servers don't experience frequent changes, the advantage of using the flashrd scheme comes once your primary flashrd upgrade image is created. With just a kernel and filesystem image, upgrades are fast and trivial across your flash-based devices. flashrd images can be created based on any running system, they don't have to be default OpenBSD system releases. (Your setup could be as simple as mounting the filesystem tree you want to duplicate under /mnt.)

What are tardirs? tmpfsdirs? vnddirs?

flashrd boots from a ramdisk kernel with some basic utilities to mount a vnd image with all of your persistent partitions, and then untars the rest of the system on tmpfs. All partitions are defined in /etc/fstab, but are mounted by /stand/rc before the real /etc/rc is run. (The real /etc/rc is actually inside the vnd image, so this has to happen first! The partitions are remounted with the options in /etc/fstab by the real /etc/rc so you only have to change fstab to change how partitions are mounted.)

The layout can be changed (including the size of any partition) by editing variables at the top of the 'flashrd' script.

tardirs are directories that are mounted on pre-sized ramdisks, thus they are not saved upon reboot (unless defined in etc/rc.shutdown tardirs variable). They must be defined in flashrd, etc/rc.conf.local and optionally in etc/rc.shutdown (if you want them saved on each shutdown) (by default, /var is the only member of tardirs)

tmpfsdirs are pre-sized ramdisks, with no content loaded to them. They must be defined in flashrd. (by default, /tmp)

vnddirs are saved to the openbsd.vnd image and mounted read-only. They must be defined in the same order in flashrd, cfgflashrd, stand/rc, bin/ro and bin/rw (by default, /root, /etc, /bin, and /usr are vnddirs). vnddirs can be mounted read-write by default simply by changing /etc/fstab.

What is the default layout?

The default layout, easy to change in the etc/rc.flashrd.sub script, is as follows:
tmpfsdirs="tmp"                         # tmpfs-only dirs
set -A tmpfssize 16M                    # dir sizes (MBytes)

vnddirs="root bin etc sbin usr"
vndsize="50 auto auto auto auto"        # min partition sizes in MBytes (or auto)

tardirs="var"
set -A tarsize 64M                      # tmpfs sizes for tar dirs (MBytes)
This means /root, /bin, /etc, /sbin and /usr are vnddirs. They are mounted read-only during normal operation. The /root directory is fixed at 50MB size. The /bin, /etc, /sbin and /usr directories are "auto" sized. The "auto" size algorithm (in the mkdist script) makes the total size of the vnd partition 150% of the size of the source material for directories over 100MB. Directories under 100MB are sized at 240% of the size of the source material.

The /var directory is a memory filesystem sized at 64MB. This can be resized in the flashrd script, or size can be changed in /etc/fstab on an existing system. It will be saved to a tar file on flash on shutdown. The memory filesystem is populated from flash on boot.

The /tmp directory is a memory filesystem sized at 16MB. This filesystem is not saved at shutdown, nor populated on boot. This can be resized in the flashrd script, or size can be changed in /etc/fstab on an existing system.

Some people want to make /home persistent across boots. The following config would make /home a 512MB on-flash filesystem that is part of the openbsd.vnd file:

export vnddirs="root bin etc sbin usr home"  # must match vnddirs= in stand/rc and fstab
export vndsize="50 auto 50 auto auto 512"   # min partition sizes (or auto) (in MBytes)
The following config would make /home a 512MB memory filesystem, saved to flash as a separate tar file on shutdown and loaded from flash on boot:
export tardirs="var home"            # must match tardirs= in stand/rc and rc.conf.local
set -A tarsize 32 512                # tmpfs sizes for tar dirs (in MBytes)
Finally, you could simply make /home a symlink to /flash/home in /etc/rc.local or some such.
ln -fs /flash/home /home

How flexible is this scheme?

Some future version of the script will allow you to define subdirectories of a vnddir as a separate paritition in vnddirs (such as /usr/src mounted on top of /usr.) For today, /usr can only be a single partition.

USB key? Really?

A USB key is a nice target for flashrd. It is cheap, easily removable and replacable without unracking and disassembling the entire machine. SLC usb keys are also cheap and recommended. Of course, flashrd images are easily upgradeable, as long as you have enough free space on the flash, so you should never need to remove the flash.

If /var is re-created as a memory filesystem on boot, how do I save changes to /var?

If you want to save things like dhcpd.leases on shutdown, you can simply set tardirs=var in /etc/rc.shutdown. If you want to do it manually (such as after you add or remove ports) then do: "tar cf /flash/var.tar -C /var ." In the case of dhcpd.leases, I suggest moving the dhcpd.leases file to the /flash partition (such as dhcpd -l /flash/dhcpd.leases) to ensure persistency.

Can I test flashrd without actually installing it?

Sure, just download an image and fire up the 'flashimg' under qemu, bochs, or another virtual environment.

What are the advantages of flashrd over flashdist?

flashrd has several advantages over non-ramdisk read-only schemes like flashdist and other variations of it. It also has advantages over other ramdisk based systems.

I wrote flashdist in 2002, to load up networking images for an outdoor 802.11 network. The script had a number of shortcomings. It also served a purpose. (You couldn't stick an OpenBSD installation CD into a Soekris 4501. PXE boot wasn't quite there, either. The CHS bootloader was unforgiving. And so on.) It occured to me later that there was still a better way to do it.

How do I install NSH on to a flashrd based system?

flashrd is designed to accommodate NSH without much hassle. NSH is intended to control configuration of most major daemons and functions for networking (routers, firewalls, VPN appliances). This is a major configuration shift and must be handled with care.

The suggested installation procedure:

Unpack and place nsh in /flash
Place nsh in /flash to avoid overwriting during flashrd system upgrade procedure

cp nsh-XXXXXXXX.tar.gz /flash
cd /flash
tar xzf nsh-XXXXXXXX.tar.gz
cd nsh
make LDADD="-ledit -ltermcap -lsqlite3 -static" CFLAGS="-O -DDHCPLEASES=\\\"/flash/dhcpd.leases\\\""

This will force NSH to be built as a static binary, so that you may upgrade the flashrd system without affecting nsh. Also it forces nsh to store dhcpd leases in /flash/dhcpd.leases (instead of /var/db, to preserve lease persistency across flashrd reboots). It is suggested you re-build nsh from a matching system (matching version and architecture) during upgrades. If you can't do this, building NSH as a static binary helps, as there are typically not changes made between OpenBSD versions that will break NSH to the point where you can't remotely login!

Enter read/write mode
rw
Activate save.sh to save NSH configurations to /flash
NSH calls /usr/local/bin/save.sh to permanently store configurations. It does this because you may want to run other commands at the same time that you store the config (such as check the config into a revision control system, or to read/write, read/only state on a partition other than /flash, or whatever else you can imagine.)

sed -e 's/etc/flash/' < save-rw.sh > /usr/local/bin/save.sh
chmod a+x /usr/local/bin/save.sh

Run nsh on startup
echo "/flash/nsh/nsh -i /flash/nshrc" >> /etc/rc.local
Remove manual network configurations
rm /etc/hostname.vr0
rm /etc/mygate
Also be sure to stop any other daemons (that you intend to control via NSH) from starting up before NSH
echo sshd_flags=NO >>/etc/rc.conf.local
echo pf=NO >>/etc/rc.conf.local
echo inetd=NO >>/etc/rc.conf.local
Link nsh to /bin
ln -s /flash/nsh/nsh /bin/nsh
Create your initial NSH configuration
Upon 'write', nsh will automatically grab interface IPs, aliases, and routes from the kernel. NSH looks in:
/var/run/pf.conf.0
/var/run/ospfd.conf.0
/var/run/ospf6d.conf.0
/var/run/bgpd.conf.0
/var/run/ripd.conf.0
/var/run/ldpd.conf.0
/var/run/relayd.conf.0
/var/run/ipsec.conf.0
/var/run/iked.conf.0
/var/run/rtadvd.conf.0
/var/run/dvmrpd.conf.0
/var/run/sasyncd.conf.0
/var/run/dhcpd.conf.0
/var/run/snmpd.conf.0
/var/run/sshd.conf.0
/var/run/ntpd.conf.0
/var/run/npppd.conf.0
/var/run/resolv.conf.0
/var/run/inetd.conf.0
/var/run/smtpd.conf.0
/var/run/ldapd.conf.0
/var/run/ifstated.conf.0
(when you 'write' a new config to disk) to store daemon configurations persistently inside the nshrc.

With this knowledge, if you wanted to control PF, DHCP and the local resolver through NSH, here's what it would look like:

cp /etc/pf.conf /var/run/pf.conf.0
cp /etc/dhcpd.conf /var/run/dhcpd.conf.0
cp /etc/ssh/sshd_config /var/run/sshd.conf.0
mv /etc/resolv.conf /var/run/resolv.conf.0
ln -s /var/run/resolv.conf.symlink /etc/resolv.conf
nsh
nsh/enable
nsh/pf enable
nsh/dhcp enable
nsh/dns local-control
nsh/sshd enable
nsh/write
nsh/quit

Resume read-only mode
ro

Now you must review /flash/nshrc to see if NSH has everything you need. Look for interface IPs being properly configured, routes being in place, BGP, OSPF, DHCP, etc... configuration being in place (and the daemons enabled), the goal here is to boot up with nsh and expect that you can manage all daemons through it!

Now it's time to reboot your flashrd system and watch nsh take over boot-up configurations, to ensure proper installation. If you haven't used nsh before, it is strongly recommended that you are on-site (or have remote serial console access) should nsh not do what you expect it to do!

How do I obtain new image files to upgrade a running system?

You can obtain new image files by building or downloading a new flashrd image, mounting the vnd file, and copying them out of the mounted partition. If your device uses a serial console, make sure you use an image where the openbsd.vnd is configured appropriately first (a pre-configured image from the flashrd web site, or one configured with cfgflashrd). If you put a PC console image on a serial device and reboot, you'll have to SSH in over the network to fix /etc/ttys.

gzip -d flashimg.i386.pccons-20130129.gz
vnconfig vnd0 flashimg.i386.pccons-20130129
mount /dev/vnd0a /mnt
cd /mnt
cp bsd openbsd.vnd var.tar /tmp
cd /tmp
umount /mnt
vnconfig -u vnd0

You could also try running mkkern and mkdist directly to obtain the raw FLASHRD bsd kernel, openbsd.vnd, and var.tar files. mkkern leaves output in a directory named with the format: /tmp/mkkern.XXXXXXXX while mkdist creates openbsd.vnd and var.tar in your current working directory under a new subdirectory vnd-YYYYMMDD.

How do I remotely upgrade a flashrd system with a new OpenBSD image?

On the flash boot image (which is mounted as /flash when booted under flashrd) you have three image files that define the whole system (well, whole system except the boot blocks and bootstrap script):

bsd
openbsd.vnd
var.tar

This is the layout accoring to the default configuration that defines 'var' as handled under 'tardirs'. If you moved var to 'vnddirs' then you would only have a flashrd ramdisk kernel and openbsd.vnd file.

To upgrade, first you need to run 'cfgflashrd' against the flash boot image that you are upgrading to. This is important as cfgflashrd will setup the /etc/boot.conf and /etc/ttys files for serial console use (if you activate com0). Alternately, use a pre-compiled disk image that already has your console type selected.

You can skip running cfgflashrd against the new flash image and instead simply copy /etc/ttys and /etc/boot.conf from the source /etc to the new /etc directory (like you see below for hostname.vr0 and mygate.)

Once you have a properly configured image that reflects the disk type and console type of your destination system, you need to extract the above mentioned image files and upload the new image files to /flash/new, modify the new /etc directory to reflect your needs, move the old files into /flash/old, move the /flash/new files to /flash, and then reboot. This might look something like:

Modify new /etc to include necessary configuration (non-NSH case)
Naturally you want to include any other configuration files that you need for your system.

tar czf /flash/onetime.tgz -C / /etc/hostname.* /etc/mygate /etc/myname /etc/ssh /etc/passwd /etc/master.passwd /etc/pwd.db /etc/spwd.db /etc/pf.conf /etc/sysctl.conf /etc/ntpd.conf /etc/resolv.conf

If you forget to copy a config over, you can later retrieve it from the /etc partition of the vnd image stored in /flash/old/openbsd.vnd

If you include an updated rc.conf.local in onetime.tgz:

Current versions of flashrd have a modified rc which calls /etc/rc.flashrd.conf.

If you include an updated rc.local in onetime.tgz:

The flashrd rc.local calls /etc/rc.flashrd.local which must be preserved as well.

If you include an updated rc.shutdown in onetime.tgz:

Finally, rc.shutdown calls /etc/rc.flashrd.shutdown, which must also be preserved if you want flashrd to archive tardirs.

Any rc.local, rc.conf.local or rc.shutdown file transferred to a new image via onetime.tgz must call these /etc/rc.flashrd scripts for flashrd to function properly. See the rc.conf, rc.conf.local and rc.shutdown scripts included with flashrd as examples.

Modify new /etc to include necessary configuration (NSH case)
Here we assume that you start nsh from /etc/rc.local and that you initially installed NSH according to the directions above.

tar czf /flash/onetime.tgz -C / /etc/rc.local /etc/rc.conf.local /etc/ssh /etc/passwd /etc/master.passwd /etc/pwd.db /etc/spwd.db

Later versions of flashrd have a modified rc.conf.local and rc.local which should be investigated (and upgraded from flashrd) before rebooting. Merge flashrd's latest rc.conf.local and rc.local with your own before saving the result to /flash/onetime.tgz.

Move new FLASHRD kernel/system image in place
cd /flash
mv bsd openbsd.vnd var.tar /flash/old
cd /flash/new
mv bsd openbsd.vnd var.tar /flash
Boot new image
reboot

How does image failover work?

If fsck -y or mount fails to mount any partition within /flash/openbsd.vnd (your default image), the bootstrap script will try to run with the /flash/old/openbsd.vnd image instead. Read /stand/rc to see exactly what happens. (It is fairly simplistic today, and I'm sure it could be improved upon.)

My shell is bash. Your script isn't working properly.

Stop using bash, it's a poor clone of the actual Korn shell (and thinks that it can replace the Korn shell!) Executing flashrd from bash seems to be problematic. OpenBSD's ksh has the features that most people use bash for anyways, such as command line history recall / editing and ksh's programmability (which gets attributed to bash by new comers.) Try it out, you'll find that installing bash is superfluous (and, at least in this case, harmful.)

Flash cards are generally reliable with frequent writes. Wear-leveling is common, and allows millions of writes on high-end flash cards. What's the point of a read-only installation of OpenBSD?

Read-only mounts are a very simple way to avoid potential flash corruption from flash write wear, and other random flash controller problems. As well, read-only use is not a burden on typical appliance-style systems. One objective of flashrd is to create a system that runs reliably for many years, with a minimal chance of failure. Thus, it incorporates an optional read-only approach. Cheap flash can last forever in this configuration. (I am not interested in spending more than $10 for flash on my $100 ALIX system.)

How do I update the local timezone on the flashrd installation?

Copy your preferred time description file to /etc/localtime. Either look in /usr/share/zoneinfo for the right file or just copy /etc/localtime over from a box that already has the right timezone.

How do I install other programs to flashrd?

Pkg tools work as expected under flashrd just like they do on a default OpenBSD install. The process is identical to a normal system, except you have to go to rw mode before installing, and go back to ro mode after. (You actually run the commands 'rw' and 'ro' to do this under flashrd.)