If you need assistance, you may find it here. You may also wish to join the mailing list or read its archive for more information.
flashrd boots within 64MB RAM and 1GB flash hosting a default OpenBSD/i386 or Open BSD/amd64 installation. At least 128MB RAM is recommended for the default flashrd setup. To hold more than one version of a system on the flash at any given time, a 4GB or larger flash is recommended. A system with at least 256MB of RAM is recommended for basic network appliance use, and 1GB or more of RAM for conventional workstation, server, or full internet BGP table use. Flashrd does not create a swap space in the default configuration, and some uses can demand significant amount of RAM to be available. Memory based filesystems can also be sized up and demand significant amounts of RAM. For small OpenBSD flash installations (perhaps flash as small as 8MB), see flashboot.
flashrd allows the disk filesystem to be used in a read-write manner as necessary or convenient. All memory filesystems can be easily eliminated from the flashrd configuration by changing parameters in the etc/rc.flashrd.sub script. The fstab can be used to permanently configure certain partitions as read-write if desired. They must be removed from the rc.flashrd.sub script on the running image if they are not to move between read-write and read-only status (but do not remove the vnddirs from rc.flashrd.sub prior to image creation, as it is used by flashrd itself).
A flashrd image can be used in an application where many copies of the same machine run concurrently. If your servers don't experience frequent changes, the advantage of using the flashrd scheme comes once your primary flashrd upgrade image is created. With just a kernel and filesystem image, upgrades are fast and trivial across your flash-based devices. flashrd images can be created based on any running system, they don't have to be default OpenBSD system releases. (Your setup could be as simple as mounting the filesystem tree you want to duplicate under /mnt.)
flashrd boots from a ramdisk kernel with some basic utilities to mount a vnd image with all of your persistent partitions, and then untars the rest of the system on mfs. All partitions are defined in /etc/fstab, but are mounted by /stand/rc before the real /etc/rc is run. (The real /etc/rc is actually inside the vnd image, so this has to happen first! The partitions are remounted with the options in /etc/fstab by the real /etc/rc so you only have to change fstab to change how partitions are mounted.)
The layout can be changed (including the size of any partition) by editing variables at the top of the 'flashrd' script.
tardirs are directories that are mounted on pre-sized ramdisks, thus they are not saved upon reboot (unless defined in etc/rc.shutdown tardirs variable). They must be defined in flashrd, etc/rc.conf.local and optionally in etc/rc.shutdown (if you want them saved on each shutdown) (by default, /var is the only member of tardirs)
mfsdirs are pre-sized ramdisks, with no content loaded to them. They must be defined in flashrd. (by default, /tmp)
vnddirs are saved to the openbsd.vnd image and mounted read-only. They must be defined in the same order in flashrd, cfgflashrd, stand/rc, bin/ro and bin/rw (by default, /root, /etc, /bin, and /usr are vnddirs). vnddirs can be mounted read-write by default simply by changing /etc/fstab.
The default layout, easy to change in the etc/rc.flashrd.sub script, is as follows:
export vnddirs="root bin etc sbin usr" # must match vnddirs= in stand/rc and fstab export vndsize="102400 auto auto auto auto" # min partition sizes (or auto) (in 512 byte sectors) export tardirs="var" # must match tardirs= in stand/rc and rc.conf.local set -A tarsize 131072 # mfs sizes for tar dirs (in 512 byte sectors) export mfsdirs="tmp" # mfs-only dirs set -A mfssize 32768 # mfs sizes (in 512 byte sectors)This means /root, /bin, /etc, /sbin and /usr are vnddirs. They are mounted read-only during normal operation. The /root directory is fixed at 50MB size. The /bin, /etc, /sbin and /usr directories are "auto" sized. The "auto" size algorithm (in the mkdist script) makes the total size of the vnd partition 150% of the size of the source material for directories over 100MB. Directories under 100MB are sized at 240% of the size of the source material.
The /var directory is a memory filesystem sized at 64MB. This can be resized in the flashrd script, or size can be changed in /etc/fstab on an existing system. It will be saved to a tar file on flash on shutdown. The memory filesystem is populated from flash on boot.
The /tmp directory is a memory filesystem sized at 16MB. This filesystem is not saved at shutdown, nor populated on boot. This can be resized in the flashrd script, or size can be changed in /etc/fstab on an existing system.
Some people want to make /home persistent across boots. The following config would make /home a 512MB on-flash filesystem that is part of the openbsd.vnd file:export vnddirs="root bin etc sbin usr home" # must match vnddirs= in stand/rc and fstab export vndsize="102400 auto 102400 auto auto 1048576" # min partition sizes (or auto) (in 512 byte sectors)The following config would make /home a 512MB memory filesystem, saved to flash as a separate tar file on shutdown and loaded from flash on boot:export tardirs="var home" # must match tardirs= in stand/rc and rc.conf.local set -A tarsize 65536 1048576 # mfs sizes for tar dirs (in 512 byte sectors)Finally, you could simply make /home a symlink to /flash/home in /etc/rc.local or some such.ln -fs /flash/home /home
Some future version of the script will allow you to define subdirectories of a vnddir as a separate paritition in vnddirs (such as /usr/src mounted on top of /usr.) For today, /usr can only be a single partition.
A USB key is a nice target for flashrd. It is cheap, easily removable and replacable without unracking and disassembling the entire machine. SLC usb keys are also cheap and recommended. Of course, flashrd images are easily upgradeable, as long as you have enough free space on the flash, so you should never need to remove the flash.
If you want to save things like dhcpd.leases on shutdown, you can simply set tardirs=var in /etc/rc.shutdown. If you want to do it manually (such as after you add or remove ports) then do: "tar cf /flash/var.tar -C /var ." In the case of dhcpd.leases, I suggest moving the dhcpd.leases file to the /flash partition (such as dhcpd -l /flash/dhcpd.leases) to ensure persistency.
Sure, just download an image and fire up the 'flashimg' under qemu, bochs, or another virtual environment.
flashrd has several advantages over non-ramdisk read-only schemes like flashdist and other variations of it. It also has advantages over other ramdisk based systems.
- Uses complete, default OpenBSD installation
- /etc/rc and everything following it runs completely unmodified, only the kernel, bootstrap, and partitioning are modified from default
- Running read-only system can be upgraded in-place by simply copying 3 files and rebooting (read-write systems can be upgraded by copying only 2 files and rebooting)
- Multiple versions of the system can exist on the same flash concurrently
- The bootstrap will attempt to run the older version, if the new version is not mountable
- No distribution list or specialized build system is required, flashrd packages up whatever binaries your system operates with
- flashrd can even create images from a running system, including any installed software
- Using a complete, default installation means that normal ports and packges can be installed and used on a flashrd system without hassle
- Cleanup in the case of failure, no filesystems or vnd images stay mounted, getting in your way for recovery
I wrote flashdist in 2002, to load up networking images for an outdoor 802.11 network. The script had a number of shortcomings. It also served a purpose. (You couldn't stick an OpenBSD installation CD into a Soekris 4501. PXE boot wasn't quite there, either. The CHS bootloader was unforgiving. And so on.) It occured to me later that there was still a better way to do it.
flashrd is designed to accommodate NSH without much hassle. NSH is intended to control configuration of most major daemons and functions for networking (routers, firewalls, VPN appliances). This is a major configuration shift and must be handled with care.
The suggested installation procedure:
Unpack and place nsh in /flashPlace nsh in /flash to avoid overwriting during flashrd system upgrade procedure
cp nsh-XXXXXXXX.tar.gz /flash
tar xzf nsh-XXXXXXXX.tar.gz
make LDADD="-ledit -ltermcap -lsqlite3 -static" CFLAGS="-O -DDHCPLEASES=\\\"/flash/dhcpd.leases\\\""
This will force NSH to be built as a static binary, so that you may upgrade the flashrd system without affecting nsh. Also it forces nsh to store dhcpd leases in /flash/dhcpd.leases (instead of /var/db, to preserve lease persistency across flashrd reboots). It is suggested you re-build nsh from a matching system (matching version and architecture) during upgrades. If you can't do this, building NSH as a static binary helps, as there are typically not changes made between OpenBSD versions that will break NSH to the point where you can't remotely login!
Enter read/write moderw
Activate save.sh to save NSH configurations to /flashNSH calls /usr/local/bin/save.sh to permanently store configurations. It does this because you may want to run other commands at the same time that you store the config (such as check the config into a revision control system, or to read/write, read/only state on a partition other than /flash, or whatever else you can imagine.)
sed -e 's/etc/flash/' < save-rw.sh > /usr/local/bin/save.sh
chmod a+x /usr/local/bin/save.sh
Run nsh on startupecho "/flash/nsh/nsh -i /flash/nshrc" >> /etc/rc.local
Remove manual network configurationsrm /etc/hostname.vr0
Also be sure to stop any other daemons (that you intend to control via NSH) from starting up before NSHecho sshd_flags=NO >>/etc/rc.conf.local
echo pf=NO >>/etc/rc.conf.local
echo inetd=NO >>/etc/rc.conf.local
Link nsh to /binln -s /flash/nsh/nsh /bin/nsh
Create your initial NSH configurationUpon 'write', nsh will automatically grab interface IPs, aliases, and routes from the kernel. NSH looks in:
(when you 'write' a new config to disk) to store daemon configurations persistently inside the nshrc.
With this knowledge, if you wanted to control PF, DHCP and the local resolver through NSH, here's what it would look like:
cp /etc/pf.conf /var/run/pf.conf.0
cp /etc/dhcpd.conf /var/run/dhcpd.conf.0
cp /etc/ssh/sshd_config /var/run/sshd.conf.0
mv /etc/resolv.conf /var/run/resolv.conf.0
ln -s /var/run/resolv.conf.symlink /etc/resolv.conf
Resume read-only modero
Now you must review /flash/nshrc to see if NSH has everything you need. Look for interface IPs being properly configured, routes being in place, BGP, OSPF, DHCP, etc... configuration being in place (and the daemons enabled), the goal here is to boot up with nsh and expect that you can manage all daemons through it!
Now it's time to reboot your flashrd system and watch nsh take over boot-up configurations, to ensure proper installation. If you haven't used nsh before, it is strongly recommended that you are on-site (or have remote serial console access) should nsh not do what you expect it to do!
You can obtain new image files by building or downloading a new flashrd image, mounting the vnd file, and copying them out of the mounted partition. If your device uses a serial console, make sure you use an image where the openbsd.vnd is configured appropriately first (a pre-configured image from the flashrd web site, or one configured with cfgflashrd). If you put a PC console image on a serial device and reboot, you'll have to SSH in over the network to fix /etc/ttys.
gzip -d flashimg.i386.pccons-20130129.gz
vnconfig vnd0 flashimg.i386.pccons-20130129
mount /dev/vnd0a /mnt
cp bsd openbsd.vnd var.tar /tmp
vnconfig -u vnd0
You could also try running mkkern and mkdist directly to obtain the raw FLASHRD bsd kernel, openbsd.vnd, and var.tar files. mkkern leaves output in a directory named with the format: /tmp/mkkern.XXXXXXXX while mkdist creates openbsd.vnd and var.tar in your current working directory under a new subdirectory vnd-YYYYMMDD.
On the flash boot image (which is mounted as /flash when booted under flashrd) you have three image files that define the whole system (well, whole system except the boot blocks and bootstrap script):
This is the layout accoring to the default configuration that defines 'var' as handled under 'tardirs'. If you moved var to 'vnddirs' then you would only have a flashrd ramdisk kernel and openbsd.vnd file.
To upgrade, first you need to run 'cfgflashrd' against the flash boot image that you are upgrading to. This is important as cfgflashrd will setup the /etc/boot.conf and /etc/ttys files for serial console use (if you activate com0). Alternately, use a pre-compiled disk image that already has your console type selected.
You can skip running cfgflashrd against the new flash image and instead simply copy /etc/ttys and /etc/boot.conf from the source /etc to the new /etc directory (like you see below for hostname.vr0 and mygate.)
Once you have a properly configured image that reflects the disk type and console type of your destination system, you need to extract the above mentioned image files and upload the new image files to /flash/new, modify the new /etc directory to reflect your needs, move the old files into /flash/old, move the /flash/new files to /flash, and then reboot. This might look something like:
Modify new /etc to include necessary configuration (non-NSH case)Naturally you want to include any other configuration files that you need for your system.
tar czf /flash/onetime.tgz -C / /etc/hostname.* /etc/mygate /etc/myname /etc/ssh /etc/passwd /etc/master.passwd /etc/pwd.db /etc/spwd.db /etc/pf.conf /etc/sysctl.conf
If you forget to copy a config over, you can later retrieve it from the /etc partition of the vnd image stored in /flash/old/openbsd.vnd
If you include an updated rc.conf.local in onetime.tgz:
Current versions of flashrd have a modified rc.conf.local which calls /etc/rc.flashrd.conf. If you replace rc.conf.local, include the same call to rc.flashrd.conf.
If you include an updated rc.local in onetime.tgz:
The flashrd rc.local calls /etc/rc.flashrd.local which must be preserved as well.
If you include an updated rc.shutdown in onetime.tgz:
Finally, rc.shutdown calls /etc/rc.flashrd.shutdown, which must also be preserved if you want flashrd to archive tardirs.
Any rc.local, rc.conf.local or rc.shutdown file transferred to a new image via onetime.tgz must call these /etc/rc.flashrd scripts for flashrd to function properly. See the rc.conf, rc.conf.local and rc.shutdown scripts included with flashrd as examples.
Modify new /etc to include necessary configuration (NSH case)Here we assume that you start nsh from /etc/rc.local and that you initially installed NSH according to the directions above.
tar czf /flash/onetime.tgz -C / /etc/rc.local /etc/rc.conf.local /etc/ssh /etc/passwd /etc/master.passwd /etc/pwd.db /etc/spwd.db
Current versions of flashrd have a modified rc.conf.local and rc.local which should be investigated (and upgraded from flashrd) before rebooting. Merge flashrd's latest rc.conf.local and rc.local with your own before saving the result to /flash/onetime.tgz.
Move new FLASHRD kernel/system image in placecd /flash
mv bsd openbsd.vnd var.tar /flash/old
mv bsd openbsd.vnd var.tar /flash
Boot new imagereboot
If fsck -y or mount fails to mount any partition within /flash/openbsd.vnd (your default image), the bootstrap script will try to run with the /flash/old/openbsd.vnd image instead. Read /stand/rc to see exactly what happens. (It is fairly simplistic today, and I'm sure it could be improved upon.)
Stop using bash, it's a poor clone of the actual Korn shell (and thinks that it can replace the Korn shell!) Executing flashrd from bash seems to be problematic. OpenBSD's ksh has the features that most people use bash for anyways, such as command line history recall / editing and ksh's programmability (which gets attributed to bash by new comers.) Try it out, you'll find that installing bash is superfluous (and, at least in this case, harmful.)
Read-only mounts are a very simple way to avoid potential flash corruption from flash write wear, and other random flash controller problems. As well, read-only use is not a burden on typical appliance-style systems. One objective of flashrd is to create a system that runs reliably for many years, with a minimal chance of failure. Thus, it incorporates an optional read-only approach. Cheap flash can last forever in this configuration. (I am not interested in spending more than $10 for flash on my $100 ALIX system.)
Copy your preferred time description file to /etc/localtime. Either look in /usr/share/zoneinfo for the right file or just copy /etc/localtime over from a box that already has the right timezone.
Pkg tools work as expected under flashrd just like they do on a default OpenBSD install. The process is identical to a normal system, except you have to go to rw mode before installing, and go back to ro mode after. (You actually run the commands 'rw' and 'ro' to do this under flashrd.)