Learning to boot from PXE
Posted on
I bought a new laptop, the GPD Pocket 4. It came with windows installed by default, and I wanted to install nix on it.
I grabbed a usb, dd'd the nixos iso image on it and tried to boot.
The laptop did not recognize the drive.
Turns out, the drive crapped out, no computer would boot off it.
The normal thing to do would've been to just go get a new usb and install off of and go about setting the laptop up. That meant I would either have to go outside or wait for a new usb to arrive. I don't want to outside and I don't want to wait to setup my laptop. I have free time now and I have no clue when I will have free time next.
The menu had two other boot options. Something about PXE over ipv4 or ipv6. I only knew that PXE allowed networked boot. So hey, let's use this time to learn something new.
# DHCP
As I've learned, the first half of this process is DHCP. When a device is connected to the network it sends out a "HEY GIVE ME AN IP" message (I don't actually know how it works and didn't bother to look it up). Then your DHCP service see's this message and responds back with an IP. As part of these requests the client and server can set "options" on these requests which can send additional information. I don't know what the client sets first, but I do know the server needs to set a boot file name and location of a TFTP server. TFTP sort of like FTP.
PXE reads the boot file (usually something.pxe) from the TFTP server and then executes its code. Other boot files are then retrieved as needed from the TFTP server.
While learning this, folks on the internet dont seem too fond of TFTP, saying it could be slow. There exists iPXE which is supposed to be a better PXE. PXE (like bioses), tend to be manufacturer specific and are not created equal. iPXE tries to be better and supports a bunch of other stuff like (like booting from an ISO, and talking in HTTP). So if this all goes well i get iPXE going, point it to the iso I've already downloaded and I'm off to the races!
Spoiler alert, I didn't get to the races.
To get iPXE running, the iPXE.pxe executable needs to be served by TFTP. I am running an OPNsense box for my router/firewall and it as enough disk space and ram that I should be able to do this whole process of it. Setting the DHCP stuff is easy enough via the UI. the iPXE client sets a client option on its DHCP requests, so you want to create a tag in OPNsense off it's user-class (iPXE) and respond with a DHCP boot (what the tab in the UI is called) value of the http server.
The flow should be:
PXE -> Gets TFTP Address -> Downloads and run iPXE iPXE -> Gets HTTP address -> Does iPXE stuff like run our iso
The DHCP stuff can be done through the UI so it was. The TFTP stuff was not availble the web ui so has to be done through ssh.
# TFTP
This was my first time shelling into a BSD box. After this whole process I was left feeling that (Free)BSD is oddly cozy. I can't explain how or why, but it just does. The login prompt from opnsense, the simple shell prompt (csh?), the man pages, the disk layout, the programs. Like even if I didn't have access to all the new version of tools (nvim / rg vs vim / grep) I still got what I wanted done and it just felt cute and cozy.
Anyway, OPNsense ships with dnsmasq and dnsmasq can also act as a TFTP server. I found this out when trying to search for a TFTP program to install via the UI. I don't know how to enable it, nor did I want to look it up (via the internet), so I just read the man page.
man dnsmasq
Reading the man page was a pleasant experience (or maybe it was just my first time reading something from section 8).
It told me exactly what the program could do and how to configure it (just searched for tftp).
The conf files were listed in at the bottom, the first being /etc/dnsmasq.conf which did not exist on my system but /usr/local/etc/dnsmasq.conf did.
The first line of that file warns you not to manually edit the file and near the bottom you see the conf-dir option set to /usr/local/etc/dnsmasq.conf.d
I saw a README in that conf dir and, doing a cat resulted in this message:
cat /usr/local/etc/dnsmasq.conf.d/README
# Dnsmasq plugin directory:
# Add your *.conf files here, read in alphabetical order
Well sure why not lets do that
vim /usr/local/etc/dnsmasq.conf.d/10-tftp.conf
enable-tftp
tftp-root=/srv/tftp
:x
mkdir -p /srv/tftp
fetch -r https://boot.ipxe.org/ipxe.efi -o /srv/tftp/
I used the web ui to restart dnsmasq, but you can also use configctl to do it via shell.
Now when I boot up the laptop I see it load up iPXE but then fail as the http server does not exist. That is progress though, now we just need to serve our iso over http.
One thing to note is that nearly all the instructions online focus on legacy/bios boot. All my devices boot via UEFI (which is why we downloaded the efi above instead of the .kpxe file). There are ways to setup DHCP to respond with the appropriate files for both uefi or bios boot, but I dont care enough. There are also other things that try to simplify this whole process like pixieboot and netboot.xyz but I am not interested in them.
# HTTP
OPNsense runs lighttpd for serving its web ui and I would like to piggy back off it for the iPXE stuff.
The trickest part here was finding out the web ui configuration lives at /usr/local/etc/lighttpd_webgui/ via ps.
I had to disable the ssl redirect option from the web ui and instead add it myself to end of my conf file, due how the confs are loaded. I could not think of a different way of getting the 443 port redirect disabled just for the ipxe paths
cat /usr/local/etc/lighttpd_webgui/conf.d/00-ipxe.conf
# Serve /srv/tftp under http://<ip>/ipxe/
alias.url += ( "/ipxe/" => "/srv/tftp/" )
url.redirect += ( "^/ipxe$" => "/ipxe/" )
$SERVER["socket"] == "0.0.0.0:80" {
ssl.engine = "disable"
$HTTP["url"] !~ "^/ipxe(?:/|$)" {
$HTTP["host"] =~ "(.*)" {
url.redirect = ( "^/(.*)" => "https://%1/$1" )
}
}
}
$SERVER["socket"] == "[::]:80" {
ssl.engine = "disable"
$HTTP["url"] !~ "^/ipxe(?:/|$)" {
$HTTP["host"] =~ "(.*)" {
url.redirect = ( "^/(.*)" => "https://%1/$1" )
}
}
}
I started off with a basic boot.ixpe file
#!ipxe
menu Choose an ISO
item nix-minmal NixOS 25.04 Minimal
item nix-gui NixOS 25.04 GUI
choose target && goto ${target}
:nix-minimal
sanboot http://10.0.0.1/ipxe/nixos-minimal-25.05.812242.3de8f8d73e35-x86_64-linux.iso
goto menu
:nix-gui
sanboot http://10.0.0.1/ipxe/nixos-graphical-25.05.812242.3de8f8d73e35-x86_64-linux.iso
goto menu
And here is what I spoiled eariler, it didnt work.
I would get a boot but then nixos would complain about /mnt/iso or something being missing and failing to go further.
This discussion has better information on why it doesn't work: https://github.com/ipxe/ipxe/discussions/962
# Proper netboot files
So my dreams of network booting off an iso are crushed, so where do I go from here?
Well it turns out the ISO comes with a bootloader, which contains instructions on how to boot a kernel with an initial ram disk (hint this when I learned what initrd means).
So can't we do the same?
The answer is yes! (or so I think).
I didnt try to extract the files out the iso, but use nix's built in netboot image generator which builds the necessary files.
I only had to tweak the generated .ixpe file to include the http urls but everything worked out in the end.
cat netboot.ipxe
#!ipxe
# Use the cmdline variable to allow the user to specify custom kernel params
# when chainloading this script from other iPXE scripts like netboot.xyz
kernel http://10.0.0.1/ipxe/bzImage init=/nix/store/hrgkskx4jqdz4nl3p1f4m1dvrr9b3lij-nixos-system-nixos-kexec-25.11pre708350.gfedcba/init initrd=initrd nohibernate loglevel=4 lsm=landlock,yama,bpf ${cmdline}
initrd http://10.0.0.1/ipxe/initrd
boot
I still wonder if I can extract the files from the graphical installer and boot KDE off the network, but now that the OS is installed my interest has waned. Maybe one day I will revisit