How to build and test your OpenWRT packages with Docker

2024-01-25

Building and testing software packages for OpenWRT is challenging because this Linux distribution often runs on the devices with exotic architecture and uses centralized configuration (UCI) with which you often need to integrate your software. In this article we will use Docker and QEMU to test package installation on MIPS architecture and discuss what scripts and other files to include in the package to better integrate your software with UCI and OpenWRT itself.

Table of contents

• Why OpenWRT?
• How to build OpenWRT package?.
  • Post-install scripts.
  • Post-delete scripts.
  • Pre-delete scripts.
  • Init scripts.
  • First-boot scripts.
  • Persist files across system upgrades.
  • Package contents.
• How to test OpenWRT package with Docker.
  • Testing packages for host architecture
  • Testing packages for non-host architecture.
• Conclusion.

Why OpenWRT?

OpenWRT is a popular Linux distribution for network routers that brings the power of Linux kernel to resource-constrained devices. Companies use it as a base for their own routers' firmware, regular people use it to replace vendor-provided firmware which is often closed-source and lack many features compared to open-source OpenWRT.

Apart from OpenWRT there is FreeBSD-based pfSense. This distribution supports only x86 (64 bit) architecture whereas OpenWRT supports x86, ARM and MIPS architectures. FreeBSD and Linux are completely different kernels, and you would need a whole different setup for building FreeBSD packages.

How to build OpenWRT package?

OpenWRT uses opkg package manager that can install ipk/opk packages from online repositories and local file system, and to build your own package you need to use opkg-utils. Ipk/opk package format is similar to deb but uses tar instead of ar to package files. The simplest way of building a package is to use opkg-build command.

# Flag `-c` replaces `ar` with `tar`. This is mandatory for OpenWRT.
$ opkg-build -c input-dir output-dir

The input-dir contains the files that you want to install plus CONTROL directory with metadata (for deb format this directory is called DEBIAN and you can use this name with opkg-build as well). We don't know other major differences between deb and ipk/opk. For this reason we ended up converting to ipk from deb that was generated by fpm — a tool that we use to produce packages for other Linux distributions. However, the contents of pre/post install/update/remove scripts is different for OpenWRT and there are other special files that you might want to include in the package.

The package includes scripts that are run prior to or after the installation, update and removal of the package. Often package maintainers include them to start/stop services and update firewall rules.

Post-install scripts

Setting up firewall rules in packages scripts is not typical of Linux distributions other than OpenWRT. This is due to the fact that OpenWRT uses Unified Configuration Interface (UCI) — a centralized way of managing system configuration. Through UCI you can setup firewall rules that are inactive by default and can later be enabled via OpenWRT's web interface or via command-line interface.

The following commands set up firewall rule to accept TCP and UDP traffic on port 1234. The rule is disabled by default.

# post-install script
uci -q batch >/dev/null <<'EOF'
add firewall rule
set firewall.@rule[-1].dest_port='1234'
set firewall.@rule[-1].src='*'
set firewall.@rule[-1].name='Allow-MyApp-any'
set firewall.@rule[-1].proto='udp tcp'
set firewall.@rule[-1].target='ACCEPT'
set firewall.@rule[-1].enabled='0'
commit firewall
EOF

Usually you run each line as a separate uci command (e.g. uci add firewall rule), but uci batch is generally faster for a large number of lines. It is up to you to enable or disable the rule by default: for public-facing services (e.g. VPNs) it is generally safe to open the port by default, for everything else (e.g. DNS resolver like Stubby) I would rather close the port by default.

Later the rule can be enabled with the following commands.

# NNN is the actual index of the rule
uci set firewall.@rule[NNN].enabled=1
uci commit firewall
fw3 reload

Post-delete scripts

Deleting the rules is more involved as you need to find the rule index and delete all the matching lines. To distinguish between the package update and package deleting you need to check the first argument of the script.

# post-delete script

delete_rule() {
    name=
    config_get name "$1" name
    if test "$name" = "Allow-MyApp-any"; then
        uci -q delete firewall."$1" || true
    fi
}

case "$1" in
0 | remove)
    . /lib/functions.sh
    config_load firewall
    config_foreach delete_rule rule
    uci -q delete firewall.mcc || true
    ;;
esac

Pre-delete scripts

This is the right place to stop your services before deleting the package. Again to distinguish between the package update and package deleting you need to check the first argument of the script.

#!/bin/sh
case "$1" in
0 | remove)
    /etc/init.d/myapp stop || true
    ;;
esac

Init scripts

OpenWRT uses procd as the init system — pid 1 process that launches all other processes in the system on boot. Procd is similar to SysV init, systemd, openrc etc., however, the syntax for the scripts is different. Here is an example of init script for a typical application that does not daemonizes itself and writes logs on standard output and standard error streams.

#!/bin/sh /etc/rc.common
USE_PROCD=1
START=98 # start order
STOP=99 # stop order
start_service() {
    procd_open_instance
    procd_set_param command /usr/bin/myapp # run the command without daemonizing
    procd_set_param respawn 0 7 0 # respawn after 7 seconds delay
    procd_set_param stdout 1 # redirect stdout to syslog
    procd_set_param stderr 1 # redirect stderr to syslog
    procd_close_instance
}

Procd has a lot of features including process jails, capabilities, etc. that are documented on OpenWRT web site.

First-boot scripts

OpenWRT firmware can come with packages pre-installed, and in this case the right place to generate firewall rules would be a uci-defaults script. Such scripts are placed in /etc/uci-defaults directory and are executed on the first system boot. After successful execution they are deleted by the system. These scripts do not have any special arguments, and generally you repeat post-install script contents there.

Beware that distributing your package as part of OpenWRT firmware image means that your clients would not be able to reclaim free disk space by deleting the package: it will be deleted only in the overlay file system, but not in the underlying real file system.

Persist files across system upgrades

Usually OpenWRT firmware is updated separately from the packages using sysupgrade command. By default all the configuration files (that you specified in CONTROL/conffiles file) are retained during the upgrade, however, your application might generate other files that you want to persist during the upgrade. To do so simply add /lib/upgrade/keep.d/myapp file to your package that lists all directories and files that need to be persisted during the upgrade.

The following file lists /etc/myapp directory as the one that should be persisted during the system upgrade.

/etc/myapp

Package contents

The final package contents would look like the following.

.
├── conffiles # files that are persisted during system upgrade and that are not overwritten by package update
├── control # package metadata
├── etc
│   ├── init.d
│   │   └── myapp # init script
│   ├── myapp
│   │   └── myapp.conf # app configuration
│   └── uci-defaults
│       └── myapp # the script that runs on the first boot
├── lib
│   └── upgrade
│       └── keep.d
│           └── myapp # the list of files that are persisted during system upgrade
├── postinst # post-install script
├── postrm # post-delete script
├── prerm # pre-delete script
└── usr
    └── bin
        └── myapp # application executable binary

Most of the contents can be generated with fpm tool using deb format as the output. Then the following script will convert deb package to ipk package.

#!/bin/sh

cleanup() {
    rm -rf "$workdir"
}

set -ex
trap cleanup EXIT
workdir="$(mktemp -d)"
mkdir -p "$workdir"/deb "$workdir"/ipk/CONTROL
cd "$workdir"/deb
# unpack deb package
ar x "$1"
tar -C "$workdir"/ipk -xf data.tar.gz
tar -C "$workdir"/ipk/CONTROL -xf control.tar.gz
# remove generated files
rm "$workdir"/ipk/CONTROL/md5sums
# patch architecture for OpenWRT
sed -i -e 's/Architecture: amd64/Architecture: x86_64/g' "$workdir"/ipk/CONTROL/control
# write ipk to /tmp
opkg-build -c "$workdir"/ipk /tmp

How to test OpenWRT package with Docker

Similar to any other major Linux distribution OpenWRT maintains rootfsDocker images of root file systems, but it has a separate image for each target architecture. The documentation is on Github. There is a separate tag for each architecture plus OpenWRT version combination. There are also sdk and imagebuilder images that are useful for building the package and the firmware image with the package pre-installed, but we will not discuss them here.

Testing packages for host architecture

Rootfs images are useful to test your package installation/removal without investing into a router running OpenWRT (although, you should definitely invest in such a device to feel comfortable with doing system upgrades, working with web UI etc.). To install and remove the package use the following commands.

$ docker run --rm -it -v /tmp:/tmp openwrt/rootfs


BusyBox v1.36.1 (2024-01-22 12:01:31 UTC) built-in shell (ash)

$ opkg update
...
$ opkg install /tmp/myapp.x86_64.ipk
Installing myapp (1.0.0) to root...
Configuring myapp.
$ opkg remove myapp
Removing package myapp from root...

Docker by default pulled and ran openwrt/rootfs image that matches the host machine's architecture (x86_64). Then we updated the package index and installed our application from the /tmp directory mounted from the host.

Testing packages for non-host architecture

Testing for an architecture other than host's is more involved and requires running a virtual machine. Luckily for us QEMU and Linux's binfmt_misc makes it easy to do so (and without Docker even noticing!).

QEMU is a tool that runs virtual machines on the host, and the most useful feature it has for us is the ability to transparently execute binary files compiled for an architecture other than the host's architecture. The following commands show how to do that on Ubuntu 22.04.3 LTS.

# install QEMUE binaries that were statically compiled for each supported architecture
$ apt-get install qemu-user-static
# list all available QEMU binaries
$ ls /usr/bin/qemu-*static
...
/usr/bin/qemu-aarch64-static
...
/usr/bin/qemu-mips-static
...
# execute a file compiled for mips
$ qemu-mips-static /tmp/file-compiled-for-mips

To execute the binary compiled for architecture X you just add qemu-X-static before the command and that's it.

Linux's Support for miscellaneous Binary Formats (binfmt_misc) allows us to execute binaries without specifying any QEMU command. Upon execution the kernel detects the actual executable format of the file and executed it via the matching QEMU binary. The matching QEMU binaries and the "magic" bytes that distinguish a particular format from all others are specified in /proc/sys/fs/binfmt_misc directory. On Ubuntu 22.04.3 LTS this directory is populated automatically after qemu-user-static package is installed.

$ ls /proc/sys/fs/binfmt_misc
...
qemu-mips
...
qemu-aarch64
...
$ cat /proc/sys/fs/binfmt_misc/qemu-mips
enabled
interpreter /usr/libexec/qemu-binfmt/mips-binfmt-P
flags: POCF
offset 0
magic 7f454c46010201000000000000000000000200080000000000000000000000000000000000000000
mask ffffffffffffff00fefffffffffffffffffeffff0000000000000000000000000000000000000020

Now to run a foreign binary you don't need anything special: it runs the same way as a native binary on the system.

# you need matching entry in /proc/sys/fs/binfmt_misc for this to work
$ /tmp/file-compiled-for-mips

Since Docker is not a virtualization platform, but a process isolation tool, binfmt-misc and QEMU allows you to transparently run Docker images that were built for other architectures. Again, you don't need anything special to do that. The following command runs OpenWRT rootfs image for MIPS architecture.

$ docker run --rm -it -v /tmp:/tmp openwrt/rootfs:mips_24kc
Unable to find image 'openwrt/rootfs:mips_24kc' locally
mips_24kc: Pulling from openwrt/rootfs
2dd8ebde9a90: Pull complete
Digest: sha256:58d0bf8e15559e0a331e23915ed0221d678c8b2a569c58c0fa25a4f991e4beca
Status: Downloaded newer image for openwrt/rootfs:mips_24kc
WARNING: The requested image platform (linux/mips_24kc) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested


BusyBox v1.36.1 (2024-01-26 09:19:40 UTC) built-in shell (ash)

$ grep ARCH /etc/os-release
OPENWRT_ARCH="mips_24kc"
$ opkg update
...
$ opkg install /tmp/myapp.mips_24kc.ipk
Installing myapp (1.0.0) to root...
Configuring myapp.
$ opkg remove myapp
Removing package myapp from root...

Docker warns that the image's architecture does not match the host's, but still successfully runs the image. This is where the power of QEMU and binfmt-misc shows itself.

Conclusion

Building OpenWRT packages is similar to any other Linux distributions, but has unique requirements if you want to integrate your package with the rest of the system.

• Generate firewall rules in post-install and uci-defaults scripts and delete them in post-delete script.
• List files (in addition to configuration files) that need to be preserved during system upgrades in /lib/upgrade/keep.d/myapp file.
• Write procd-compatible init script.
• Use opkg-build to generate ipk/opk package.
• Optionally, use fpm tool to generate deb package and then convert it to ipk/opk.

Testing OpenWRT packages is also very similar to other Linux distributions, provided that you installed QEMU and used binfmt-misc kernel feature to transparently run foreign binaries on your host. OpenWRT maintains root file system images for each combination of version and architecture all of which you can directly run on your host and in your CI/CD pipeline.

Staex is a secure public network for IoT devices that can not run a VPN such as smart meters, IP cameras, and EV chargers. Staex encrypts legacy protocols, reduces mobile data usage, and simplifies building networks with complex topologies through its unique multi-hop architecture. Staex is fully zero-trust meaning that no traffic is allowed unless specified by the device owner which makes it more secure than even some private networks. With this, Staex creates an additional separation layer to provide more security for IoT devices on the Internet, also protecting other Internet services from DDoS attacks that are usually executed on millions of IoT machines.

To stay up to date subscribe to our newsletter, follow us on LinkedIn and Twitter for updates and subscribe to our YouTube channel.