Native Linux 7.2.4 On Esp32-s3

Native Linux on an ESP32-S3

I've been working on getting Linux to run natively on an ESP32-S3 for a while now.

The whole thing runs on a single N16R8 board: 16 MB flash, 8 MB octal PSRAM, no SD card, no external RAM and no second computer sitting behind it doing the actual work.

Right now it's running Linux 7.2.4 on the ESP32-S3's Xtensa core, with Bash, BusyBox, MicroPython, WiFi, SSH, writable /etc and /home, cron and a few other normal Linux tools.

The project originally started mostly because I wanted to see how far Linux could actually be pushed on an ESP32-S3. Getting the kernel to boot was only part of it though. Once that worked, most of the work became trying to make the system actually usable with 8 MB of RAM and no normal MMU.

That's where a lot of the weird stuff in the project comes from.

How it works

The ESP32-S3 has two cores, so I'm splitting the chip between Linux and ESP-IDF.

Core 0 runs the Espressif firmware and handles WiFi, Bluetooth and some flash operations. Core 1 runs Linux.

The two sides talk through shared memory. So Linux isn't directly driving everything on the chip, especially the WiFi radio. Network packets and commands are passed between Linux and the ESP-IDF side.

Linux itself runs NOMMU. Programs use Xtensa FDPIC, and the kernel + read-only rootfs can execute directly from flash, which helps a lot when there's only 8 MB of PSRAM to work with.

A normal boot currently looks roughly like this:

Linux version 7.2.4-forkbank ...  Welcome to Buildroot  buildroot login: root  # free -h               total        used        free Mem:           7.7M        2.8M        4.4M

Despite the hardware, there's enough room to run Bash, MicroPython, SSH sessions and some normal command line programs.

fork()

fork() ended up being one of the more interesting problems.

On a normal Linux system the MMU and copy-on-write do most of the heavy lifting. The ESP32-S3 doesn't have that kind of MMU, so you can't just give the parent and child their own normal virtual address spaces.

The first implementation basically had to swap private process memory by copying it when execution moved between forked processes.

It worked, but it was expensive.

Later I started using the ESP32-S3 cache MMU for part of this. It's not a real process MMU and I'm not trying to pretend it is, but it can remap aligned 64 KB regions.

So instead of copying those 64 KB banks every time the active forked process changes, the kernel can switch the cache mappings.

Small and unaligned mappings still have to be copied, but the difference is pretty big.

With three busy forked Bash processes, the slowest switch was around 29 ms with the MMU path disabled. With the cache-MMU path enabled it dropped to about 5.6 ms.

That doesn't suddenly turn the ESP32-S3 into a normal MMU Linux machine, but it makes fork() much less painful and lets some workloads work that weren't very practical before.

Where it is now

The current release is 0.9 and uses Linux 7.2.4.

A normal board boots to login in around 15 seconds. There's roughly 4 MB available RAM when the test suite starts, depending a bit on what's enabled.

WiFi works as a client, and once the board is connected I can use SSH normally. WiFi credentials can also be configured from a phone over Bluetooth.

There's persistent writable storage for /etc and /home, so things like accounts, passwords, SSH keys and user files survive reboots.

I also keep board tests for the releases instead of just checking whether it reaches a shell and calling it done. The current image is tested with repeated boots, WiFi, SSH, fork workloads and some extra checks on the actual hardware.

There are obviously still limits. It's an 8 MB NOMMU microcontroller, so large applications and lots of processes will run into RAM pretty quickly.

But that's also kind of the point of the project: finding out how far this hardware can actually go instead of stopping at "Linux technically boots".

Source, images, build scripts, benchmarks and the test results are here:

https://github.com/paulneja/Linux-on-esp32-S3