Ever since I got my RISCv computer, more like a fast microcontroler, I’ve wanted to play with it. Due to other projects taking priority at the time, I didn’t have time to do this up until this summer vacation.
This summer, I finally “finished” my previous project, which I may or may not write about in the future, and had some spare time to spend on this one.
So I started on my journey which took much more work in areas I never expected, and was very easy in areas where I thought it would be very difficult.
For those that do not know what MMU is, neither did I at the start! I did not even know that this chip did not have one, and what problems this would bring.
First I will describe how I set the kernel up, with a bootloader, then getting a proper self-build compiler working, and lastly running applications.
The board I got is the sipeed MAIX bit, which is nice and small. It has a USB-c port used for flashing it, a camera module which I will be entirely ignoring, and a SD-card slot. It has some on-board flash and 8MB of RAM, 2 of which are dedicated to the neural coprocessor, which is unfortunate.
When I got this board I immediately looked if people already got linux to run on it, which was of course the case. So I already knew that was possible, and I could use it as a starting point. The repository I used was k210-linux-nommu which was the starting point of this project.
At the time this was a bit outdated, so I updated the linux version to the latest version. Although there were some problems around the config, this went mostly smoothly. All I had to do was configure the DT (device tree, basically a description of the target device) to the correct board. With this set to the right board, I could boot the latest version of the kernel!! The k210-linux-nommu now also contains this updated kernel version, if anyone is interested in this.
This repository is based on buildroot however, and I did want to build my own thing that did not depend on other projects too much. I wanted the compiler, libc, and programs to be build by me, without relying on others work. This is a project where I wanted to learn, not just copy paste others work.
Yes, now it was time to boot of an SD card. The first step is always obtaining a microSD card, which are ridiculously expensive for the small sizes. Step two is naturally figuring out if the SD card reading in my laptop still worked, which is does well enough for this project.
Since the k210 always boots of the flash first and doesn’t have direct access to the SD card, we need a bootloader. This can then mount the SD card and read the kernel image into RAM and boot it.
The obvious choice here is U-boot, which can do all this. Although it is not completely obvious from the docs, it is very easy. You just have to put the image and DTB, even if unused, on the SD card, give it the correct names and it just works!
While I was at it, I also enabled linux to mount a second partition on the SD card as root filesystem. This enables me to just mount it on my laptop, and move files to the filesystem without rebuilding a linux image. This works really well and, surprisingly, without any issues.
One caveat is that U-Boot has a watchdog enabled by default, which was disabled in the linux kernel image. So after shamefully mailing a maintainer, they responded by telling me to enable this linux watchdog.
This however did not fix the issue, and I am still not sure why this is happening. If anyone knows they can let me know and I will update the instructions.
So for now, my board just resets ever 30 seconds. This is not really an issue for me as that is more than enough to 1: boot, 2: run a program to test and 3: see it fail, but can be really annoying if you are trying to do anything more than that.
I of course wanted to run some real programs on it, more than just the busybox tools included in the repository, luckily there already was a GCC compiler package in arch specifically for risv64. Happy me tried to compile a package, compile glibc and try to run it! Well that didn’t exactly work.
The biggest issue is that this linux is very small, and only supports the bFLT binary format (you can read more about this here). This was very annoying, but I continued. I tried to google, I tried to search on github, everywhere, on how to convert elf to a bFLT file.
Finally, after many people suggesting to just use the ucLinux distribution, just use buildroot, or something along those lines, I found elf2flt. This was both a blessing and a curse, as there was apparently a good way to do it, but that meant I had to do it, which turned out to be a challenge.
This is where I heavily looked into how buildroot build their flat binaries. They also used elf2flt, but I just could not get it to work. I started working on replicating buildroot’s setup, so I can bisect what flags for building gcc made the difference. After much trial and error I got their compile flags to work, and started to switch out flags to see what changed it.
I found 2 issues: one was linker IDs, which were inserted into the binaries but not handled by elf2flt. The second one was enabling PIE, which did not seem to be supported. So with that out of the way I had a working binary! Well almost, I also had to realize that glibc was not the best choice, and instead of putting a lot of effort into getting it to work, I switched to uclibc-ng.
Yet another problem was which compiler (and linker) flags needed to be used to compile the programs. Here I did have some small problems, but passing -r to the elf2flt linker and -fPIC to the compiler, I managed to get rid of any errors.
With all these changes, I was finally able to produce an executable in the bFLT format, which was amazing!
Now there was the task to run bigger programs, like bash. To stick with the goal of running arch-like, I wanted to use the arch compile flags etc, using their package tooling. And it worked!! With some caveats.
The first is that linking libraries didn’t work out of the bat, but the issue was that I compiled them with wrong compiler flags. After I updated them to the same I used to compiling binaries, it did, actually, just work.
And, there it was, bash on nommu riscv target with just 6MB of usable memory! The only caveat is that the “–disable-bang-history” is needed when configuring bash. Enabling this seems to hit OOM errors. Although a bit odd, this is likely due to inefficient memory allocation. If anyone figured out how to make this work, please let me know!
You can do it yourself! I’ve uploaded AUR packages that allow you to compile programs for riscv uclibc (nommu).
You can search for riscv64-linux-uclibc on the AUR and you will find the relevant packages. Install all of them, and you can run riscv64-linux-uclibc-gcc to compile any program. Keem in mind you have to compile with -Wl,-elf2flt=-r -fPIC, and optionally with -Os to reduce the size of your binaries. A couple features are disabled in gcc, as they are incompatible with the target, mostly nommu. For the rest, the PKGBUILD is based on the official arch packages, so should behave the same.