30 June 2021

CHERI Compilation Issues and Getting Around Them

Background

The program we intend to port over has multiple parts and requires a build tool chain to be able to compile properly. This is usually done with CMake and compiled with something like g++. To get the code to compile for a CHERI (DSbD) secured system we either need to compile the code natively on the system we intend to use, or we need to cross compile the software using the build tools on our host system. This will also allow us to evaluate the current build system and how easily a native application can be built to support DSbD technologies.

Important Note: Due to some incompatibilities with older versions of CMake we have now moved to using an Ubuntu 20.04 LTS Server host machine rather than an 18.04 one. The current versions at the time of writing this are below:

  • Ubuntu 18.04: CMake 3.10.2
  • Ubuntu 20.04: CMake 3.16.3

Issues with native compilation

Our first port of call was to try and compile the program on the native system inside QEMU. To do this we used the riscv64-purecap virtual machine. The CheriBSD virtual machine is very barebones by default and requires some additional software to be installed before we can try compiling on it.

The CheriBuild tool allows us to compile and install additional programs as we need them. You can use python3 cheribuild.py --list-targets to show all of the available options to install. Note: You may see multiple versions of the same software listed as a version is provided for each of the architecture options, the architecture type is usually appended at the end like shown here: python-riscv64-purecap.

Here is a basic list of software we needed to install to get started. Note you will need to run python3 cheribuild.py disk-image-riscv64-purecap to merge the changes into the disk image.

  • python3 cheribuild.py libunwind-riscv64-purecap
  • python3 cheribuild.py libcxx-riscv64-purecap
  • python3 cheribuild.py libcxxrt-riscv64-purecap

The issue we ran into was when we were trying to build CMake for the native system, we are presented with several errors during compilation (Note: --pass-k-to-make is used so it does not fail on the first error). Here is the command that we ran:

python3 cheribuild.py cmake-riscv64-purecap --clean --pass-k-to-make

The errors from running this are below:

  • SystemInformation.cxx:1270:12: error: cast from provenance-free integer type to pointer type will give pointer that can not be dereferenced [-Werror,-Wcheri-capability-misuse]
  • sys/types.h:206:19: error: typedef redefinition with different types ('__ssize_t' (aka 'long') vs '__intptr_t' (aka '__intcap'))
  • setopt.c:32:10: fatal error: 'linux/tcp.h' file not found

This causes the compilation to fail. CMake was only added to CheriBuild very recently so support looks to be limited at the moment, this means we will need to look into cross compiling the software instead.

Compiling using the host system

This method relies on using the native build tools as well as the capability aware compilers provided by the CHERI project.

The first step was to run python3 cheribuild.py run-riscv64-purecap -d – this will install and run the riscv64-purecap virtual machine but more importantly it will install most of the dependencies required to cross compile our code. We will also need to run the same commands as we did for native compilation.

Note you will need to run python3 cheribuild.py disk-image-riscv64-purecap to merge the changes into the disk image.

  • python3 cheribuild.py libunwind-riscv64-purecap
  • python3 cheribuild.py libcxx-riscv64-purecap
  • python3 cheribuild.py libcxxrt-riscv64-purecap

The cheri-examples repository is a good place to start when looking at cross compilation with CMake and CHERI. Importantly it contains a toolchain file. The file helps CMake determine the location of the capability aware compilers and config files required to cross compile the application. We are using the riscv64-purecap.cmake file.

The repo listed also contains some great examples and the README includes details on how to use the toolchain to compile these using CMake and Make/Ninja. This is setup to work directly with the config files and build tools that are installed when you run a riscv64-purecap virtual machine.

Basic compilation steps

  1. Copy the toolchain file (e.g., riscv64-purecap.cmake) into the same directory as the code you want to compile.
  2. Create a build directory outside of this one and cd into it.
  3. Run: cmake -B <build dir> -S <source folder> -DCMAKE_TOOLCHAIN_FILE=riscv64-purecap.cmake
    Note: Add -DCMAKE_BUILD_TYPE=Debug if you want to compile with extra debug information.
  4. Run: make all (This will compile the application)

Once complete you should end up with a build inside your current build folder containing the executable files.

You can now attempt to run this on the QEMU virtual machine.