uCache

This repository contains the artifacts of the FAST'26 paper: "uCache: A Customizable Unikernel-based IO Cache".

Overview

Repository structure

The project is structured as follows:

.
├── benchmark: Contains the benchmarks scripts
├──── results: (not gitted) contains the csv files generated by the benchmark scripts
├──── plots: contains the python scripts to generate the plots
├── nix: Contains Nix definition for the VMs and for competitors
├── osv: Contains the uCache prototype (in the ucache branch)
├──── include/osv/ ucache.hh,ufs.hh: headers for uCache and the uVFS.
├──── core/ ucache.cc,ufs.cc: code for uCache and the uVFS.
├──── benchmarks: contains the benchmarks code for OSv.
├──── core/llfree/: our port of LLFree to OSv.
├──── modules/ libext/lwext: source code for the ext4 implementation of OSv.
└── VMs: (not gitted) Contains the VM images created for the experiments

Our evaluation setup uses just and Nix to facilitate reproduction. For each command, we give the command to run and explain what it is doing in the background.

Benchmarks

Our evaluation revolves around 4 experiments:

• mmapbench (in Linux and OSv)
• IO benchmark (fio in Linux and our custom benchmark on OSv)
• vmcache (in Linux and OSv)
• DuckDB (in Linux and OSv)

Setup the environment

To run the experiments, you should have a server with at least 64 cores and 256GiB of RAM. The disk footprint of experiments (VM images and results) is around 10GiB.

The server should also include a spare NVMe SSD (capacity >=1.5TiB) that can be overwritten. The experiments presented in the paper use a PCI 5.0 SSD, the KIOXIA CM-7.

• Cloning this repository: As the repository contains several layer of submodules, we recommend to clone this repository using the --recursive flag.
  For example, you may clone with git clone git@github.com:TUM-DSE/uCache.git ae_ucache --recursive

Using your own machine

1. Nix

If you choose to use your own machine and have a working Nix installation, please skip this step.

If you do not have Nix installed, you can either install it using the instructions here, or use our provided Docker image that sets up a working Nix environment with just -f benchmarks/init.just enter_docker
TODO: make the docker image work

2. Benchmarks configuration

The different "justfiles" contain parameters of the evaluation. You do not need to modify the parameters in the individual justfiles (under the benchmark directory).

However, you should modify the PCI ID of the SSD you will use for the experiments. You can get this PCI ID by running sudo lspci and finding your device. If the SSD is currently mounted with the nvme driver, you may match the PCI ID to the current mount number by running ls /sys/bus/pci/devices/0000:{PCI_ID}/nvme. Once you have the PCI ID, modify the ssd_id variable at the top of the justfile in the root directory (line 5).

Using our cluster's machine

In case you do not have a machine that can reproduce all experiments, we can provide access to a machine in our cluster (hostname: irene) with a PCI 5.0 SSD.

IMPORTANT: The /home/ directory is synchronized across our cluster using NFS. Please use the /scratch/{your_username}/ folder instead.

Setup the benchmarks

1. To build the VM images, you can execute: just -f benchmarks/init.just build_images.
   Explanation:

This recipe internally calls linux-image-init and osv-image-init. We reuse the Linux VM image for all baseline and build one OSv image for each benchmark.  
The Linux image is built using the nix/image.nix definition.

Expected time: ~5 mins.

2. To initialize the filesystem in the SSD, execute just reset_fs
   Explanation:

This recipe switches to the NVMe driver and creates an ext4 filesystem.
After mounting the FS on a temporary location, it creates a file /cache and writes zeroes to it (~1300GiB).
Afterwards it re-binds the SSD to the vfio driver.

Expected time: ~5 mins with a PCI 5.0 SSD.

3. To ensure that the VMs (and the VMM) are working as intended, execute just -f benchmarks/init.just check_vms. Explanation:

This recipe boots a Linux VM and runs a simple command using ssh.
Then it executes the mmapbench OSv VM.

Expected time: ~1 min.

Note: we use the recipe just ssh "poweroff" to stop the Linux VM which leads in some cases to the following error: error: Recipe 'ssh' failed on line 19 with exit code 255. This is not a malfunctionning of the scripts.
Note: If the Linux VM fails because of permissions on the key file, simply run chmod 0600 nix/keyfile.

Run the benchmarks

1. Microbenchmarks

Execute using just -f benchmarks/microbench.just run
Explanation:

This executes the mmapbench benchmark with varying number of threads and varying memory quota.
For uCache, it also executes with multiple page sizes.
The data generated by this experiment is used for figures 6 and 8.
Expected time: ~3 hours.

2. IO performance

Execute using just -f benchmarks/fio.just run
Explanation:

This executes the fio/spdk_nvme_perf benchmark on Linux and our custom benchmark on OSv.
The data generated by this experiment is used for figure 7.
Expected time: ~45 mins.

3. vmcache

Execute using just -f benchmarks/vmcache.just run
Explanation:

This execute the TPC-C workload on vmcache using the POSIX backend (pread/madvise)n, the exmap backend, and uCache.
The data generated by this experiment is used for figure 9.
Expected time: ~1 hours.

4. DuckDB

This experiment needs the Parquet file to be copied onto the SSD. To do so, use just -f benchmarks/init.just copy_tpch_files [folder containing the parquet files].
If you need to generate the files, please refer to the official DuckDB documentation. Our experiments use the scale factor 3000. Note that the DuckDB documentation indicates that the data generator takes around 8h 30 mins and uses up to 1800GiB of memory for SF=3000. Exporting data to the Parquet requires additional processing time.
To simplify the reproduction of the experiments on our cluster's machine, we will link the generated files into the reviewer's folders so that they can simply reuse them.
The experiment can then be executed using just -f benchmarks/duckdb.just run
Explanation:

This executes all queries from the TPC-H benchmark except 15 using the default DuckDB and our port of DuckDB to uCache.
The data generated by this experiment is used for figure 10.
Expected time: ~1 hours.

Plots

To generate the plots, execute ./benchmarks/plots/run_all.sh

Explanation:

This executes all plotting python scripts contained in the "benchmarks/plots" directory. Those scripts output pdf files in the "benchmarks/results" directory.