3.1. gem5
gem5 is a modular computer architecture simulator that can be configured to simulate a very broad range of different systems. For now, we maintain our own fork of gem5 on GitHub, which contains our SimBricks adapters, a Python configuration script for full system x86 simulations with SimBricks adapters, and a few other extensions, such as MSI-X support. In the long term, we hope to upstream these changes.
3.1.1. SimBricks Adapters
We have added SimBricks adapters for PCI and Ethernet. The source for these adapters and additional helper code are in src/simbricks in the gem5 repo.
3.1.1.1. Common Functionality
Much of the functionality for initialization and synchronization is agnostic to
the specific SimBricks protocol used. We have factored out this code into a
base class simbricks::base::Adapter
that just implements the SimBricks base
protocol layer. The simbricks::base::GenericBaseAdapter
class is a generic
helper class that casts incoming and outgoing messages to the appropriate type
specified in the template. Both adapters use this. Note that the adapters do not
extend this class, but instead inherit and implement the Interface
member
class. The adapters then create an instance of the GenericBaseAdapter
and
pass in a reference to themselves through the interface parameter in the
constructor.
The base adapter code also takes care of properly instantiating and connecting
multiple adapters. The simbricks::base::InitManager
singleton object,
collects references to all adapters and exposes functions to wait until a
specific adapter is ready and connected. The InitManager
does this
asynchronously to avoid deadlocks due to mismatched initialization order in
different simulators.
3.1.1.2. Ethernet
The Ethernet adapter implements a gem5 EtherInt
port, through which it can
be connected to gem5-internal NICs. We have tested this with gem5’s e1000 NIC.
The adapter is implemented in simbricks::ethernet::Adapter
. This adapter is
also a good starting point for future adapters, as it is simple but demonstrates
how to use the base adapter functionality, and both receives and sends messages.
3.1.1.3. PCI
The PCI adapter simbricks::pci::adapter
is more complicated as it handles
many more message types and interacts with the re st of gem5 in multiple ways
(MMIO, DMA, Interrupts).
To make matters worse, it also does some gymnastics to implement a PCI device in gem5 that uses the asynchronous timing memory protocol in gem5, instead of the default atomic protocol semantics for PCI devices in gem5. For this we override the MMIO port created by the PCI device super class, and implement our own timing port. Hopefully, in the future, gem5 will offer a less backward way of doing this but for now it works without drastically changing gem5’s abstractions and all the other devices using them.
3.1.2. Configuration
gem5 is configured through Python scripts. These scripts can be parametrized
through the command line. Part of the configuration can be specified and adapted
by varying the command line parameters, while many will require you to directly
change the Python configurations. We include our reference configuration for x86
full system simulation capable of running Linux and with various SimBricks
adapter configurations in configs/simbricks/simbricks.py
. This script
heavily includes parts of the common gem5 configuration.
3.1.3. Checkpointing
gem5 supports checkpoint and restore. The most common use-case for this is accelerating repeated simulations by checkpointing system state after boot and running future simulations from there. Note that SimBricks does not currently support distributed checkpoints. To leverage this feature for accelerating boot, we carefully configure our simulations to checkpoint before executing anything that affects state in other simulators, in particular before loading device drivers. On resume, each gem5 instance will restore from its own checkpoint while the rest of the simulators will just start again from their respective initial state. As this state never changed in the checkpointed system either for these components, this is still a consistent system state.
3.1.4. Usage Notes
gem5 only supports raw hard disk images. The SimBricks Makefile contains commands to build the raw images from the qcow2 images. (see section Error Opening images/output-base/base.raw)
gem5-kvm simulations require
kvm
support on the host and appropriate permissions for the user to access/dev/kvm
. Note that unlike QEMU, gem5 will fail with an error and not silently fall back to something slower.gem5-kvm configurations require
/proc/sys/kernel/perf_event_paranoid
to be set to1
or lower on the host system. gem5 Panic PerfKvmCounter::attach describes how to do so.