1.2. SimBricks Orchestration
Our orchestration framework replaces hand-crafted scripts for setting up and running experiments. Instead, experiments are described in a declarative fashion. The orchestration framework then takes care of the details, manages launching the respective component simulators, sets up the SimBricks communication channels between them, and monitors their execution. All output is collected in a JSON file, which allows easy post-processing afterwards.
1.2.1. Concepts
To declare experiments, we use multiple important concepts and terminology, which we now introduce.
1.2.1.1. Experiments
An experiment defines which component simulators to run and how they are
connected. To define one, instantiate the class
Experiment in your own Python
module, which has member functions to further define the component simulators to
run. SimBricks comes with many pre-defined experiments, which can serve as
starting guides and are located in the repository under
experiments/pyexps.
- class simbricks.orchestration.experiments.Experiment(name: str)
Base class for all simulation experiments.
Contains the simulators to be run and experiment-wide parameters.
- checkpoint
Whether to use checkpoint and restore for simulators.
The most common use-case for this is accelerating host simulator startup by first running in a less accurate mode, then checkpointing the system state after boot and running simulations from there.
1.2.1.2. Runs
Experiments can be executed multiple times, for example, to gain statistical insights when including a random or non-deterministic component. We call each execution a run of the experiment. Each run produces its own output JSON file. The file name includes the number of the run.
The number of runs can be specified when invoking experiments/run.py. When using simulator checkpointing, we use one run to boot the simulator and take the checkpoint, and a second one to carry out the actual experiment. This is the reason for two output JSON files being produced in this case. For more information, see Checkpointing.
1.2.1.3. Component Simulators
SimBricks defines multiple, ready-to-use component simulators in the module
orchestration/simulators.py. These include host, NIC, network, and PCIe
device simulators. Each simulator is defined by a class deriving from
Simulator, which provides the
necessary commands for their execution. We also offer more specialized base
classes for the different component types, which implement common member
functions, for example, to connect NICs or network component simulators to a
host simulator.
- class simbricks.orchestration.simulators.Simulator
Base class for all simulators.
- resreq_cores() int
Number of cores this simulator requires during execution.
This is used for scheduling multiple runs and experiments.
- resreq_mem() int
Number of memory in MB this simulator requires during execution.
This is used for scheduling multiple runs and experiments.
- prep_cmds(env: ExpEnv) List[str]
Commands to prepare execution of this simulator.
- run_cmd(env: ExpEnv) str | None
Command to execute this simulator.
- class simbricks.orchestration.simulators.HostSim(node_config: NodeConfig)
Bases:
SimulatorBase class for host simulators.
- sync_mode
Synchronization mode.
0 is running unsynchronized, 1 synchronized. Depending on the concrete simulator, there may be additional modes.
- sync_period
Period in nanoseconds of sending synchronization messages from this device to connected components.
- pci_latency
Latency in nanoseconds for sending messages to components connected via PCIe.
- class simbricks.orchestration.simulators.NetSim
Bases:
SimulatorBase class for network simulators.
- sync_mode
Synchronization mode.
0 is running unsynchronized, 1 synchronized. Depending on the concrete simulator, there may be additional modes.
- sync_period
Synchronization period in nanoseconds from this network to connected components.
- eth_latency
Ethernet latency in nanoseconds from this network to connected components.
- class simbricks.orchestration.simulators.PCIDevSim
Bases:
SimulatorBase class for PCIe device simulators.
- sync_mode
Synchronization mode.
0 is running unsynchronized, 1 synchronized. Depending on the concrete simulator, there may be additional modes.
- sync_period
Period in nanoseconds of sending synchronization messages from this device to connected components.
- pci_latency
Latency in nanoseconds for sending messages to components connected via PCI.
1.2.2. Node and App Config
To configure the workload and the software environment of nodes, use the classes
NodeConfig and
AppConfig. The former is passed to
every host simulator and defines, for example, the networking configuration like
IP address and subnet mask, how much system memory the node has, but also which
disk image to run. You can read more about the latter under Images.
The NodeConfig contains an
attribute for an instance of
AppConfig, which defines the
workload or the concrete commands that are executed on the node. You can also
override config_files() to
specify additional files to be copied into the host. These are specified as key
value pairs, where the key represents the path/filename inside the simulated
guest system and the value is an IO handle of the file to be copied over.
- class simbricks.orchestration.nodeconfig.NodeConfig
Defines the configuration of a node or host.
- ip
IP address.
- prefix
IP prefix.
- cores
Number of CPU cores.
- memory
Amount of system memory in MB.
- disk_image
Name of disk image to use or absolute path to image.
- mtu
Networking MTU.
- run_cmds() List[str]
Commands to run on node.
- cleanup_cmds() List[str]
Commands to run to cleanup node.
- config_files(environment: ExpEnv) Dict[str, IO]
Additional files to put inside the node, which are mounted under /tmp/guest/.
Specified in the following format: filename_inside_node: IO_handle_of_file
- class simbricks.orchestration.nodeconfig.AppConfig
Defines the application to run on a node or host.
- run_cmds(node: NodeConfig) List[str]
Commands to run for this application.
- config_files(environment: ExpEnv) Dict[str, IO]
Additional files to put inside the node, which are mounted under /tmp/guest/.
Specified in the following format: filename_inside_node: IO_handle_of_file
1.2.3. Unsynchronized vs. Synchronized
SimBricks offers two modes of operation, unsynchronized and synchronized, which are defined on a per component basis. The default is the unsynchronized mode that is meant purely for functional testing. Unsynchronized components advance virtual time as quickly as they possibly can, which means that measurements taken on them are meaningless and cross-component measurements inaccurate.
The synchronized mode, in contrast, is meant for accurate measurements and has
to be enabled per component, for example, by setting
simbricks.orchestration.simulators.PCIDevSim.sync_mode or
simbricks.orchestration.simulators.HostSim.sync_mode. Running
synchronized means that a simulator waits to process incoming messages from
connected simulators at the correct timestamps. For technical details, see
Synchronization.
1.2.4. Link Latency and Synchronization Period
Most of the pre-defined simulators in orchestration/simulators.py provide an attribute for tuning link latencies and the synchronization period. Both are configured in nanoseconds and apply to the message flow from the configured simulator to connected ones.
Some simulators have interfaces for different link types, for example, NIC
simulators based on NICSim have a
PCIe interface to connect to a host and an Ethernet link to connect to the
network. The link latencies can then be configured individually per interface
type.
The synchronization period defines the simulator’s time between sending synchronization messages to connected simulators. Generally, for accurate simulations, you want to configure this to the same value as the link latency. This ensures an accurate simulation. With a lower value we don’t lose accuracy, but we send more synchronization messages than necessary. The other direction is also possible to increase simulation performance by trading-off accuracy using a higher setting. For more information, refer to the section on Synchronization in the Architectural Overview.
1.2.5. Images
All our host simulators boot up a proper Operating System and therefore require
a disk image. We already provide a minimal base image using Ubuntu and some
experiment-specific derivatives with additional packages installed. If you just
want to copy in additional files for your experiment, such as drivers and
executables, you don’t need to build your own image. You can just override the
method config_files() of
AppConfig or
NodeConfig to mount additional
files under /tmp/guest inside the simulated OS.
For anything more than that, for example to install additional packages, you
need to build your own image. You can find information on how to do so under
Add a Custom Image. The specific image that you want to use for a
host in your experiment is specified in the
NodeConfig class via the attribute
disk_image.
1.2.6. Checkpoints
Some of our host simulators support taking checkpoints. Using these can
dramatically speed up the boot process by executing two runs for an experiment.
In the first, the simulator is booted in unsynchronized mode using an inaccurate
CPU model. When the boot process is completed meaning the workload defined via
the class AppConfig can be
executed, a checkpoint is taken. In the second run, the simulator is switched
into synchronized mode, the CPU model replaced with the accurate one, and the
workload executed. Checkpointing can be enabled by setting the attribute
checkpoint on the
Experiment class.
When running an experiment multiple times, e.g. because you are tweaking the
workload, the checkpoint doesn’t have to be recreated all the time. When
invoking the
orchestration framework
without the --force flag, it won’t re-execute experiments and runs for which
an output JSON file already exists. So if you delete only the output file of the
second run, you can save the time for creating the checkpoint.
1.2.7. Distributed Simulations
For the moment, refer to our GitHub Q&A on this topic.