1.3. SimBricks Orchestration

Our orchestration framework replaces hand-crafted scripts for setting up and running experiments. Instead, experiments are described in a declarative fashion. The orchestration framework then takes care of the details, manages launching the respective component simulators, sets up the SimBricks communication channels between them, and monitors their execution. All output is collected in a JSON file, which allows easy post-processing afterwards.

1.3.1. Concepts

To declare experiments, we use multiple important concepts and terminology, which we now introduce.

1.3.1.1. Experiments

An experiment defines which component simulators to run and how they are connected. To define one, instantiate the class Experiment in your own Python module, which has member functions to further define the component simulators to run. SimBricks comes with many pre-defined experiments, which can serve as starting guides and are located in the repository under experiments/pyexps.

class simbricks.orchestration.experiments.Experiment(name: str)

Base class for all simulation experiments.

Contains the simulators to be run and experiment-wide parameters.

checkpoint

Whether to use checkpoint and restore for simulators.

The most common use-case for this is accelerating host simulator startup by first running in a less accurate mode, then checkpointing the system state after boot and running simulations from there.

add_host(sim: HostSim): Add a host simulator to the experiment.

add_nic(sim: Union[NICSim, I40eMultiNIC]): Add a NIC simulator to the experiment.

add_pcidev(sim: PCIDevSim): Add a PCIe device simulator to the experiment.

add_network(sim: NetSim): Add a network simulator to the experiment.

1.3.1.2. Runs

Experiments can be executed multiple times, for example, to gain statistical insights when including a random or non-deterministic component. We call each execution a run of the experiment. Each run produces its own output JSON file. The file name includes the number of the run.

The number of runs can be specified when invoking the orchestration framework, see Running Experiments. When using simulator checkpointing, we use one run to boot the simulator and take the checkpoint, and a second one to carry out the actual experiment. This is the reason for two output JSON files being produced in this case. For more information, see Checkpointing.

1.3.1.3. Component Simulators

SimBricks comes with multiple, ready-to-use component simulators for your experiments in simbricks.orchestration.simulators. These include host, NIC, network, and PCIe device simulators. On the orchestration side, each simulator is implemented in a class deriving from Simulator, which provides the necessary commands for their execution. We also offer more specialized base classes for the different component types, which implement common member functions, for example, to connect NICs or network component simulators to a host simulator.

class simbricks.orchestration.simulators.Simulator

Base class for all simulators.

resreq_cores()

Number of cores this simulator requires during execution.

This is used for scheduling multiple runs and experiments.

resreq_mem()

Number of memory in MB this simulator requires during execution.

This is used for scheduling multiple runs and experiments.

prep_cmds(env: ExpEnv) → List[str]: Commands to prepare execution of this simulator.

run_cmd(env: ExpEnv) → Optional[str]: Command to execute this simulator.

class simbricks.orchestration.simulators.HostSim(node_config: NodeConfig)

Bases: Simulator

Base class for host simulators.

sync_mode: Synchronization mode. 0 is running unsynchronized, 1 synchronized. Depending on the concrete simulator, there may be additional modes.

sync_period: Period in nanoseconds of sending synchronization messages from this device to connected components.

pci_latency: Latency in nanoseconds for sending messages to components connected via PCIe.

add_nic(dev: NICSim): Add a NIC to this host.

add_pcidev(dev: PCIDevSim): Add a PCIe device to this host.

add_netdirect(net: NetSim): Add a direct connection to a network to this host.

class simbricks.orchestration.simulators.NetSim

Bases: Simulator

Base class for network simulators.

sync_mode: Synchronization mode. 0 is running unsynchronized, 1 synchronized. Depending on the concrete simulator, there may be additional modes.

sync_period: Synchronization period in nanoseconds from this network to connected components.

eth_latency: Ethernet latency in nanoseconds from this network to connected components.

connect_network(net: NetSim): Connect this network to the listening peer net

class simbricks.orchestration.simulators.PCIDevSim

Bases: Simulator

Base class for PCIe device simulators.

sync_mode: Synchronization mode. 0 is running unsynchronized, 1 synchronized. Depending on the concrete simulator, there may be additional modes.

sync_period: Period in nanoseconds of sending synchronization messages from this device to connected components.

pci_latency: Latency in nanoseconds for sending messages to components connected via PCI.

class simbricks.orchestration.simulators.NICSim

Bases: PCIDevSim

Base class for NIC simulators.

eth_latency: Ethernet latency in nanoseconds from this NIC to the network component.

set_network(net: NetSim): Connect this NIC to a network simulator.

1.3.2. Node and App Config

To configure the workload and the software environment of nodes, use the classes NodeConfig and AppConfig. The former is passed to every host simulator and defines, for example, the networking configuration like IP address and subnet mask, how much system memory the node has, but also which disk image to run. You can read more about the latter under Add a Custom Image.

The NodeConfig contains an attribute for an instance of AppConfig, which defines the workload or the concrete commands that are executed on the node. You can also override config_files() to specify additional files to be copied into the host. These are specified as key value pairs, where the key represents the path/filename inside the simulated guest system and the value is an IO handle of the file to be copied over.

class simbricks.orchestration.nodeconfig.NodeConfig

Defines the configuration of a node or host.

ip: IP address.

prefix: IP prefix.

cores: Number of CPU cores.

memory: Amount of system memory in MB.

disk_image: Name of disk image to use.

mtu: Networking MTU.

app: Optional[AppConfig]: Application to run on simulated host.

run_cmds() → List[str]: Commands to run on node.

cleanup_cmds() → List[str]: Commands to run to cleanup node.

config_files() → Dict[str, IO]

Additional files to put inside the node, which are mounted under /tmp/guest/.

Specified in the following format: filename_inside_node: IO_handle_of_file

class simbricks.orchestration.nodeconfig.AppConfig

Defines the application to run on a node or host.

run_cmds(node: NodeConfig) → List[str]: Commands to run for this application.

config_files() → Dict[str, IO]

Additional files to put inside the node, which are mounted under /tmp/guest/.

Specified in the following format: filename_inside_node: IO_handle_of_file

1.3.3. Synchronized vs. Unsynchronized

For most component simulators in your experiment, you can decide whether to run them synchronized or unsynchronized by setting sync_mode or sync. By default, all simulators run unsynchronized to simulate as fast as possible. When you are conducting measurements, however, you need to run synchronized, or you won’t get meaningful performance numbers.

Running synchronized means that a simulator waits to process incoming messages from connected simulators at the correct timestamps. For technical details, see Synchronization. In contrast, unsynchronized lets a simulator advance its virtual time as fast as it can. It still handles and exchanges messages with connected simulators, but it won’t wait for incoming messages and instead advances its virtual time when there’s nothing available to process.

1.3.4. Link Latency and Synchronization Period

Most of the pre-defined simulators in simbricks.orchestration.simulators provide an attribute for tuning link latencies and the synchronization period. Both are configured in nanoseconds and apply to the message flow from the configured simulator to connected ones.

Some simulators have interfaces for different link types, for example, NIC simulators based on NICSim have a PCIe interface to connect to a host and an Ethernet link to connect to the network. The link latencies can then be configured individually per interface type.

The synchronization period defines the simulator’s time between sending synchronization messages to connected simulators. Generally, for accurate simulations, you want to configure this to the same value as the link latency. This ensures an accurate simulation. With a lower value we don’t lose accuracy, but we send more synchronization messages than necessary. The other direction is also possible to increase simulation performance by trading-off accuracy using a higher setting. For more information, refer to the section on Synchronization in the Architectural Overview.

1.3.5. Running Experiments

To run experiments using our orchestration framework, use the experiments/run.py script. For your convenience, you can also use simbricks-run in the Docker images from anywhere to run experiments. In practice, running experiments will look similar to this:

$ python run.py --verbose --force pyexps/simple_ping.py
# only available inside docker images
$ simbricks-run --verbose --force pyexps/simple_ping.py

Here are all the command line arguments for the experiments/run.py script:

usage: run.py [-h] [--list] [--filter PATTERN [PATTERN ...]] [--pickled] [--runs N]
              [--firstrun N] [--force] [--verbose] [--pcap] [--repo DIR] [--workdir DIR]
              [--outdir DIR] [--cpdir DIR] [--hosts JSON_FILE] [--shmdir DIR]
              [--parallel] [--cores N] [--mem N] [--slurm] [--slurmdir DIR] [--dist]
              [--auto-dist] [--proxy-type TYPE]
              EXP [EXP ...]

positional arguments:
  EXP                   Python modules to load the experiments from

options:
  -h, --help            show this help message and exit
  --list                List available experiment names
  --filter PATTERN [PATTERN ...]
                        Only run experiments matching the given Unix shell style patterns
  --pickled             Interpret experiment modules as pickled runs instead of .py files
  --runs N              Number of repetition of each experiment
  --firstrun N          ID for first run
  --force               Run experiments even if output already exists (overwrites output)
  --verbose             Verbose output, for example, print component simulators\' output
  --pcap                Dump pcap file (if supported by component simulator)

Environment:
  --repo DIR            SimBricks repository directory
  --workdir DIR         Work directory base
  --outdir DIR          Output directory base
  --cpdir DIR           Checkpoint directory base
  --hosts JSON_FILE     List of hosts to use (json)
  --shmdir DIR          Shared memory directory base (workdir if not set)

Parallel Runtime:
  --parallel            Use parallel instead of sequential runtime
  --cores N             Number of cores to use for parallel runs
  --mem N               Memory limit for parallel runs (in MB)

Slurm Runtime:
  --slurm               Use slurm instead of sequential runtime
  --slurmdir DIR        Slurm communication directory

Distributed Runtime:
  --dist                Use sequential distributed runtime instead of local
  --auto-dist           Automatically distribute non-distributed experiments
  --proxy-type TYPE     Proxy type to use (sockets,rdma) for auto distribution

1.3.6. Images

All host simulators require a disk image. We already provide a base image with Ubuntu. If you just want to copy in additional files, e.g., drivers and executables for your workload, you don’t need to build your own image. You can use the method config_files() of NodeConfig and AppConfig to mount additional files under /tmp/guest inside the simulated OS.

For more than that, you need to build your own images. You can find the commands to do so in images/rules.mk. When building an image, it is started with Qemu in a VM with active internet access. The image-specific script located in images/scripts is then executed on the guest system to modify the image, e.g., changing config files, installing packages, or building required projects from source. In order to use your image in experiments, set the attribute disk_image on NodeConfig. This requires that your image is stored as images/out-<image_name>/<image_name>. If your host simulator requires a raw image, execute make images/out-<image_name>/<image_name>.raw to convert your image.

1.3.7. Checkpoints

Some of our host simulators support taking checkpoints. Using these can dramatically speed up the boot process by executing two runs for an experiment. In the first, the simulator is booted in unsynchronized mode using an inaccurate CPU model. When the boot process is completed meaning the workload defined via the class AppConfig can be executed, a checkpoint is taken. In the second run, the simulator is switched into synchronized mode, the CPU model replaced with the accurate one, and the workload executed. Checkpointing can be enabled by setting the attribute checkpoint on the Experiment class.

When running an experiment multiple times, e.g. because you are tweaking the workload, the checkpoint doesn’t have to be recreated all the time. When invoking the orchestration framework without the --force flag (see Running Experiments), it won’t re-execute experiments and runs, for which an output JSON file already exists. So, if you delete only the output file of the second run, you can save the time for creating the checkpoint.

1.3.8. Distributed Simulations

For the moment, refer to our GitHub Q&A on this topic.

1.3. SimBricks Orchestration

1.3.1. Concepts

1.3.1.1. Experiments

1.3.1.2. Runs

1.3.1.3. Component Simulators

1.3.2. Node and App Config

1.3.3. Synchronized vs. Unsynchronized

1.3.4. Link Latency and Synchronization Period

1.3.5. Running Experiments

1.3.6. Images

1.3.7. Checkpoints

1.3.8. Distributed Simulations

1.3.9. Slurm