2.3.2. Implementation#

Tip

If you’re new to SimBricks and have not already read it you might want to check out our documentation background on Adapters.

As we have seen before does SimBricks enable the modular connectiion and synchronization of component simulators to create virtual prototypes.

Now we will have a look at the practical steps one has to take in order to integrate a component simulator into the SimBricks platform for the first time. We can divide this integration process roughly into two steps:

  1. Adapter Implementation: We discussed that SimBricks defines standardized interfaces and achieves simulator integration by the implementation of an Adapter between the SimBricks interface and the simulators internal abstractions.

  2. Orchestration Framework Integration: Once this adapter is implemented the respective simulator must be integrated into the SimBricks Orchestration Framework for Virtual Prototypes. This will allow users to easily make use of the integrated simulator within python scripts that are used by SimBricks to configure virtual prototypes.

2.3.2.1. Adapter#

The Adapter required for the integratoin into SimBricks must use a simulator’s extension API to act as a native device on one side and on the other side it must send and receive SimBricks messages to and from other Adapters.

2.3.2.1.1. SimBricks Interface and Message Types#

As we saw in the background section are SimBricks interfaces designed around natural component boundaries. For instance a PCIe interface connects a host simulator to a hardware device simulator whereas an Ethernet interface may connect a NIC simulator and a network simulator.

Understanding these interfaces and the respective message types associated with such an interface is a crucial first step in writing an Adapter. Typically these interfaces abstract key transactions.

When implementing an Adapter users must either re-use any of the mesage types already supported by SimBricks or implement their own Message Types depending on their needs.

Lets look at an example: the SimBricks PCIe interface currently supports the following message types between host and device: INIT_DEV, DMA_READ/WRITE/COMPLETE, MMIO_READ/WRITE/COMPLETE and INTERRUPT.

Let’s look at a concrete example of these message types, namely the DMA_WRITE message that is issued when a device issue a DMA write access to the host, demonstrating how interactions are structured:

Listing 2.4 Example SimBricks Message-Type from SimBricks Core Library#
struct SimbricksProtoPcieD2HWrite {
  // message type specific / custom fields
  uint64_t req_id;
  uint64_t offset;
  uint16_t len;
  uint8_t pad[30];
  // standard fields
  uint64_t timestamp;
  uint8_t pad_[7];
  uint8_t own_type;
  // optional message payload
  uint8_t data[];
}

Message types for any protocol are defined using structs specific to each type. Like in the shown example message type, do message types across SimBricks interfaces share a common structure with the following order:

  1. Header:

    • Includes type-specific fields and standard fields for synchronization and identification.

      • Type specific fields are in the given case the req_id, offset and len field.

      • The padding (pad) following those fields has to be adjusted.

      • Standard header fields are timestamp, pad, and the own_type field.

    • Has to be ache-line-sized.

  2. Payload (optional):

    • Variable-length, used for transmitting data.

Each message type is identified by a unique integer stored in the own_type field.

Note

The header always starts with message type specific fields, and ends with standard SimBricks message fields for synchronization and message identification.

Warning

Users must avoid conflicts with fields in the base SimBricks protocol when implementing custom message types.

The total message size is determined by channel parameters configured at runtime. Additionally is the shown example protocol (PCIe) asymmetric, requiring distinct message types for the Host-to-device (H2D) and the device-to-host (D2H) sending directions. Other protocols, like the Ethernet protocol, are symmetric. In that case both sides send packets in a send-and-forget manner, simplifying the implementation as not as many distinct message types are needed.

Adapters interpret these incoming messages, translating them into actions within the simulator. Similarly, they send these messages to communicate events back to their peers.

See also

For more exaples of such message types check out our Core Lib References

2.3.2.1.2. Actual Adapter Implementation#

Once the Adapters interface is determined and the respective message types are implemented, the next step is to actually implement the Adapter logic. Every Adapter implementation involves three main steps: Initialization, the Handling Incoming Messages as well as Polling and Synchronization.

For illustration we will have a look at the Adapter Code used by SimBricks to integrate the Verilator simulation of the Corundum NIC . This is also an example for an Adapter that implements both, the SimBricks PCIe interface as well as the SimBricks ethernet interface and would thus in a virtual prototype connect to both, a host and a network.

Important

We only show the part of the Adapter that is specific to SimBricks and it’s Core Library.

In between the functions and the functions whos implementation is not shown in the following example its the programmers responsibility to interact with the respective simulator. That means there one would need to deal with a simulator’s internal abstractions through the API they expose to trigger actions or schedule actions depending on the messages received through functions as shown in the example.

These internal abstractions depend on the actual simulator. If you want to get an idea of concrete examples on how to do this check out some of our Adapter examples.

2.3.2.1.2.1. Initialization#

  • Establish connections with peer simulators.

    • For this we use the SimBricks library helpers to establish communication channels.

  • Exchange initial protocol-specific messages.

    • Example: In PCIe, the device simulator sends device information (e.g., BARs, interrupts) to the host.

  • In Listing 2.5 you can see the initialization code from our Corundum Verilator Adapter.

Listing 2.5 SimBricks Corundum Verilator Adapter Initialization Code.#
 1...
 2
 3int main(int argc, char *argv[]) {
 4
 5  ...
 6
 7  struct SimbricksBaseIfParams netParams;
 8  struct SimbricksBaseIfParams pcieParams;
 9
10  SimbricksNetIfDefaultParams(&netParams);
11  SimbricksPcieIfDefaultParams(&pcieParams);
12
13  ...
14
15  struct SimbricksProtoPcieDevIntro di;
16  memset(&di, 0, sizeof(di));
17
18  di.bars[0].len = 1 << 24;
19  di.bars[0].flags = SIMBRICKS_PROTO_PCIE_BAR_64;
20
21  di.pci_vendor_id = 0x5543;
22  di.pci_device_id = 0x1001;
23
24  ...
25
26  pcieParams.sock_path = argv[1];
27  netParams.sock_path = argv[2];
28
29  if (SimbricksNicIfInit(&nicif, argv[3], &netParams, &pcieParams, &di)) {
30    return EXIT_FAILURE;
31  }
32
33  ...
34
35}

2.3.2.1.2.2. Handling Incoming Messages#

  • Poll the incoming queue for messages.

  • Interpret the SimBricks messages and call corresponding simulator functions to process events.

  • Message handling typically involves a switch statement to manage different message types.

    • Example: Handling an MMIO_READ message involves retrieving the corresponding memory-mapped data and responding.

  • In Listing 2.6 you can see the example code for handling incoming messages from our Corundum Verilator Adapter.

    • The main simulation loop polls the incoming queue for each channel.

    • You can see that in this Adapter two poll functions (poll_h2d, poll_n2d) are used. One to handle messages coming from the host interface and another to handle message received on the ethernet interface.

    • Each of these function triggers different actions in the simulator (in this case Verilator) depending on the Message Type they receive from their respective interface. An example of how this might be handled is shown in the h2d_read function that will read the received message and triggers the mmio read by interacting with Verilators top level module.

Listing 2.6 SimBricks Corundum Verilator Adapter Code Handling Incoming Messages.#
  1...
  2
  3static void h2d_read(MMIOInterface &mmio,
  4                   volatile struct SimbricksProtoPcieH2DRead *read) {
  5  if (read->offset < 0x80000) {
  6    volatile union SimbricksProtoPcieD2H *msg = d2h_alloc();
  7    volatile struct SimbricksProtoPcieD2HReadcomp *rc;
  8
  9    ...
 10
 11    rc = &msg->readcomp;
 12    memset((void *)rc->data, 0, read->len);
 13    uint64_t val = csr_read(read->offset);
 14    memcpy((void *)rc->data, &val, read->len);
 15    rc->req_id = read->req_id;
 16
 17    SimbricksPcieIfD2HOutSend(&nicif.pcie, msg,
 18                              SIMBRICKS_PROTO_PCIE_D2H_MSG_READCOMP);
 19  } else {
 20    mmio.issueRead(read->req_id, read->offset, read->len);
 21  }
 22}
 23
 24...
 25
 26static void poll_h2d(MMIOInterface &mmio) {
 27  volatile union SimbricksProtoPcieH2D *msg =
 28      SimbricksPcieIfH2DInPoll(&nicif.pcie, main_time);
 29  uint8_t t;
 30
 31  if (msg == NULL)
 32    return;
 33
 34  t = SimbricksPcieIfH2DInType(&nicif.pcie, msg);
 35
 36  // std::cerr << "poll_h2d: polled type=" << (int) t << std::endl;
 37  switch (t) {
 38    case SIMBRICKS_PROTO_PCIE_H2D_MSG_READ:
 39      h2d_read(mmio, &msg->read);
 40      break;
 41
 42    case SIMBRICKS_PROTO_PCIE_H2D_MSG_WRITE:
 43      h2d_write(mmio, &msg->write);
 44      break;
 45
 46    case SIMBRICKS_PROTO_PCIE_H2D_MSG_READCOMP:
 47      h2d_readcomp(&msg->readcomp);
 48      break;
 49
 50    case SIMBRICKS_PROTO_PCIE_H2D_MSG_WRITECOMP:
 51      h2d_writecomp(&msg->writecomp);
 52      break;
 53
 54    case SIMBRICKS_PROTO_PCIE_H2D_MSG_DEVCTRL:
 55      break;
 56
 57    case SIMBRICKS_PROTO_MSG_TYPE_SYNC:
 58      break;
 59
 60    case SIMBRICKS_PROTO_MSG_TYPE_TERMINATE:
 61      std::cerr << "poll_h2d: peer terminated" << std::endl;
 62      pci_terminated = true;
 63      break;
 64
 65    default:
 66      std::cerr << "poll_h2d: unsupported type=" << t << std::endl;
 67  }
 68
 69  SimbricksPcieIfH2DInDone(&nicif.pcie, msg);
 70}
 71
 72...
 73
 74static void poll_n2d(EthernetRx &rx) {
 75  volatile union SimbricksProtoNetMsg *msg =
 76      SimbricksNetIfInPoll(&nicif.net, main_time);
 77  uint8_t t;
 78
 79  if (msg == NULL)
 80    return;
 81
 82  t = SimbricksNetIfInType(&nicif.net, msg);
 83  switch (t) {
 84    case SIMBRICKS_PROTO_NET_MSG_PACKET:
 85      n2d_recv(rx, &msg->packet);
 86      break;
 87
 88    case SIMBRICKS_PROTO_MSG_TYPE_SYNC:
 89      break;
 90
 91    default:
 92      std::cerr << "poll_n2d: unsupported type=" << t << std::endl;
 93  }
 94
 95  SimbricksNetIfInDone(&nicif.net, msg);
 96}
 97
 98...
 99
100int main(int argc, char *argv[]) {
101
102  ...
103
104  do {
105    poll_h2d(mmio);
106    poll_n2d(rx);
107  } while
108
109  ...
110
111}

2.3.2.1.2.3. Polling and Synchronization#

  • Poll messages and synchronize the simulator’s clock:

    • Basic simulators: Poll queues, advancing time based on the next message timestamp.

    • Complex event-based simulators: Schedule an event to process the next message and re-schedule after processing.

  • Ensure the simulation clock never progresses ahead of incoming messages.

  • Periodically send dummy messages when no data messages are available to ensure the peer simulator can progress.

  • In Listing 2.7 tou can see example code from our Corundum Verilator adapter that handles polling and synchronization.

Listing 2.7 SimBricks Corundum Verilator Adapter Polling and Synchronization Code.#
 1...
 2
 3int main(int argc, char *argv[]) {
 4
 5  ...
 6
 7  struct SimbricksBaseIfParams netParams;
 8  struct SimbricksBaseIfParams pcieParams;
 9
10  ...
11
12  SimbricksNetIfDefaultParams(&netParams);
13  SimbricksPcieIfDefaultParams(&pcieParams);
14
15  ...
16
17  if (argc >= 6)
18    main_time = strtoull(argv[5], NULL, 0);
19  if (argc >= 7)
20    netParams.sync_interval = pcieParams.sync_interval =
21        strtoull(argv[6], NULL, 0) * 1000ULL;
22  if (argc >= 8)
23    pcieParams.link_latency = strtoull(argv[7], NULL, 0) * 1000ULL;
24  if (argc >= 9)
25    netParams.link_latency = strtoull(argv[8], NULL, 0) * 1000ULL;
26  if (argc >= 10)
27    clock_period = 1000000ULL / strtoull(argv[9], NULL, 0);
28
29  ...
30
31  int sync_pci = SimbricksBaseIfSyncEnabled(&nicif.pcie.base);
32  int sync_eth = SimbricksBaseIfSyncEnabled(&nicif.net.base);
33
34  ...
35
36  top->clk = !top->clk;
37  top->eval();
38
39  ...
40
41  while (!exiting) {
42    int done;
43    do {
44      done = 1;
45      if (SimbricksPcieIfD2HOutSync(&nicif.pcie, main_time) < 0) {
46        ...
47      }
48      if (SimbricksNetIfOutSync(&nicif.net, main_time) < 0) {
49        ...
50      }
51    } while (!done);
52
53    do {
54      ...
55    } while (
56        !exiting &&
57        ((sync_pci &&
58          SimbricksPcieIfH2DInTimestamp(&nicif.pcie) <= main_time) ||
59        (sync_eth && SimbricksNetIfInTimestamp(&nicif.net) <= main_time)));
60
61    /* falling edge */
62    top->clk = !top->clk;
63    main_time += clock_period / 2;
64    top->eval();
65
66    // adjust simulator state
67    ...
68
69    /* raising edge */
70    top->clk = !top->clk;
71    main_time += clock_period / 2;
72
73    ...
74
75    top->eval();
76  }
77  ...
78}

Here you will find more Adapter implementations of already supported simulators.

In case you want to have a look at some more actual Adapter code, have a look at one of the following Adapters:

2.3.2.2. Orchestration Framework#

TODO