Skip to content

Changes

Page history
Updated Docs for BXE v2.0.0 authored by Farzad Fatollahi-Fard's avatar Farzad Fatollahi-Fard
Show whitespace changes
Inline Side-by-side
Building-Hardware.md
View page @ 5e73c173
---
title: BXE FireSim Documentation - Building Hardware
version: v1.0.0
version: v2.0.0
firesim:
version: v1.16.0
url: https://github.com/firesim/firesim/tree/1.16.0
docs-url: https://docs.fires.im/en/stable/Building-a-FireSim-Xclbin.html
version: a78040a
url: https://github.com/firesim/firesim/tree/a78040a
docs-url: https://docs.fires.im/en/stable/Getting-Started-Guides/On-Premises-FPGA-Getting-Started/Building-a-FireSim-Bitstream/Xilinx-Alveo-U250.html
chipyard:
version: 766ea73
url: https://github.com/ucb-bar/chipyard/tree/766ea73
doc-url: https://chipyard.readthedocs.io/en/stable/
aro-cn: D2021-2106030006
---
......@@ -12,20 +16,20 @@ aro-cn: D2021-2106030006
This guide will show you how to change FireSim to build a dual-core Rocket RISC-V core. You can use this guide to modify Chipyard[^2] to build your custom architectures.
This work was funded by the iARPA AGILE Program. [^1]
This work was funded by the IARPA AGILE Program. [^1]
[[_TOC_]]
# Building a Dual-Core Rocket
In this example, we will be modifying FireSim to generate a dual-core RISC-V Rocket Core (`vitis_firesim_rocket_dualcore_no_nic`) and booting it up on the attached FPGA. Generating bitstreams for the Alveo U250 can take a **few hours** to complete, so it's recommended to do your work in a [persistent session](home#opening-a-persistent-session-on-bxe-firesim-nodes).
In this example, we will be modifying FireSim to generate a dual-core RISC-V Rocket Core (`alveo_u250_firesim_rocket_dualcore_nic`) and booting it up on the attached FPGA. Generating bitstreams for the Alveo U250 can take a **few hours** to complete, so it's recommended to do your work in a [persistent session](home#opening-a-persistent-session-on-bxe-firesim-nodes).
## Modifying FireSim Configuration Files
FireSim has two `.yaml` files that describes the designs available to build, and well as where to build new designs.
Both of these files can be found in `~/firesim/deploy`. Examples of unmodified FireSim configuration files can be found in `~/firesim/deploy/sample-backup-configs`.
Both of these files can be found in `$FIRESIM_ROOT/deploy`. Examples of unmodified FireSim configuration files can be found in `$FIRESIM_ROOT/deploy/sample-backup-configs`.
### `~/firesim/deploy/config_build.yaml`
### `$FIRESIM_ROOT/deploy/config_build.yaml`
This files configures the build environment for FireSim. This tells the tools where to build designs, what base shell to use, and what designs to build.
First, we'll tell FireSim where to build our designs. Edit the `build-farm` section to resemble the following:
......@@ -35,61 +39,48 @@ build_farm:
base_recipe: build-farm-recipes/externally_provisioned.yaml
recipe_arg_overrides:
# REQUIRED: (replace this) default location of build directory on build host.
default_build_dir: /home/bxeuser/firesim/deploy/build_dir/
default_build_dir: /home/bxeuser/FIRESIM_BUILD_DIR
# REQUIRED: List of IP addresses (or "localhost"). Each can have an OPTIONAL
# argument, called "override_build_dir", specifying to override the default
# build directory.
#
# Ex:
# build_farm_hosts:
# # use localhost and don't override the default build dir
# - localhost
# # use other IP address (don't override default build dir)
# - "111.111.1.111"
# # use other IP address (override default build dir for this build host)
# - "222.222.2.222":
# override_build_dir: /scratch/specific-build-host-build-dir
build_farm_hosts:
- localhost
- wilson.lbl.gov
```
This tells FireSim to place all of the build files in `/home/bxeuser/firesim/deploy/build_dir/`. You can change this to any location you'd like.
This tells FireSim to place all of the build files in `/home/bxeuser/firesim/FIRESIM_BUILD_DIR` on the run farm hosts. You can change this to any location you'd like.
Next we need to tell FireSim which designs we would like to build. `builds_to_run` lists all of the recipes that we'd like FireSim to build from scratch. Since we only want FireSim to build our `vitis_firesim_rocket_dualcore_no_nic` recipe, we make sure we add it and comment out the other recipes. Here's what the `builds_to_run` section should look like:
Next we need to tell FireSim which designs we would like to build. `builds_to_run` lists all of the recipes that we'd like FireSim to build from scratch. Since we only want FireSim to build a `alveo_u250_firesim_rocket_dualcore_nic` recipe, we make sure we add it and comment out the other recipes. Here's what the `builds_to_run` section should look like:
```yaml
builds_to_run:
# this section references builds defined in config_build_recipes.yaml
# if you add a build here, it will be built when you run buildbitstream
# Unnetworked designs use a three-domain configuration
# Tiles: 1600 MHz
# <Rational Crossing>
# Uncore: 800 MHz
# <Async Crossing>
# DRAM : 1000 MHz
# - firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
# - firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
# All NIC-based designs use the legacy FireSim frequency selection, with the
# tiles and uncore running at 3.2 GHz to sustain 200Gb theoretical NIC BW
# - firesim_supernode_rocket_singlecore_nic_l2_lbp
# - firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
# - firesim_boom_singlecore_nic_l2_llc4mb_ddr3
# Configs for tutorials
# - firesim_rocket_singlecore_no_nic_l2_lbp
# - firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3
# - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3
# - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf
# - firesim_gemmini_rocket_singlecore_no_nic
# - firesim_gemmini_printf_rocket_singlecore_no_nic
# Configs for vitis/xrt
# - vitis_firesim_rocket_singlecore_no_nic
# - vitis_firesim_gemmini_rocket_singlecore_no_nic
- vitis_firesim_rocket_dualcore_no_nic
# Configs for BXE
- alveo_u250_firesim_rocket_dualcore_nic
```
In summation, your `~/firesim/deploy/config_build.yaml` should look like this:
In summation, your `$FIRESIM_ROOT/deploy/config_build.yaml` should look like this:
```yaml
# Build-time build design / AGFI configuration for the FireSim Simulation Manager
# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
# this refers to build farms defined in config_build_farm.yaml
# this refers to build farms defined in config_build_farm.yaml
build_farm:
base_recipe: build-farm-recipes/externally_provisioned.yaml
recipe_arg_overrides:
# REQUIRED: (replace this) default location of build directory on build host.
default_build_dir: /home/bxeuser/firesim/deploy/build_dir/
default_build_dir: /home/bxeuser/FIRESIM_BUILD_DIR
# REQUIRED: List of IP addresses (or "localhost"). Each can have an OPTIONAL
# argument, called "override_build_dir", specifying to override the default
# build directory.
......@@ -104,16 +95,19 @@ build_farm:
# - "222.222.2.222":
# override_build_dir: /scratch/specific-build-host-build-dir
build_farm_hosts:
- localhost
- wilson.lbl.gov
builds_to_run:
# this section references builds defined in config_build_recipes.yaml
# if you add a build here, it will be built when you run buildbitstream
# Configs for BXE
- alveo_u250_firesim_rocket_dualcore_nic
# Unnetworked designs use a three-domain configuration
# Tiles: 1600 MHz
# Tiles: 1000 MHz
# <Rational Crossing>
# Uncore: 800 MHz
# Uncore: 500 MHz
# <Async Crossing>
# DRAM : 1000 MHz
# - firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
......@@ -133,18 +127,28 @@ builds_to_run:
# - firesim_gemmini_rocket_singlecore_no_nic
# - firesim_gemmini_printf_rocket_singlecore_no_nic
# Configs for vitis/xrt
# Configs for Vitis/XRT
# - vitis_firesim_rocket_singlecore_no_nic
# - vitis_firesim_gemmini_rocket_singlecore_no_nic
- vitis_firesim_rocket_dualcore_no_nic
# Config for RHSResearch Nitefury II
# - nitefury_firesim_rocket_singlecore_no_nic
# Configs for Xilinx Alveo U250/U280
# - alveo_u250_firesim_rocket_singlecore_no_nic
# - alveo_u250_firesim_gemmini_rocket_singlecore_no_nic
# - alveo_u200_firesim_rocket_singlecore_no_nic
# - alveo_u280_firesim_rocket_singlecore_no_nic
# Config for Xilinx VCU118
# - xilinx_vcu118_firesim_rocket_singlecore_4GB_no_nic
agfis_to_share:
# - firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
# - firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
# - firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
# - firesim_boom_singlecore_nic_l2_llc4mb_ddr3
- firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
- firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
- firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
- firesim_boom_singlecore_nic_l2_llc4mb_ddr3
# - firesim_supernode_rocket_singlecore_nic_l2_lbp
- firesim_supernode_rocket_singlecore_nic_l2_lbp
# Configs for tutorials
# - firesim_rocket_singlecore_no_nic_l2_lbp
......@@ -156,26 +160,29 @@ share_with_accounts:
# To share with a specific user:
somebodysname: 123456789012
# To share publicly:
# public: public
```
### `~/firesim/deploy/config_build_recipes.yaml`
Next we need add our `vitis_firesim_rocket_dualcore_no_nic` to the list of FireSim hardware recipes. The recipe is what FireSim uses to tell Chipyard what to design to build, as well as any FPGA specific parameters. The top of the file has schema for all of the parameters available.
### `$FIRESIM_ROOT/deploy/config_build_recipes.yaml`
Next we need add our `alveo_u250_firesim_rocket_dualcore_nic` to the list of FireSim hardware recipes. The recipe is what FireSim uses to tell Chipyard what to design to build, as well as any FPGA specific parameters. The top of the file has schema for all of the parameters available.
Add the following to the end of `~/firesim/deploy/config_build_recipes.yaml`:
Add the following to the end of `$FIRESIM_ROOT/deploy/config_build_recipes.yaml`:
```yaml
# BXE FireSim Dual-Core Rocket Recipe
vitis_firesim_rocket_dualcore_no_nic:
alveo_u250_firesim_rocket_dualcore_nic:
PLATFORM: xilinx_alveo_u250
TARGET_PROJECT: firesim
DESIGN: FireSim
TARGET_CONFIG: FireSimDualRocketMMIOOnlyConfig
PLATFORM_CONFIG: BaseVitisConfig
deploy_triplet: null
TARGET_CONFIG: WithNIC_FireSimDualRocketConfig
PLATFORM_CONFIG: BaseXilinxAlveoU250Config
deploy_quintuplet: null
platform_config_args:
fpga_frequency: 140
fpga_frequency: 60
build_strategy: TIMING
post_build_hook: null
metasim_customruntimeconfig: null
bit_builder_recipe: bit-builder-recipes/vitis.yaml
bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u250.yaml
```
We have now completed configuring FireSim. The `TARGET_CONFIG` section is the specific design in Chipyard we'd like to build. However, this design doesn't exist in Chipyard...
......@@ -183,12 +190,12 @@ We have now completed configuring FireSim. The `TARGET_CONFIG` section is the sp
## Creating a Design with Chipyard Mixins
Chipyard allows us to use their library of standard architecture and hardware components to build designs. These are called *Mixins*, and the list of components is out of scope for this tutorial. For now, we'll just use these to design a dual-core Rocket core.
Let's add our design to the list of designs available to FireSim. Open `~/firesim/target-design/chipyard/generators/firechip/src/main/scala/TargetConfigs.scala` and add the following to the end of the file:
Let's add our design to the list of designs available to FireSim. Open `$CHIPYARD_ROOT/generators/firechip/src/main/scala/TargetConfigs.scala` and add the following to the end of the file:
```scala
// BXE FireSim Dual-Core Rocket
class FireSimDualRocketMMIOOnlyConfig extends Config(
new WithDefaultMMIOOnlyFireSimBridges ++
class FireSimDualRocketConfig extends Config(
new WithDefaultFireSimBridges ++
new WithDefaultMemModel ++
new WithFireSimConfigTweaks ++
new freechips.rocketchip.subsystem.WithNBigCores(2) ++
......@@ -198,68 +205,52 @@ class FireSimDualRocketMMIOOnlyConfig extends Config(
And that's it! We're ready to build our design!
## Generating a Bitstream
With the FireSim build environment and Chipyard design configured, we're ready to launch the build. With FireSim loaded into our enivornment (`source sourceme-f1-manager.sh --skip-ssh-setup`), we can launch a build.
With the FireSim build environment and Chipyard design configured, we're ready to launch the build. With FireSim loaded into our environment (`source sourceme-manager.sh --skip-ssh-setup`), we can launch a build.
```bash
cd ~/firesim
# source sourceme-f1-manager.sh --skip-ssh-setup
cd $FIRESIM_ROOT
# source sourceme-manager.sh --skip-ssh-setup
firesim buildbitstream
```
As stated before, this may take **MANY HOURS** to complete. Grab yourself a cup of your favorite beverage and let the machine build. You should see the following successful build message when done:
```
[localhost] out: [15:36:22] Starting bitstream generation..
[localhost] out: Starting optional post-route physical design optimization.
[localhost] out: [16:01:28] Phase 1 Physical Synthesis Initialization
[localhost] out: [16:05:03] Phase 2 SLL Register Hold Fix Optimization
[localhost] out: [16:05:34] Phase 3 Critical Path Optimization
[localhost] out: Finished optional post-route physical design optimization.
[localhost] out: [16:16:51] Creating bitmap...
[localhost] out: [16:22:30] Writing bitstream ./level0_i_level1_level1_i_ulp_my_rm_partial.bit...
[localhost] out: [16:22:30] Finished 6th of 6 tasks (FPGA bitstream generation). Elapsed time: 00h 46m 08s
[localhost] out: [16:23:11] Run vpl: Step impl: Completed
[localhost] out: [16:23:12] Run vpl: FINISHED. Run Status: impl Complete!
[localhost] out: INFO: [v++ 60-1441] [16:23:12] Run run_link: Step vpl: Completed
[localhost] out: Time (s): cpu = 00:07:56 ; elapsed = 07:02:18 . Memory (MB): peak = 2218.742 ; gain = 0.000 ; free physical = 14228 ; free virtual = 29502
... More Vitis Output ...
[localhost] out:
[wilson.lbl.gov] out:
xilinx_alveo_u250/
xilinx_alveo_u250/metadata
xilinx_alveo_u250/firesim.bit
xilinx_alveo_u250/firesim.mcs
FireSim FPGA Build Completed
Your bitstream has been created!
Add
vitis_firesim_rocket_dualcore_no_nic:
xclbin: /home/bxeuser/firesim/deploy/build_dir//platforms/vitis/cl_FireSim-FireSimDualRocketMMIOOnlyConfig-BaseVitisConfig/bitstream/build_dir.xilinx_u250_gen3x16_xdma_4_1_202210_1/firesim.xclbin
deploy_triplet_override: FireSim-FireSimDualRocketMMIOOnlyConfig-BaseVitisConfig
alveo_u250_firesim_rocket_dualcore_nic:
bitstream_tar: file:///home/bxeuser/firesim/deploy/results-build/2024-07-17--17-33-00-alveo_u250_firesim_rocket_dualcore_nic/cl_xilinx_alveo_u250-firesim-FireSim-WithNIC-FireSimDualRocketConfig-BaseXilinxAlveoU250Config/firesim.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
to your config_hwdb.yaml to use this hardware configuration.
Build complete! Vitis bitstream ready. See /home/bxeuser/firesim/deploy/built-hwdb-entries/vitis_firesim_rocket_dualcore_no_nic.
Build complete! Xilinx Alveo xilinx_alveo_u250 bitstream ready. See /home/bxeuser/firesim/deploy/built-hwdb-entries/alveo_u250_firesim_rocket_dualcore_nic.
The full log of this run is:
/home/bxeuser/firesim/deploy/logs/2023-05-05--00-25-06-buildbitstream-1OSMVEVQTQSX2648.log
/home/bxeuser/firesim/deploy/logs/2024-07-17--17-33-00-buildbitstream-3P5LQGXVOH0DNATS.log
```
When complete, you'll find a bitstream in your build directory: `/home/bxeuser/firesim/deploy/build_dir//platforms/vitis/cl_FireSim-FireSimDualRocketMMIOOnlyConfig-BaseVitisConfig/bitstream/build_dir.xilinx_u250_gen3x16_xdma_4_1_202210_1/firesim.xclbin`
This is important for when we want to run the simulation. Speaking of which...
# Running a Custom Design
Now that the build is complete, we need to tell FireSim that this design is available to be run. We'll add `vitis_firesim_rocket_dualcore_no_nic` to FireSim's hardware database, then configure the simulation to use the new design.
Now that the build is complete, we need to tell FireSim that this design is available to be run. We'll add `alveo_u250_firesim_rocket_dualcore_nic` to FireSim's hardware database, then configure the simulation to use the new design.
## `~/firesim/deploy/config_hwdb.yaml`
## `$FIRESIM_ROOT/deploy/config_hwdb.yaml`
Using the path generated in the build step, we add an entry to the hardware database. Add the following to the end of the file:
```yaml
vitis_firesim_rocket_dualcore_no_nic:
xclbin: /home/bxeuser/firesim/deploy/build_dir//platforms/vitis/cl_FireSim-FireSimDualRocketMMIOOnlyConfig-BaseVitisConfig/bitstream/build_dir.xilinx_u250_gen3x16_xdma_4_1_202210_1/firesim.xclbin
deploy_triplet_override: FireSim-FireSimDualRocketMMIOOnlyConfig-BaseVitisConfig
alveo_u250_firesim_rocket_dualcore_nic:
bitstream_tar: file:///home/bxeuser/firesim/deploy/results-build/2024-07-17--17-33-00-alveo_u250_firesim_rocket_dualcore_nic/cl_xilinx_alveo_u250-firesim-FireSim-WithNIC-FireSimDualRocketConfig-BaseXilinxAlveoU250Config/firesim.tar.gz
deploy_quintuplet_override: null
custom_runtime_config: null
```
## `~/firesim/deploy/config_runtime.yaml`
Now we're ready to deploy this design in the simulation. We modify this file to tell FireSim what the target design we're going to simulate. In the `target_config` section, change the `default_hw_config` to point to our new design, `vitis_firesim_rocket_dualcore_no_nic`. It should resemble this:
## `$FIRESIM_ROOT/deploy/config_runtime.yaml`
Now we're ready to deploy this design in the simulation. We modify this file to tell FireSim what the target design we're going to simulate. In the `target_config` section, change the `default_hw_config` to point to our new design, `alveo_u250_firesim_rocket_dualcore_nic`. It should resemble this:
```yaml
target_config:
......@@ -269,7 +260,7 @@ target_config:
switching_latency: 10
net_bandwidth: 200
profile_interval: -1
default_hw_config: vitis_firesim_rocket_dualcore_no_nic
default_hw_config: alveo_u250_firesim_rocket_dualcore_nic
plusarg_passthrough: ""
```
......@@ -277,7 +268,7 @@ target_config:
We can now run the design and boot Linux on this design. Following the tutorial from before:
```bash
cd ~/firesim
cd $FIRESIM_ROOT
firesim launchrunfarm
firesim infrasetup
firesim runworkload
......
......