Building-Hardware.md

@@ -15,6 +15,220 @@ This work was funded by the iARPA AGILE Project. [^1]
 
 [[_TOC_]]
 
+# Building a Dual-Core Rocket with Chipyard
+
+In this example, we will be modifying FireSim to generate a dual-core RISC-V Rocket Core (`vitis_firesim_rocket_dualcore_no_nic`) and booting it up on the attached FPGA. Generating bitstreams for the Alveo U250 can take a **few hours** to complete, so it's recommended to do your work in a [persistent session](home#opening-a-persistent-session-on-bxe-firesim-nodes).
+
+## Modifying FireSim Configuration Files
+FireSim has two `.yaml` files that describes the designs available to build, and well as where to build new designs.
+
+Both of these files can be found in `~/firesim/deploy`. Examples of unmodified FireSim configuration files can be found in `~/firesim/deploy/sample-backup-configs`.
+
+### `~/firesim/deploy/config_build.yaml`
+This files configures the build environment for FireSim. This tells the tools where to build designs, what base shell to use, and what designs to build.
+
+First, we'll tell FireSim where to build our designs. Edit the `build-farm` section to resemble the following:
+
+```yaml
+build_farm:
+  base_recipe: build-farm-recipes/externally_provisioned.yaml
+  recipe_arg_overrides:
+    # REQUIRED: (replace this) default location of build directory on build host.
+    default_build_dir: /home/bxeuser/firesim/deploy/build_dir/
+    build_farm_hosts:
+      - localhost
+```
+
+This tells FireSim to place all of the build files in `/home/bxeuser/firesim/deploy/build_dir/`. You can change this to any location you'd like.
+
+Next we need to tell FireSim which designs we would like to build. `builds_to_run` lists all of the recipes that we'd like FireSim to build from scratch. Since we only want FireSim to build our `vitis_firesim_rocket_dualcore_no_nic` recipe, we make sure we add it and comment out the other recipes. Here's what the `builds_to_run` section should look like:
+
+```yaml
+builds_to_run:
+    # this section references builds defined in config_build_recipes.yaml
+    # if you add a build here, it will be built when you run buildbitstream
+
+    # Unnetworked designs use a three-domain configuration
+    # Tiles: 1600 MHz
+    #    <Rational Crossing>
+    # Uncore: 800 MHz
+    #    <Async Crossing>
+    # DRAM : 1000 MHz
+    # - firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
+    # - firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
+
+    # All NIC-based designs use the legacy FireSim frequency selection, with the
+    # tiles and uncore running at 3.2 GHz to sustain 200Gb theoretical NIC BW
+    # - firesim_supernode_rocket_singlecore_nic_l2_lbp
+    # - firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
+    # - firesim_boom_singlecore_nic_l2_llc4mb_ddr3
+
+    # Configs for tutorials
+    # - firesim_rocket_singlecore_no_nic_l2_lbp
+    # - firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3
+    # - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3
+    # - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf
+    # - firesim_gemmini_rocket_singlecore_no_nic
+    # - firesim_gemmini_printf_rocket_singlecore_no_nic
+
+    # Configs for vitis/xrt
+    # - vitis_firesim_rocket_singlecore_no_nic
+    # - vitis_firesim_gemmini_rocket_singlecore_no_nic
+    - vitis_firesim_rocket_dualcore_no_nic
+```
+
+In summation, your `~/firesim/deploy/config_build.yaml` should look like this:
+
+```yaml
+# Build-time build design / AGFI configuration for the FireSim Simulation Manager
+# See https://docs.fires.im/en/stable/Advanced-Usage/Manager/Manager-Configuration-Files.html for documentation of all of these params.
+
+# this refers to build farms defined in config_build_farm.yaml
+build_farm:
+  base_recipe: build-farm-recipes/externally_provisioned.yaml
+  recipe_arg_overrides:
+    # REQUIRED: (replace this) default location of build directory on build host.
+    default_build_dir: /home/bxeuser/firesim/deploy/build_dir/
+    # REQUIRED: List of IP addresses (or "localhost"). Each can have an OPTIONAL
+    # argument, called "override_build_dir", specifying to override the default
+    # build directory.
+    #
+    # Ex:
+    # build_farm_hosts:
+    #     # use localhost and don't override the default build dir
+    #     - localhost
+    #     # use other IP address (don't override default build dir)
+    #     - "111.111.1.111"
+    #     # use other IP address (override default build dir for this build host)
+    #     - "222.222.2.222":
+    #         override_build_dir: /scratch/specific-build-host-build-dir
+    build_farm_hosts:
+       - localhost
+
+builds_to_run:
+    # this section references builds defined in config_build_recipes.yaml
+    # if you add a build here, it will be built when you run buildbitstream
+
+    # Unnetworked designs use a three-domain configuration
+    # Tiles: 1600 MHz
+    #    <Rational Crossing>
+    # Uncore: 800 MHz
+    #    <Async Crossing>
+    # DRAM : 1000 MHz
+    # - firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
+    # - firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
+
+    # All NIC-based designs use the legacy FireSim frequency selection, with the
+    # tiles and uncore running at 3.2 GHz to sustain 200Gb theoretical NIC BW
+    # - firesim_supernode_rocket_singlecore_nic_l2_lbp
+    # - firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
+    # - firesim_boom_singlecore_nic_l2_llc4mb_ddr3
+
+    # Configs for tutorials
+    # - firesim_rocket_singlecore_no_nic_l2_lbp
+    # - firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3
+    # - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3
+    # - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf
+    # - firesim_gemmini_rocket_singlecore_no_nic
+    # - firesim_gemmini_printf_rocket_singlecore_no_nic
+
+    # Configs for vitis/xrt
+    # - vitis_firesim_rocket_singlecore_no_nic
+    # - vitis_firesim_gemmini_rocket_singlecore_no_nic
+    - vitis_firesim_rocket_dualcore_no_nic
+
+agfis_to_share:
+    # - firesim_rocket_quadcore_nic_l2_llc4mb_ddr3
+    # - firesim_rocket_quadcore_no_nic_l2_llc4mb_ddr3
+    # - firesim_boom_singlecore_no_nic_l2_llc4mb_ddr3
+    # - firesim_boom_singlecore_nic_l2_llc4mb_ddr3
+
+    # - firesim_supernode_rocket_singlecore_nic_l2_lbp
+
+    # Configs for tutorials
+    # - firesim_rocket_singlecore_no_nic_l2_lbp
+    # - firesim_rocket_singlecore_sha3_nic_l2_llc4mb_ddr3
+    # - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3
+    # - firesim_rocket_singlecore_sha3_no_nic_l2_llc4mb_ddr3_printf
+
+share_with_accounts:
+    # To share with a specific user:
+    somebodysname: 123456789012
+    # To share publicly:
+```
+
+### `~/firesim/deploy/config_build_recipes.yaml`
+Next we need add our `vitis_firesim_rocket_dualcore_no_nic` to the list of FireSim hardware recipes. The recipe is what FireSim uses to tell Chipyard what to design to build, as well as any FPGA specific parameters. The top of the file has schema for all of the parameters available.
+
+Add the following to the end of `~/firesim/deploy/config_build_recipes.yaml`:
+
+```yaml
+# BXE FireSim Dual-Core Rocket Recipe
+vitis_firesim_rocket_dualcore_no_nic:
+    DESIGN: FireSim
+    TARGET_CONFIG: FireSimDualRocketMMIOOnlyConfig
+    PLATFORM_CONFIG: BaseVitisConfig
+    deploy_triplet: null
+    platform_config_args:
+        fpga_frequency: 140
+        build_strategy: TIMING
+    post_build_hook: null
+    metasim_customruntimeconfig: null
+    bit_builder_recipe: bit-builder-recipes/vitis.yaml
+```
+
+We have now completed configuring FireSim. The `TARGET_CONFIG` section is the specific design in Chipyard we'd like to build. However, this design doesn't exist in Chipyard...
+
+## Creating a Design with Chipyard Mixins
+Chipyard allows us to use their library of standard architecture and hardware components to build designs. These are called *Mixins*, and the list of components is out of scope for this tutorial. For now, we'll just use these to design a dual-core Rocket core.
+
+Let's add our design to the list of designs available to FireSim. Open `~/firesim/target-design/chipyard/generators/firechip/src/main/scala/TargetConfigs.scala` and add the following to the end of the file:
+
+```scala
+// BXE FireSim Dual-Core Rocket
+class FireSimDualRocketMMIOOnlyConfig extends Config(
+  new WithDefaultMMIOOnlyFireSimBridges ++
+  new WithDefaultMemModel ++
+  new WithFireSimConfigTweaks ++
+  new freechips.rocketchip.subsystem.WithNBigCores(2) ++
+  new chipyard.config.AbstractConfig)
+```
+
+And that's it! We're ready to build our design!
+
+## Generating a Bitstream
+With the FireSim build environment and Chipyard design configured, we're ready to launch the build. With FireSim loaded into our enivornment (`source sourceme-f1-manager.sh --skip-ssh-setup`), we can launch a build.
+
+```bash
+cd ~/firesim
+# source sourceme-f1-manager.sh --skip-ssh-setup
+firesim buildbitstream
+```
+
+As stated before, this may take **MANY HOURS** to complete. Grab yourself a cup of your favorite beverage and let the machine build. You should see the following successful build message when done:
+
+```
+[localhost] out: [15:36:22] Starting bitstream generation..
+[localhost] out: Starting optional post-route physical design optimization.
+[localhost] out: [16:01:28] Phase 1 Physical Synthesis Initialization
+[localhost] out: [16:05:03] Phase 2 SLL Register Hold Fix Optimization
+[localhost] out: [16:05:34] Phase 3 Critical Path Optimization
+[localhost] out: Finished optional post-route physical design optimization.
+[localhost] out: [16:16:51] Creating bitmap...
+[localhost] out: [16:22:30] Writing bitstream ./level0_i_level1_level1_i_ulp_my_rm_partial.bit...
+[localhost] out: [16:22:30] Finished 6th of 6 tasks (FPGA bitstream generation). Elapsed time: 00h 46m 08s 
+[localhost] out: [16:23:11] Run vpl: Step impl: Completed
+[localhost] out: [16:23:12] Run vpl: FINISHED. Run Status: impl Complete!
+[localhost] out: INFO: [v++ 60-1441] [16:23:12] Run run_link: Step vpl: Completed
+[localhost] out: Time (s): cpu = 00:07:56 ; elapsed = 07:02:18 . Memory (MB): peak = 2218.742 ; gain = 0.000 ; free physical = 14228 ; free virtual = 29502
+```
+
+When complete, you'll find a bitstream in your build directory: `/home/bxeuser/firesim/deploy/build_dir//platforms/vitis/cl_FireSim-FireSimDualRocketMMIOOnlyConfig-BaseVitisConfig/bitstream/build_dir.xilinx_u250_gen3x16_xdma_4_1_202210_1/firesim.xclbin`
+
+This is important for when we want to run the simulation. Speaking of which...
+
+# Running a Custom Design
+
 
 ---