Getting started with switch gateware development
This page provide information for people who want to contribute to the
development of switch's HDL. It is meant to explain first steps to get
going with simulation and synthesis assuming a developer
uses and has installed HDLmake (explained below), Xilinx ISE, ModelSim
(supporting multi-language simulation) on Linux PC.
A sketch (unfinished) depicting architecture of switch HDL can be found here. The figure provides a high level of details when zoomed in; while when zoomed out, it should be handy in getting a global idea of HDL architecture. Note that the switch is under development and some details of the drawing might be out of date, the general picture remains.
A description of registers that are used to control HDL modules by software can be found here (v3.3 and v4)
HDLmake tool is used for WR switch HDL development to make developers' life easier.
HDLmake
- It is a tool for generating multi-purpose makefiles for FPGA projects that is used to simulate and synthesize switch HDL
- The below explanation of the switch HDL simulation/synthesis assumes that you are familiar with HDLmake, have it installed, added to your PATH and can use it.
- The website of the HDLmake project is here
- Handful information about using HDLmake can be found in Chapter 3.4.2 of the Getting Started with the SPEC tutorial that is part of Getting Started with SPEC project
- Beware: we are currently using ISYP branch of HDLmake
HDL directory structure
ip_cores - contains external cores used by the project (included in the
git repo as submodules)
modules - include switch-specific & FPGA-independent VHDL modules
platform - contains FPGA-dependent VHDL code
sim - contains SystemVerilog models, drivers and register layouts used
by testbenches
syn - contains ISE project files for synthesis (e.g. if you want to
synthesize for 18-port switch, you should go into syn/scb_18ports)
testbench - contains testbenches for top-level of the switch and some
modules, also for a network of switches
top - contains top-levels and constraint (UCF) files
The top/bare_top/ contains scb_top_bare.vhd which is a configurable top level entity of the switch. This entity is used by both, synthesis and testbench top levels. In other words, this is a configurable IP which needs some more VHDL to simulate or synthesize.
Switch Testbench
The testbench of the switch is written in SystemVerilog and is located in testbench/scb_top/main.sv. It uses top/bare_top/scb_top_bare.vhd indirectly:
- scb_top_bare.vhd is instantiated in top/bare_top/scb_top_sim.vhd
- scb_top_sim.vhd is wrapped by testbench/scb_top/scb_top_sim_svwrap.svh
- scb_top_sim_svwrap.svh is used in the main testbench: testbench/scb_top/main.sv
WRS gateware synthesis (v4.2)
ISE projects for switch synthesis are defined for different number of ports (i.e. 8, 15, and 18) in the syn directory (e.g.: syn/scb_18ports for 18 port synthesis). The 8-port version is used by developers to speed up testing all new functionalities and bugfixes since the full 18-port synthesis takes ~3h. The ISE project in syn/scb_18ports relates to the top entity and UCF files in top/scb_18ports directory (scb_top_synthesis.vhd and scb_top_sythesis.ucf). The scb_top_sythesis.vhd instantiates top/bare_top/scb_top_bare.vhd with proper parameters.
You need to make sure that you have all the tools for the switch synthesis. You will need:
- git - to download the sources from our official repository
- hdlmake and make - to create a project file and run the synthesis
- 64-bit version of Xilinx ISE 14.5 or above - for the actual synthesis and bitstream generation
# First you need to setup your environment
/opt/Xilinx/<version>/ISE_DS/settings64.sh
export XILINX=/opt/Xilinx/<version>/ISE_DS
# Download the HDL sources
git clone --recursive git:https://www.ohwr.org/white-rabbit/wr-switch-hdl.git
cd wr-switch-hdl
git checkout wr-switch-sw-v4.2
git submodule update
# Generate SDB Metadata package with synthesis information
cd top/bare_top
./gen_sdbsyn.py --user <your name> --project WRS_18p --ver <ISE version>
# Run the synthesis
cd ../../syn/scb_18ports
hdlmake --ise-proj --make-ise
make
This can take ~3 hours
# Deploy your new gateware
scp scb_top_synthesis.bin root@<your_switch_ip_address>:/wr/lib/firmware/18p_mb-LX240T.bin
- Reboot the switch so that your new gateware is loaded
Steps to run switch simulation
Version of HDL compatible with v3.3 software, tag: wr-switch-sw-v3.3
The following steps are needed to simulate the switch
# Clone the repo with submodules
git clone --recursive
git:https://www.ohwr.org/white-rabbit/wr-switch-hdl.git
# Checkout tag that is compatible with v3.3 software
(wr-switch-sw-v3.3) and updates submodules
git checkout wr-switch-sw-v3.3
git submodule update
# Add symbolic link in sim to wr-cores simulation drivers:
cd sim
ln -s ../ip_cores/wr-cores/sim wr-hdl
# Enter testbench/scb_top and generate Makefile using HDLmake
cd testbench/scb_top
hdlmake --make-sim
# Most likely a scary message about dependency problem will appear. It can be ignored. The message is of a kind:
<code class="shell">
Generating makefile for simulation...
Cannot depending file for file /home/mlipinsk/Development/new_repos/wr-switch-hdl-ni/testbench/scb_top/main.sv: simdrv_wrsw_nic.svh
Cannot depending file for file /home/mlipinsk/Development/new_repos/wr-switch-hdl-ni/testbench/scb_top/main.sv: simdrv_txtsu.svh
Cannot depending file for file /home/mlipinsk/Development/new_repos/wr-switch-hdl-ni/testbench/scb_top/main.sv: if_wb_master.svh
Cannot depending file for file /home/mlipinsk/Development/new_repos/wr-switch-hdl-ni/testbench/scb_top/main.sv: if_wb_slave.svh
Cannot depending file for file /home/mlipinsk/Development/new_repos/wr-switch-hdl-ni/testbench/scb_top/main.sv: wb_packet_source.svh
Cannot depending file for file /home/mlipinsk/Development/new_repos/wr-switch-hdl-ni/testbench/scb_top/main.sv: wb_packet_sink.svh
Cannot depending file for file /home/mlipinsk/Development/new_repos/wr-switch-hdl-ni/testbench/scb_top/main.sv: scb_top_sim_svwrap.svh
</code>
# Execution of hdlmake should result in generation of proper Makefile (such as the Makefile generated for me - it will not work you your PC, but can be useful to see)
# Simulate the switch by opening ModelSim, changing the directory to
testbench/scb_top, and by running run.do script
do run.do
# This will finish finish with "# **** Error: Cannot open macro file: wave.do". This is because the repo does not contain wave.do file, so add a wave.do file and run again.
# Now you should see Ethernet frames being sent but not received ("# [port 1] tx 32"). This is because there are 2 simulation bugs in the v3.3 release which needs to be fixed
## The switch does not allow traffic through due to misconfiguration which needs to be fixed by changing the configuration of ports in testbench/scb_top/main.sv file ("rtu.set_port_config(dd, 1, 1, 1);" to "rtu.set_port_config(dd, 1, 1, 0);"), as described in the patch below:
<code class="patch">
@@ -236,7 +236,7 @@ module main;
rtu.set_bus(cpu_acc, 'h60000);
for (int dd=0;dd<g_num_ports;dd++)
begin
- rtu.set_port_config(dd, 1, 1, 1);
+ rtu.set_port_config(dd, 1, 0, 1);
end
</code>
## There is also problem with wishbone driver, add this line (as line 13) " m_default_xfer_size = 4;" to sim/wr-hdl/if_wishbone_accessor.shv
<code class="patch">
index ce1958f..f98f54f 100644
@@ -10,6 +10,7 @@ virtual class CWishboneAccessor extends CBusAccessor;
function new();
m_cycle_type = CLASSIC;
+ m_default_xfer_size = 4;
endfunction // new
virtual task set_mode(wb_cycle_type_t mode)
</code>
- now you should re-run the simulation :
make clean
do run.do - You should see frames being forwarded by the switch
Version of HDL compatible with v4 software (currently master, once released will be wr-switch-sw-v4)
The following steps are needed to simulate the switch
- Clone the repo with submodules
git clone --recursive git:https://www.ohwr.org/white-rabbit/wr-switch-hdl.git - Add symbolic link in sim to wr-cores simulation drivers:
cd sim
ln -s ../ip_cores/wr-cores/sim wr-hdl - Enter testbench/scb_top and generate Makefile using HDLmake
hdlmake --make-sim - This should result in generation of proper Makefile (such as the
Makefile
generated for me - it will not work you your PC, but can be useful
to see)
cd testbench/scb_top - Simulate by running run.do script
do run.do - You should see frames flowing
30 October 2015