\renewcommand{\thesection}{}

Copyright
==============

This document is copyrighted (under the Berne Convention) by Seven
Solutions company and is formally licensed to the public under **GPL v2.0** license.
Report content can be copied, modified, and redistributed.

The Seven Solutions Logo can not be modified in any form, or by any means without prior
written permission by Seven Solutions.

Licenses
===============

~~~~~~~
The "WR SPEC Starting Kit" (as defined above) is provided under the terms of GPL v2.0
Copyright (C) 2019 - Seven Solutions

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
~~~~~~~~~


The [SPEC] and [FMC-DIO] have been released under the **CERN OHL** licence.

~~~~~~~
Copyright CERN 2011.
This documentation describes Open Hardware and is licensed under the CERN OHL v. 1.1.

You may redistribute and modify this documentation under the terms of the
CERN OHL v.1.1. (http://ohwr.org/cernohl). This documentation is distributed
WITHOUT ANY EXPRESS OR IMPLIED WARRANTY, INCLUDING OF
MERCHANTABILITY, SATISFACTORY QUALITY AND FITNESS FOR A
PARTICULAR PURPOSE. Please see the CERN OHL v.1.1 for applicable
conditions
~~~~~~~~~~~~~~~~~


\clearpage



Revision table
===============


------------------------------------------------------------------------
 Rev      Date          Author          Comments
-----  ----------- -------------------  --------------------------------
 1.0   23/05/2013   Benoit Rat\         First release
                    [Seven Solutions]

 1.1   26/07/2013   Benoit Rat\         Correct errors in submodule init
                    [Seven Solutions]

 2.0   15/05/2014   Benoit Rat\         Updating for v2.0 release
                   [Seven Solutions]

 3.0   10/12/2019   Manuel Castilla\    Updating for v3.0 release
                    Benoit Rat\
                    Miguel Jimenez\
                    Francesco Colella\
                    [Seven Solutions]

 3.1   15/01/2021   Jorge Machado\      Updating for v3.1 release
                    Benoit Rat\
                    Miguel Jimenez\
                    [Seven Solutions]
------------------------------------------------------------------------

You can also check the [Changelog section](#changelog) for more information.


Glossary
===========

NIC
:   Network Interface Card

DIO
:   Digital Input/Output

PTP
:   Precise Time Protocol, a time synchronization protocol

EEPROM
:   Electrically Erasable Programmable Read-Only Memory, a non-volatile memory

FMC
:   FPGA Mezzanine Card, an ANSI standard for mezzanine card form factor.

LEMO
:   LEMO is the name of a push-pull connectors made by the LEMO company.

SFP
:   Small form-factor pluggable transceiver, a hot-pluggable transceiver for optical fiber

PCIe
:   Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard

LM32
:   LatticeMico32 is a 32-bit microprocessor soft core optimized for field-programmable gate arrays (FPGAs).

HDL
:   Hardware description language

SPEC
:   Simple PCIe FMC carrier

UART
:   Universal Asynchronous Receiver/Transmitter

WB
:   Whishbone Bus, an open system bus interconnect architecture designed for reuse

WRC / WRPC
:   White Rabbit (PTP) Core, the main funcionality shared by all WR elements.

WR
:   White Rabbit

WRS
:   White Rabbit Switch

FRU
:   Field Replaceable Unit


\clearpage

\renewcommand{\thesection}{\oldthesection}
\setcounter{section}{0}

Introduction
=============


About the White Rabbit technology
----------------------------------

White Rabbit ([WR]) is an extension to Ethernet network with **accurate
synchronization** and Gigabit data transfer capability. It has been
conceived to fulfill the following goals:\


Time Precision
:	The [WR] technology provides a common clock for physical layer in the entire network, allowing synchronization at sub-nanosecond accuracy with picoseconds precision.

Scalability
:	The [WR] network is designed to be highly scalable up to thousands of nodes. It also intends to be as modular as possible and compatible with non-[WR] standard Ethernet devices. Finally, all products related to this technology available at the Open Hardware Repository ([OHWR]) represent “open developments”, i.e. all the hardware designs, firmware and software and documentation are fully available. This shall facilitate the new incremental developments and customization of the available designs and products.

Distance Range
:	Taking into account the size and ranges of the majority of industrial and scientific facilities, the [WR] network specifications have been chosen to support distance ranges of 10 km between [WR] network elements(nodes or switches) using fiber cables.


The **WR Starting Kit** has been designed to test and check if the White
Rabbit technology meets your requirements. It shall also facilitate the
understanding on the technology capabilities and possibilities.
WR Starting kit can also be used to evaluate what can be done with it
and how to integrate it on your own project.

You can find more information on other components and use cases on:

* Our webpage <http://www.sevensols.com/>
* The official wiki page <http://www.ohwr.org/projects/white-rabbit/wiki>


About the WR Starting Kit
----------------------------------

This WR Starting Kit uses two nodes, each one composed of a [SPEC] and one
[FMC-DIO] card.

A node makes basic operations such as input time-stamping or programmable output pulse generation.
Additionally, specific software and gateware layers allow to use it as a standard network
interface card implementing the White Rabbit technology functionalities.
Network packages with accurate time-stamping information are generated/timestamped at
the hardware level to achieve the highest accuracy.

This WR Starting Kit is based on different projects:

* [spec-sw], driver to communicate to the [SPEC] card through PCIe. It
also includes a set of tools to experiments.
* [wr-nic],  gateware/software that includes the Network Interface Card(NIC) capabilities.
* [fmc-dio-5chttla],  gateware/software that includes the support for FMC-DIO 5ch TTL
* [wrpc-sw], White Rabbit PTP firmware for the synchronization.


About this document
--------------------

This document is intended to be a step by step user friendly tutorial
to start with the White Rabbit technology. It includes the description
of some simple experiments to illustrate [WR] capabilities. Some
concepts are deliberately avoided to ease the comprehension of this document.
If you want to know more about these concepts, please access to the
related documents in the [Reference section](#references).

The rest of the document provides an explanation about the system setup, software driver, FPGA configuration and
some application examples.

This will allow easily validating the main capabilities of White Rabbit
technology.

We have also included a chapter called [Quick Start Guide for
Developers](#quick-start-guide-for-developers) in order to facilitate
the starting stages of development processes, indicating where to find
the different source projects and how to compile them.


Changelog
-----------


In this section, we resume the main changes that happens at each release of the starting-kit

### wr-starting-kit-v3.1

This release focused on adding new features, in particular:

* Generation of a pulse train
    * with a configurable frequency, width and number of pulses (including indefinite)
    * starting at given TAI
    * with the possibility of updating the pulse width during generation
* Generation of interrupts (virtual channel 5):
    * with a configurable frequency
    * starting at a given TAI
    * that are timestamped
* Leap second information added to timestamps provided by the FMC-DIO to the software
* Demo application that uses the interrupt generation and timestamps with leap seconds
* Possibility to mask interrupts in the FMC-DIO module

See details in repositories of submodules:

* [wr-nic-wr-starting-kit-v3.1](https://www.ohwr.org/project/wr-nic/tags)
* [fmc-bus-v2017-06](https://www.ohwr.org/project/fmc-bus/tags)
* [spec-sw-wr-starting-kit-v3.1](https://ohwr.org/project/spec-sw/tags)
* [coht-vic-v1.3.0](https://gitlab.cern.ch/cohtdrivers/coht-vic/-/tags)
* [fmc-dio-5chttla-wr-starting-kit-v3.1](https://www.ohwr.org/project/fmc-dio-5chttla/tags)
* [wrpc-v4.2](https://www.ohwr.org/project/wr-cores/wikis/Documents/WR-PTP-Core-v4.2-Files)
* [ppsi-v2016.12-91](https://www.ohwr.org/project/ppsi/tags)

### wr-starting-kit-v3.0

This release focused to update all different modules constituting WR Starting Kit to use their newer release versions. This release does not provide specific updates to the features or functionalities of the WR Starting Kit.

* Update spec-sw with new VIC core to support kernels (v4.15-v4.18)
* Use new wr-core v4.2 with their update tools (see changelog of wrc-v4.2)
* Separate NIC, FMC-DIO, SPEC-SW into different modules
* Update submodules to work with new ohwr gitlab URL
* Remove etherbone support to avoid conflict in epfilters

A detailed changelog can be found on the wiki page of each submodules:

* [wr-nic-v2.0.1-1](https://www.ohwr.org/project/wr-nic/tags)
* [fmc-bus-v2017-06](https://www.ohwr.org/project/fmc-bus/tags)
* [spec-sw-v2017-10-19](http://www.ohwr.org/projects/spec-sw/tags)
* [coht-vic-v1.2.2-3](https://www.ohwr.org/project/coht-vic/tags)
* [fmc-dio-5chttla-v3.0](https://www.ohwr.org/project/fmc-dio-5chttla/tags)
* [wrpc-v4.2](https://www.ohwr.org/project/wr-cores/wikis/Documents/WR-PTP-Core-v4.2-Files)
* [ppsi-v2016.12-91](https://www.ohwr.org/project/ppsi/tags)


### wr-starting-kit-v2.0

* The driver now support kernel from 2.8.x to 3.5.x
* A valid EEPROM data is now **mandatory** in order to start the kernel
* The gateware is now accessible remotely/standalone through ethernet using etherbone core.
* First DIO channel output is now reserved for outputting PPS
* PPSi has been introduced to improve compatibility with other PTP devices.
* GrandMaster locking has been improved
* FMC bus has been updated to the stable version which is officially included in linux kernel (>3.5)
* Calibration has been improved using t42p procedure


A detailed changelog can be found on the wiki page of each submodules:

* [wr-nic-v2.0](http://www.ohwr.org/projects/wr-nic/wiki/Release_v2)
* [spec-sw-v2014-02](http://www.ohwr.org/projects/spec-sw/wiki#version-2014-02)
* [fmc-bus-v2014-02](http://www.ohwr.org/news/352)
* [wrpc-v2.1](http://www.ohwr.org/projects/wr-cores/wiki/Wrpc_release-v21)
* [ppsi-v2013.11](http://www.ohwr.org/news/345)



System Setup
===========

This section describes the main elements required to start with the White-Rabbit based
on the *"White Rabbit (SPEC-based) Starting Kit"*. The different
hardware elements and theirs connections are described in this section.

What do you need?
----------------

In order to use the WR Starting Kit and setup the different
experiments you will need:

* An oscilloscope with at least 150Mhz bandwidth (500Mhz is recommended).
* Two PCs with at least one `PCIe x4` port (`x8` & `x16` are also compatible). The use of two PCs is recommended because in some cases there are problems using a PC with 2xPCIe depending on the kernel.
* The Operative System **Ubuntu LTS 64bit (Long Term Support)** installed.
* Two mini-USB (B) cables (not provided with the kit).

This tutorial has been tested and verified with Ubuntu LTS 16.04.6 (kernel v.4.15.9) and Ubuntu LTS 18.04.2 (kernel v.4.18.0-17), this mean that **standard support will only be given for these releases**.
However, the `spec-sw` driver should work with other releases, distributions and architectures.

> ***Notes:*** Ubuntu 18.04 LTS is supported but some versions (i.e, 18.04.3 with kernel v5.0) does not work correctly unless you [force to used a previous kernel version](#set-specific-kernel-version).
> ***Warning:*** Automatic update of Ubuntu LTS might also upgrade the kernel version to a newer one and thus break the compatibility of the WR-Starting-Kit drivers. You should [fix to the current working kernel version](#set-specific-kernel-version) or disable automatic update in order to avoid this problem.

> ***Warning:*** Ubuntu LTS 14.04 is not supported.

![Configuration with one two PCs](img/ssk_configs.png)

WR Starting Kit components
-----------------

The WR Starting Kit is composed of various elements that you should find
in the package[^standardssk] :

* 2x [SPEC]s boards
* 2x [FMC-DIO]s 5CH TTL A

* 2x SFPs LC
    * AXGE-1254-0531 (blue)
    * AXGE-3454-0531 (violet)
* 1x LC-LC cable (2m)
* 3x LEMO cable (2m)
* 3x LEMO-BNC Adaptor

![The components of the WR Starting Kit](img/ssk_components.jpg)

[^standardssk]: Only with the standard version of WR Starting Kit.


Physical setup
-------------------


* Plug the [SPEC+FMCDIO] into your `PCIe x{4,8,16}` port.
* You should also connect a mini-USB cable to the UART of the board.
(A Virtual UART is also available, but it is safer to use the physical
one).
* Connect the two [SPEC+FMCDIO] boards using the SFPs and the optical fiber cable (LC-LC)
    * In the demo we have put the violet SFP on the PC01 SPEC board (wri1)
    * ... and the blue SFP on the PC02 (wri1)
* Start Ubuntu LTS.
* Prepare an oscilloscope with at least two input channels to access
the [FMCDIO] outputs.


![Mounting two SPECs in two PC](img/ssk_inside.jpg)

![Connecting SFPs and fiber](img/ssk_sfp.jpg)

![The UART use a mini-USB cable to communicate](img/ssk_usb.jpg)


Installation
=======================

Once you have your system running with the boards plugged, you need to:

* Check that the boards has been detected
* Install the tools to build drivers, etc...
* The drivers : `spec-sw`, `fmc-dio-5chttla`, `wr-nic`


SPEC detection
----------------

First you should check that the boards has been correctly detected on your PCIe bus by doing the following:

~~~~~{.sh}
>:$ lspci | grep CERN
~~~~~~~~~~

You should obtain something similar as:

~~~~~{.sh}
01:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
~~~~~~~~~~

Where `01` represent the ID of the PCIe slot given by your motherboard.

> **Note**: The PCIe slot numbering are normally ordered from top to bottom on the motherboard,
this mean that the board with ID `01` will be at the top.


Tools
-----------------

Then you need to install all the tools that you will need:

* **git**: A powerful & distributed version control system
* **build-essentials**: Contains various binaries to build source code
* **linux-source**: Might be useful to compile drivers & kernel modules
* **minicom**: Hyperterminal for linux
* **texinfo, texlive, emacs**: Tools to build documentation
* **lib**: libreadline-dev


You can also run this command[^debian] for minimal setup:

~~~~{.sh}
sudo apt-get install git build-essential python-minimal wget \
libreadline-dev linux-headers-$(uname -r) minicom
~~~~~~~~~

and this one if you also want to generate documentation (not mandatory):

~~~~{.sh}
sudo apt-get install texinfo emacs texlive-xetex pandoc
~~~~~~~~~


[^debian]: This sample command is for debian's like distributions.
However similar packages exist for other distributions.

Compile & Install the project {#wrsk-comp-inst}
-------------------

The first step is to install the driver to communicate with the card using
the PCIe port. To do so you need to get the latest release[^commitgit] of [wr-starting-kit] project which
includes the [spec-sw] project.

~~~~{.sh}
## Create the "root" directory
>:$ mkdir -p ~/wr/
>:$ cd ~/wr/

## Clone the repository
>:$ git clone https://ohwr.org/project/wr-starting-kit.git
>:$ cd wr-starting-kit

## Checkout the stable release
>:$ git checkout -b wr-starting-kit-v3.1 wr-starting-kit-v3.1
~~~~~~~~~~~~

Now you should execute the Makefile that should perform everything!

~~~~{.sh}
>:$ make
>:$ sudo make install
~~~~~~~~~~~~

[^commitgit]: You should only use our package or proper release (tagged commit).
The **master** branch might have the latest source but support is only offered for tagged release. The other branches are normally used for development and are not stable.


Loading the driver
------------------------

The steps described in the previous section whould be done once. However, in
order to enable the wr-nic you should execute the modprobe[^errmodprobe] command
to load all kernel drivers every time computer is started:

~~~~{.sh}
>:$ sudo modprobe spec
>:$ sudo modprobe htvic
>:$ sudo modprobe wr-nic
>:$ sudo modprobe wr-dio
~~~~~~~~

[^errmodprobe]: We have found a problem in some distribution with `modprobe` command.
You should use `insmod` instead and  look at the [Frequently Added Questions sections](#frequently-added-questions).


You should expect to obtain a new interface:

~~~~{.sh}
>:$ ifconfig -a | grep -A 2 wr
wri1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::xxxx:xxxx:xxxx:xxxx  prefixlen 64  scopeid 0x20<link>
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
~~~~~~~~~~~~

If it is not the case you might have an empty EEPROM that need to be
written with proper value as explained in the next section.


Creating Valid EEPROM
----------------------

> ***Notes:*** If you already have the new interfaces wrX while doing
`ifconfig`, this mean that you can **skip** this section.

On older manufactured [FMC-DIO] (before 2014), the EEPROM was not
configured with the proper information
which is now needed by the fmc-bus to identify properly which [FMC] has been plugged
into the [SPEC].

Therefore if you try to load the kernel driver

~~~~~{.sh}
>:$ sudo modprobe spec
~~~~~~~~~~

you might obtain the following warning on `dmesg`:

~~~~~{.sh}
>:$ dmesg
...
[    7.824668] spec 0000:01:00.0: mezzanine 0
[    7.824669]       EEPROM has no FRU information
...
~~~~~~~~~~

This information, called the Field Replaceable Unit (FRU), contains the type of FMC board, its serial number, etc.
In order to write it in case you have an old FMC board without any EEPROM written,
you should execute the following commands

Don't forget to modify the bus id: `FmcDio5cha-XXXX` and the S/N
`7S-DIO-v2-Sxx-XXX` with your specific devices setup.

~~~~~{.sh}
##First you need to go in the fru-generator folder
>:$ cd wr-starting-kit/spec-sw/fmc-bus/tools/

## And compile the library
>:$ make

## Then, find out on which bus id you have the boards
>:$ ls /sys/bus/fmc/devices/
fmc-0100

### Enter root mode in the terminal
>:$ sudo bash

## Write the eeprom for fmc-0100: the first/the top one fmc board.
>:# FRU_VENDOR="CERN" FRU_NAME="FmcDio5cha" FRU_PART="EDA-02408-V2-0" \
./fru-generator -s  7S-DIO-v2-S03-058 > /sys/bus/fmc/devices/fmc-0100/eeprom

## Finally, Go back to user and starting-kit root directory
>:# exit
>:$ cd -
~~~~~~~~~~

### Reload the kernel driver

Once you have correctly written the EEPROM of the [FMC-DIO], you need
to reload the kernel driver by reboot the PC

~~~~{.sh}
>:$ sudo reboot
~~~~~~~~~~

you should now obtain something like this on `dmesg`:

~~~~~{.sh}
...
[   10.901519] spec 0000:01:00.0: mezzanine 0
[   10.901521]       Manufacturer: CERN
[   10.901521]       Product name: FmcDio5cha
...
~~~~~~~~



Setting Master & Slave using White Rabbit Core
=============================================

The "SPEC+FMCDIO" board is configured by default to be a slave node. In a White Rabbit network, slave nodes are synchronized to a master such as a Whtie Rabbit Switch ([WRS]).

In order to enable the synchronization capabilities we need to set up one board to be in master mode
(faking to be a [WRS]) and the other one in slave mode (default mode).  This is done through the
shell of White Rabbit Core that allows this functionality among others.


Connect to the UARTs
-----------------------

To configure the [SPEC] board in master/slave mode you need to access to the White Rabbit Core (WRC) through the UART (virtual or physical).

### Physical UART

To access to the physical, you first need to connect the mini-USB of the [SPEC] to your PC. Then, you need to run
 a terminal emulator[^sudomc] (such as minicom) on the device created while plug-in the USB cable.

~~~~~{.sh}
>:$ sudo minicom --device=/dev/ttyUSB0 -b 115200
>:$ sudo minicom --device=/dev/ttyUSB1 -b 115200
~~~~~~~~~~

> ***Note:*** ttyUSB0 might not correspond to your first board, and ttyUSB1 to the second one.
This depends on how you have plugged the USB cable to your USB of your machine.

> ***Note:*** When the terminal emulator starts, you might need to press enter before seeing the wrc# prompt.

> ***Note:*** In order to leave the minicom emulator, you need press the following combination of keys: CTRL+a, then z, then x and Enter.

### Virtual UART

An easier way to perform this operation is to access to the virtual
UART of the [SPEC] board by using the wrpc-vuart tool.
You first need to know the `bus_id` of your board

> ***Notes:*** The VUART can not work if the USB is already connected as the Physical UART
has the priority.

~~~~~{.sh}
>:$ lspci | grep CERN
01:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
~~~~~~~~~~

 This means that your first board (wri1) is at **01**:00.0
 Thus, for each board you can open a terminal and run the following command:

~~~~~{.sh}
>:$ sudo wrpc-vuart -f /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/resource0 -o 0x20500
~~~~~~~~~~

> ***Warning:***  Be aware that if you try to remove the spec kernel modules with the virtual UART
activated, your PC will hang and thus you will need to reboot. You must close
the virtual UART before doing this kind of operation.


[^sudomc]: On Ubuntu LTS, you need sudo permission to access to a
ttyUSB*X* device. It might be a good idea to add your user to the dialout group to
obtain appropriate permission on the device: `sudo usermod -a -G dialout $USER`


The White Rabbit Core Shell
----------------------------

Once you are in the UART you should obtain the WR PTP Core console (`wrc#`).
A complete reference of the shell commands is included in the [wrpc.pdf] manual or
you can read them in the [Wiki](https://www.ohwr.org/project/wr-cores/wikis/wrpc-core)

The most useful commands are repeated here for your convenience

* `gui`
* `mode master`
* `mode slave`
* `ptp stop`
* `ptp start`

For the tutorial we will use the following names:

* `wrc1#` for `wrc#` console of the main board (wri1/busID=0x0100) on the PC01
* `wrc2#` for `wrc#` console of the main board (wri1/busID=0x0100) on the PC02


Initial Configuration using SDBFS partition
------------------------------------

Since version 3.0 of WR PTP Core ([wrpc-core]), the on-board flash memory chip on the carrier
is used as a default place for storing calibration parameters and an init script.

The storage format of this information is organized in an SDBFS filesystem that **MUST BE** pre-formatted and
should be properly configured at least once (in case it has not been done during the manufacturing tests).

### Checking if SDBFS is properly formatted

The first thing to do is to verify that the SDBFS has been properly formatted and configured.
To do so you should check the trace messages when opening the [UART connection](#connect-to-the-uarts).

In case you see an error while WR Core is initializing such as:

~~~~~{.sh}
WR Core: starting up...
...
No SDB filesystem in i2c eeprom
~~~~~~~~~~

Or

~~~~~{.sh}
WR Core: starting up...
...
get_persistent_mac: SDB error
get_persistent_mac: SDB file is empty
get_persistent_mac: Using W1 serial number
~~~~~~~~~~

This means that SDBFS has not been founded or that SDBFS is not properly formatted (corrupted).
In this case you should format SDBFS as detailed in the [next section](#formatting-sdbfs).

In case SDBFS is found but an error message about an "empty SDB file" appears when retrieving
the MAC address of the device: you should jump to
[configuring the MAC address](#configuring-mac-address) section.

~~~~~{.sh}
WR Core: starting up...
sdbfs: found at 3014656 in Flash
...
get_persistent_mac: SDB file is empty
get_persistent_mac: Using W1 serial number
~~~~~~~~~~

If none of these warnings arise during initialization this mean that you can skip the next
steps and go directly to [configuring slave & master mode](#configure-in-slave-master-mode) section.


### Formatting SDBFS

> **Warning**: You should skip the following step if you have verified that
> the SDBFS is [properly formatted](#checking-if-sdbfs-is-properly-formatted).

Starting from v3.0 you have to write the empty
SDBFS filesystem image to the flash before running the WRPC. The simplest way of
doing this is by calling a WR PTP Core shell command:

~~~~~{.sh}
wrc# sdb fs 0
~~~~~

You should see the output similar to:

~~~~~{.sh}
filename: .                 ; first: 2e0000; last: 32007f
filename: wr-init           ; first: 2f0000; last: 2f00ff
filename: calibration       ; first: 300000; last: 30007f
filename: mac-address       ; first: 310000; last: 310005
filename: sfp-database      ; first: 320000; last: 32007f
Formatting SDBFS in Flash(0x2e0000)...
~~~~~

After formatting the SDBFS, you should reconfigure the
[MAC address](#configuring-mac-address) and the
[calibration](#calibration).


> **Note:** Other methods exists to format SDBFS with a pre-existing binary files:
> through the PCIe bus ([wrpc-sw].pdf > Appendix C), or using a Xilinx JTAG cable when
> [running in standalone](#run-in-standalone).

### Configuring MAC address

> **Note:** Configuring properly the MAC address is recommanded but not mandatory.

In case you get the messages at the initialization of WR PTP Core:

~~~~~{.sh}
WR Core: starting up...
...
get_persistent_mac: SDB file is empty
get_persistent_mac: Using W1 serial number
~~~~~

Your [SPEC] board will use an "auto-genereted" MAC address derived from the thermometer serial number (W1) that is a unique identifier. However, even if this MAC address should be indeed unique within WR network, it is not created according to IEEE rules and using  [IEEE's OUI].

You should get the MAC for your board from its manufacturer. To configure the
address and store it into the Flash/EEPROM (so that it's automatically loaded
every time the WRPC starts) you should type two commands in the shell:

~~~~~{.sh}
wrc# mac set xx:xx:xx:xx:xx:xx
wrc# mac setp xx:xx:xx:xx:xx:xx
~~~~~

where `xx:xx:xx:xx:xx:xx` is the MAC address of your board.

To verify it back, you should run

~~~~~{.sh}
wrc# mac get
~~~~~

After rebooting the machine.


Configure in slave & master mode
--------------------------------

Now we can perform the following operation on both PCs:

* Check if you have the correct SFP and its corresponding value

~~~~~{.sh}
wrc1# sfp match
AXGE-3454-0531 #purple
SFP matched, dTx=179480 dRx=153494 alpha=-68210600
~~~~~~~

~~~~~{.sh}
wrc2# sfp match
AXGE-1254-0531 #blue
SFP matched, dTx=179410 dRx=153544 alpha=67216940
~~~~~~~

> ***Notes:*** If you obtain different value or you get an error such as **Could not match to DB**, this means
> that you should add proper SFP parameters to the SPEC database. First , list the actual sfp database with `sfp show`
and then check the [Calibration Section](#calibration) that explain how to add SFPs parameters.

* Setup one board in master (the one with the purple SFP)

~~~~~{.sh}
wrc1# ptp stop
wrc1# mode master
PTP stop
Locking PLL...
~~~~~~~

* And let the other one in slave mode (since slave mode is the default
mode, this should not be necessary but you might have set it before in master mode)

~~~~~{.sh}
wrc2# ptp stop
wrc2# mode slave
PTP stop
Locking PLL
~~~~~~~

### Start the synchronization with PTP daemon


You should run the PTP daemon on both cards

~~~~~{.sh}
wrc1# ptp start
wrc2# ptp start
~~~~~~~

> ***Note:*** The PTP daemon needs to be started only after changing the mode (or stopping it with a command), otherwise it is up and running by default.

Finally you can run the WRC shell in `gui` mode to obtain more information

~~~~~{.sh}
wrc2# gui
~~~~~~~~~~

~~~~~{.sh}
WR PTP Core Sync Monitor wr-starting-kit-v3.1
Esc = exit

TAI Time:                  Thu, Jan 1, 1970, 02:01:57

Link status:
wru1: Link up   (RX: 12860, TX: 8181) IPv4: BOOTP running
Mode: WR Slave   Locked Calibrated

PTP status: slave

Synchronization status:
Servo state:               TRACK_PHASE
Phase tracking:            ON
Aux clock 0 status:        enabled

Timing parameters:
Round-trip time (mu):               684638 ps
Master-slave delay:                 354511 ps
Master PHY delays:           TX:    121751 ps, RX:     99027 ps
Slave PHY delays:            TX:         0 ps, RX:      1600 ps
Total link asymmetry:               -24384 ps
Cable rtt delay:                    462260 ps
Clock offset:                            1 ps
Phase setpoint:                        561 ps
Skew:                                    0 ps
Update counter:                          7
--
~~~~~~~~~~~~~~~~~~

You can see that the slave node is locked, calibrated and that the phase tracking is enabled.

> ***Notes:*** We also recommend you to set up an init script if you do not want to repeat these operations at each reboot. You can look at the [wrpc.pdf] for more information and use the `init show` command to check the one you have running.




Check the network interface
-------------------------------

Once the drivers are loaded and the PTP has started you might check if the new network interfaces is up.
In order to do this you should just execute:

~~~~{.sh}
>:$ sudo ifconfig | grep -A 2 wr
wri1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
~~~~~~~~~~~~


Some Illustrative Experiments
==================

To perform the experiments we recommend you to plug the LEMO cable to your
oscilloscope by using the LEMO-BNC adapter provided in your package.

It is also recommended to follow the order of the tutorial since the
concepts that are intended to be described and tested are explained in
the same order as the experiments.

> ***Notes:*** Before each experiment you should make sure that you set
up your boards in *master/slave* mode and that the PTP daemons are running on both.


Playing with the DIO channels
----------------------------------------------------------------------------------------

The first step to perform is to check if the [FMCDIO] channels are accessible using the installed driver.
The `wr-dio-cmd` program, let you quickly test the I/O features of the [FMCDIO] boards.

This is the general syntax of the command[^sudo]: 

`wr-dio-cmd: use "wr-dio-cmd [-V] <dev> <cmd> [...]"`

[^sudo]: As the application use `ioctl` you should call with root privilege by using sudo.

### Setting the mode

The first thing is to set the correct mode for each channel; you can use a command that set the
mode for a specific channel[^dionum] or for all of them at the same
time.

	mode <channel> <mode> [<channel> <mode> ...]
	mode <modech0><modech1><modech2><modech3><modech4>

The available modes are:

* `I/i`: Input
* `0/1`: Output, steady state fixed at `0` low or `1` high.
* `D/d`: DIO core output
* `P/p` (Channel 0): Output PPS (Pulse Per Second) Generator from PTP core.
* `C/c` (Channel 4): Clock Input to PTP core: allow to input a high frequency signal without interruption throwing
* `-` : This operator is used to keep the current configuration on the channel

>> ***Note:*** If the Letter is capitalized the I/O channel is connected to 50-ohm termination resistor, otherwise it is not. e.g. The `I` is input with 50-ohm termination resistor and `i` is without it.

> ***Note:*** The first channel (channel 0) supports only the P/p as output mode, it also supports input mode, i.e. I/i. You will not be able to use D,d,1,0 modes.


For example, you can use the following command:

~~~~~{.sh}
## Configure channel 1 as input with termination, 2 as input, 4 as fixed low
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode -Ii-0
~~~~~~~~~~~~~~~~



### Default Mode

After reprogramming/rebooting the FGPA, the channels are set up by default as:

1. Channel 0 (connector #1) output a PPS signal: `p`
2. Channel 1 (connector #2) low state: `0`.
3. Channel 2 (connector #3) same as ch1: low state: `0`
4. Channel 3 (connector #4) is configured as input without termination impedance: `i`.
5. Channel 4 (connector #5) is in Clock Mode with termination: `C`.

To reset to the default mode you can reset/reprogram the FPGA or set it back with:

~~~~~{.sh}
## Revert back to default mode
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode p00ic.
~~~~~~~~~~~~~~~~


### Generating a pulse and a train of pulses

The command is used in the following formats:

	pulse <channel> <duration> <when> [<period> <count>]

You should plug the LEMO cable in the connector #5 (channel 4), and connect the BNC adapter to your oscilloscope.

![Connecting LEMO on connector #5 (channel 4)](img/ssk_lemo.jpg)

Then you should set the proper datetime on the free-running master SPEC so that the slave also recover this proper date[^tai].

~~~~~{.sh}
## Forward PC datetime (UTC) to master SPEC
>:$ sudo wrpc-vuart -f /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/resource0 -o 0x20500 -c "time setsec $(date +%s)"
~~~~~~

> ***Note:*** After setting time on the master SPEC with the command above, it might be
necessary to stop/start ptp. If the `wr-dio-cmd` command does not result in generation of pulses and timestamps on the master SPEC, please open the virtual UART and stop/start ptp:

~~~~~{.sh}
>spusa:$ sudo wrpc-vuart -f /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/resource0 -o 0x20500 
wrc# ptp stop
wrc# ptp start
~~~~~~

[^tai]: PTP protocol normally use TAI timescale instead of UTC timescale, however for this example we directly forward the PC unix time (UTC).

Finally you need to active a trigger pulse in your oscilloscope. Then you can
try the following commands in the slave SPEC device:

~~~~~{.sh}
## Set channel 4 as Output
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 4 D

## Pulse channel 4 for 0.1 seconds now
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .1 now

## Pulse for 10 microseconds in the middle of the next second
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .00001 +1.5

## Pulse for 1ms at 17:00 today
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .001 $(date +%s --date 17:00)
~~~~~~~~

> ***Note:*** Please note that the `pulse` command activates the [FMCDIO] mode
> (without changing the termination)
> ***Warning:*** You must ensure that WR date is synchronized with system one.
> See setsec command in [Generating pulse](#generating-pulse) section.


If the two optional parameters `[<period> <count>]` of the `pulse` command
are specified, a train of pulses is generated. The number of pulses can
be specified using `<count>`. If this parameter is set to -1, the pulses
are generated until new configuration is specified. Setting  `<count>` to
zero stops pulse generation.

~~~~~{.sh}
## Starting now, generate train of 5 pulses on channel 4 with pulse with of 0.1 second period of 0.5 second
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .1 now 0.5 5


## Starting in the middle of the next second, generate an indefinite train of pulses on channel 4 with pulse width of 1 millisecond and period of 10 milliseconds
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .001 +1.5 .01 -1


## Update the pulse width in channel 4 from 1 to 5 millisecond
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 update_width 4 .005

## Stop the pulse generation
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .001 +1.5 .05 0


## Starting at 17:00 today, generate train of 5 pulses on channel 4 with pulse width of 2 milliseconds and period of 10 milliseconds
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .002 $(date +%s --date 17:00) 0.01 5
~~~~~~~~


### Time stamping output

Once you have generated the pulse you can retrieve its timestamp by executing:

~~~~~{.sh}
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 4
ch 4,       378.788588536 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       381.268701864 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       387.284885816 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       469.500000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       500.500000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       504.500000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       530.500000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       534.500000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,       542.500000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
...
~~~~~~~~

> ***Notes:*** If you see other timestampings messages coming from channel 2
> this mean you have not disabled the PPS mode.


### Time stamping input & output.

The idea here, is to create a pulse from one [FMCDIO] and to timestamp it as local
and remote input. Then we can compare both time stamps and measure the
interval between generation and reception of the pulse.

The configuration is done as indicated in the figure below:

![Time-stamping configuration](img/ssk_playdio.png)

> ***Note:*** We can consider PC1 as "spusa" and PC2 as "tornado"

> ***Note:*** The LEMO cables should be of identical lenght/delay. Ideally, the delay should be bellow 8ns.


~~~~~{.sh}
## Configure #2 & #5 (ch1 & ch4) as ouput and #3 (ch2) as input with termination on wri1 (spusa).
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode iDI-D

## Configure #4 (ch3) as input on wri1 (tornado)
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 3 I

## Then flush the previous timestamp (spusa and tornado)
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp &> /dev/null
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp &> /dev/null


## Schedule the pulse to the next common event on two outputs
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 1 .00001 +2
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .00001 +2

## Then (after 60s), you should run stamp on the wri1
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp
ch 1,    186789.999999992 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 2,    186790.000000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 4,    186792.999999992 (leap second: value=37, valid=1, flag59=0, flag61=0)

>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 3
ch 3,    186792.000000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
~~~~~~~~~~~~~~~~~


We can observe that the difference between the output and input timestamps is the same for the "local" and "remote" connection (i.e. "local": output=>ch1@spusa, input=>ch2@spusa, "remote": output=>ch4@spusa, input=>ch2@tornado). The resolution of the timestamps is 8 nanoseconds, so if the lenghts (delays) of the cables are below 8ns, we can observe the minimum difference between the timestamps due it its resolution, i.e. 8ns. This is a very simple example to show how the two [SPEC]s are synchronized.

[^dionum]: The connector on the [FMC-DIO] panel are enumerated from #1 to #5 and correspond to the channel 0 to 4 on the `wr-dio-cmd`.
[^8ns]: The DIO core work at a frequency of 125Mhz, therefore the minimum difference between two timestamped events is of 8ns.
However with the next experiment you will find out that the synchronization is better.

Synchronization of PPS
----------------------------------------------------------------------------------------

This experiment shows synchronization of PPS outputs between master and slave SPEC boards using WR protocol.

### Setup

In order to perform a more illustrative experiment, this example should
be done with two [SPEC+FMCDIO] boards separated by a 5km fiber
on two different PCs.
However to simplify the experiments we are using 2m fiber cable and the two boards connected on two different PCs.
The first connector (ch 0) of each board outputs the PPS and is connected to the oscilloscope using LEMO cables of the same lenght & BNC adapter.

> ***Note:*** For this experiment, the [calibration](#calibration) parameters should have been previously configured.

![Setup for the PPS synchronization](img/ssk_pps-setup.png)


### Run PPS command

Once the synchronization is enabled, you can see the PPS on channel 0
of both boards.

> ***Notes:*** To generate a PPS signal the PLL has to be lock. See "I can't generate a PPS signal" on [Frequently Added Questions](#frequently-added-questions) .

If you don't see this pulse, you might want to force the pulse mode:

~~~~~~{.sh}
## run pps on channel 0 of the "first" card (spusa)
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 0 p

## run pps on channel 0 of the "second" card (tornado)
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 0 p
~~~~~~~~~~


### Result

The following two figures show the *pulse-per-second* signals
retrieved from two different *simple-DIO* cards.

![Synchronization of the two PPS (5ns/div)](img/ssk_pps-res.jpg)

To avoid the unstable signal after the pps go high you can
add a 50-ohm resistor termination if the oscilloscope allows it.


The Network Interface Card (NIC) Synchronization
---------------------------------------

### Introduction

The *wr-nic* driver registers a Linux network interface card for
each [SPEC] device that it drives.  The cards are called `wri%d` (i.e.,
*wri1*, *wri2*, ...).

The [SPEC] can transmit to/receive from the host computer normal Ethernet traffic in addition to the WR-PTP frames (the WR-PTP frames remain invisible to the host computer).

> ***Notes:*** If a user wants to use the WR-NIC on a real network to transfer
data he should assign to the wri%d Interface an IP address. However the *White Rabbit* synchronization happens at Ethernet
layer therefore no IP address is needed. Moreover if you want to communicate between two WR-NIC SPEC boards that are on the same
computer you should not use an IP protocol because the data will never
be put on the fiber since the computer knows
that the two IPs are on the same machine.

### Timestamp Mechanism

The WR-NIC supports hardware timestamping for user frames through the
standard Linux mechanisms.  Timestamps are currently reported with
a resolution of 8ns (*White Rabbit* does much better, but the code
to illustrate this is not developed for this simple demo, yet).

Unfortunately the Linux mechanisms are not trivial: the application must
enable timestamping on both the hardware interface and the specific
socket it is using, and it must issue several *ioctl* and *setsockopt*
commands. Moreover, timestamps are returned to user space using the
*recvmsg* system call, which is more difficult to deal with than the
normal *send* or *recv* commands.

To simplify use of timestamps for Ethernet frames,
this package includes the `stamp-frame` program in the
`tools` directory.  The program is a minimal implementation of the basic
time-synchronization protocols (like NTP and PTP), but excluding the
synchronization itself. The idea is sending a frame from one host to
another, and receiving a second frame back; the departure and arrival
times are recorded and collected at a single place, so they can
be reported to
the user.

### Synchronization And Timestamping

The `stamp-frame` example supports two modes of operations. In
*listen* mode, it binds to an Ethernet interface and listens forever:
it waits for the forward frames and replies to them; in *normal* mode it
sends the forward frame and reports data as soon as it gets a reply.

~~~~~{.sh}
## On a terminal run
>spusa:$ sudo stamp-frame wri1 listen
tools/stamp-frame: Using interface wri1, with all timestamp options active

## On another terminal (maybe on another host) run
>tornado:$ sudo stamp-frame wri1
tools/stamp-frame: Using interface wr1, with all timestamp options active
timestamp    T1: 1576494173.573300256
timestamp    T2: 1576494173.573325576
timestamp    T3: 1576494173.583573592
timestamp    T4: 1576494173.583598920
round trip time:         0.000050648
forward    time:         0.000025320
backward   time:         0.000025328
~~~~~~~~~~~~

This is done four times: departure and arrival of the forward frame, followed
by departure and arrival of the backward frame. Thus,
time stamps T1 and T4 are collected at the
original sender (here: *wri1 "tornado"*) while T2 and T3 are collected at the
remote host (here: *wri1 "spusa"*).  The times above are all consistent
because the two [SPEC] cards are synchronized with *White Rabbit*. The
reported forward and backward times match the fact that we used
a 10km fiber to connect the two cards; the difference between them
is due to the different speed of light in the two directions,
because the two SFP transceivers we plugged to use different wave lengths.

The following example shows the output for two forcibly-unsynchronized
cards.  The difference between the two clocks is clearly a
few seconds; the
*round trip time* is correct nonetheless, because it is a difference of
differences:

~~~~~{.sh}
   timestamp    T1:        40.229249168
   timestamp    T2:        25.340447600
   timestamp    T3:        25.340468816
   timestamp    T4:        40.229469040
   round trip time:         0.000198656
   forward    time:       -14.888801568
   backward   time:        14.889000224
~~~~~~~~~~~~

The code in `stamp-frame`
is designed to be simple to be reused, but there is one non-obvious detail
that is worth describing here.  Whereas the receive timestamp is
returned to user-space together with the frame it refers to, the
transmit timestamp is only known after the relevant frame left the
computer.  For this reason, in order
to communicate the TX timestamp of a frame to your peer, you will need to
send another message which carries the departure time of the previous
frame.  This further message is usually called *follow-up*,
and `stamp-frame` respects this tradition.

> ***Note:*** Round trip time (RTT) is calculated with formula RTT = (T4-T1)-(T3-T2),
> Forward (FW) and Backward delays (BW) are computed using FW = T2-T1 and BW =
> T4-T3 formulas respectively. In unsynchronized scenarios FW and BW delays
> could present opposite signs because of the subtraction of timestamps from different
> devices whose time bases are unrelated.


Forwarding train of pulses
-----------------------------------

This experiment merges the concepts seen before with the [FMCDIO] channels and the NIC synchronization.

### Summary

A typical application for *White Rabbit* (or any time
synchronization system) is being able to generate output signals at the
same time in different output boards; (in the framework of a
distributed instrumentation facility); another typical application is
time stamping input events.

Thus in these experiments we are going to transmit an external pulse at frequency in the 100Hz range using the WR Starting Kit:

* The user supplies a pulse every 100Hz on the channel 0 of the master card.
* The **dio-ruler** on master host reads the UTC time of the rising edge of the external pulse upon IRQ.
* Then the **dio-ruler** forwards it (as an **ioctl** structure) to its local and remote selected outputs.
* At the slave workstation the **dio-agent** waits to receive
timestamped **ioctl** order and passes it to the *DIO* driver.
* The DIO will generate a *1ms* pulse at the programmed time.
* On the screen of the oscilloscope we should see a constant time offset between the two pulses.

The figure below illustrates how the connection must be done between the different components:

![Setup to transmit a 100Hz signal](img/ssk_100Hz.png)

> ***Note:*** The cables between DIO outputs and the scope shoud be of the same lenght.

> ***Note:*** The forwarding of pulse is fully done in software, this has been developed as a proof of concept with limited bandwith (~100Hz) and reliability (it is not recommanded to be used in production)

### How does it work?


The example is made up of two programs: `wr-dio-agent` and `wr-dio-ruler`: the ruler rules and the agent acts only.

These programs transmit using raw Ethernet frames to force the optical fiber transmission in the case when the two boards are on the same machine.
It also transmits using broadcast addresses to avoid complication on
selecting to who it should be transmitted.
The simplification adopted above, however, most likely prevents the programs from working within a more complex
network topology.

> ***Notes:*** If you are using two SPEC boards on a single PC, by using an IP protocol (i.e, UDP) the data will never enter the optical fiber between the two boards
because the OS see that the destination is local and therefore routes them directly to the receiver without using the sender.

The *agent* is a "dumb" agent in charge of forwarding *WR* packet to the *DIO* core.

The *ruler* waits for timestamps to appear on a specific input channel;
when a positive-going edge occurs the *ruler* is notified and replicates the edge on one or more outputs.
Each output can be local or remote, and can use a different delay from the input pulse.

This is the general syntax of the commands:

`wr-dio-ruler: Use "wr-dio-ruler [-V] <wr-if> <dio-dev> IN<ch> {L,R}<ch>+<delay-as-decimal> [...]`

`wr-dio-agent: Use "wr-dio-agent [-V] <wr-if> <dio-dev>"`

> ***Notes:***  All pulses generation are driven by
host software, after a hardware interrupt reports the input event.
For this reason, you will not be able to reliably replicate pulses with
delays smaller than a few hundred microseconds, depending on the
processing power of your computer and the load introduced by other
processes.  For remote connections, you must also count the overhead
of network communication as well as transmission delays over
the fiber (a 10km fiber introduces a
delay of 50 microseconds).

[^loopback]:

### The setup

Let's start by running the "dumb" agent on the slave board in charge of forwarding *ioctl* packet to the *DIO* core:

~~~~~~{.sh}
## Set remote1 as DIO output (and disable remote2)
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode -D0--

## Then, Run in background the agent
>tornado:$ sudo wr-dio-agent wri1 /dev/fmc-dio-1:0 &
~~~~~~~~~~

Then you need to connect the output of the generator[^fake100hz] to channel 1 of the master [SPEC+FMCDIO] board.
The generated waveform should be a 0-5V pulse at ~100Hz (5ms at 5V then 5ms at 0V)


[^fake100hz]: If you lack a wave form generator, you can make an output pulse with
`wr-dio-cmd <dev> mode <ch> P` or other means and use it as a trigger. Check the [Two PCs example](#two-pcs-example).



You should setup the channel 0 as input and check if the 100Hz signal is correctly timestamped:

~~~~~~{.sh}
## Set channel 0 of master board as Input and disable PPS signal (if the FPGA is reflashed).
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode I-0--

## Retrieved the time stamped value
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp
ch 0,      3573.281851462 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 0,      3573.291851460 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 0,      3573.301851460 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 0,      3573.311851456 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 0,      3573.321851454 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 0,      3573.331851458 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 0,      3573.341851454 (leap second: value=37, valid=1, flag59=0, flag61=0)
...
~~~~~~~~~~~~~~~~~~~~~~~~~~

Here we can see that we have correctly timestamped each 10ms (+/- 10ns)[^10ns].

[^10ns]: The 10 nanoseconds jitter is due to our waveform generators used in the experiments.
This should disappear by using a good quality waveform generator.


Once you know that your input is correct, you need to capture each
event on channel 0 and replicate them with a delay of 1ms
on both the local (channel 3) and the remote card (channel 1)

~~~~~~{.sh}
## Set local3 as Output
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 3 D

## Creating the ruler to forward from input 0 to local3 and remote1
>spusa:$ sudo wr-dio-ruler wri1 /dev/fmc-dio-1:0 IN0 L3+0.001 R1+0.001 &
wr-dio-ruler: configured for  local channel 3, delay 0.001000000
wr-dio-ruler: configured for remote channel 1, delay 0.001000000
~~~~~~~~~~~~~

### Result

To check if the experiment works well you can compare the timestamp of the two outputs

~~~~~~~~{.sh}
## Checking the timestamp on local output
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 3
ch 3,      5340.004864960 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 3,      5340.014864960 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 3,      5340.024864968 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 3,      5340.034864964 (leap second: value=37, valid=1, flag59=0, flag61=0)
...

## Checking the timestamp on remote output
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 1
ch 1,      5340.004864962 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 1,      5340.014864960 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 1,      5340.024864968 (leap second: value=37, valid=1, flag59=0, flag61=0)
ch 1,      5340.034864960 (leap second: value=37, valid=1, flag59=0, flag61=0)
...
~~~~~~~~~~~~~~~

Or compare the signals[^3cables] on the oscilloscope as in the next figure:

![Result with a 100Hz train](img/ssk_100Hz-train.jpg)

> ***Notes:*** You will not see timestamps from ch0 on spusa if you run the command ***stamp 0***, this is because the wr-dio-ruler reads and clears the relevant interrupts registers.

> ***Notes:*** To stop the experiment (or restart it), you can use the following commands:

~~~~~{.sh}
>spusa:$ sudo killall wr-dio-ruler
>tornado:$ sudo killall wr-dio-agent
~~~~~~~~~~~~~~~~

[^3cables]: To do this you need to connect from the waveform generator to the input with a LEMO cable that is not provided.

> ***Notes:*** This code was developed as a demo sample therefore: the delays should be no
more than the interval between input pulses, because the tools
reprograms all output triggers at each input event. The output pulse is fixed at 1ms


### Additional example without need for generator

This experiment shows how to use the *ruler* and *agent* without a generator.

![Transmit PPS between two PCs](img/ssk_txpps.png)

The input events on `spusa` are replicated to one local channel and two remote channels,
with a delay of 1ms. The input events in this case are from a *pulse-per-second* signal from
channel 0.

~~~~~{.sh}

   >spusa:# sudo wr-dio-cmd /dev/fmc-dio-1:0 mode P--DI

   >tornado:# sudo wr-dio-cmd /dev/fmc-dio-1:0 mode I-D-D
   >tornado:# sudo wr-dio-agent wri1 /dev/fmc-dio-1:0 &

   >spusa:# sudo wr-dio-ruler wri1 /dev/fmc-dio-1:0 IN4 L3+.001 R4+.001 R2+.001 &
   wr-dio-ruler: configured for  local channel 3, delay 0.001000000
   wr-dio-ruler: configured for remote channel 4, delay 0.001000000
   wr-dio-ruler: configured for remote channel 2, delay 0.001000000

   [... wait a few seconds ...]

   >spusa:# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 3
   ch 3,       385.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 3,       386.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 3,       387.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 3,       388.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   >tornado:# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 2
   ch 2,       385.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 2,       386.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 2,       387.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 2,       388.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   >tornado:# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 4
   ch 4,       385.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 4,       386.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 4,       387.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)
   ch 4,       388.001000000 (leap second: value=37, valid=1, flag59=0, flag61=0)

   ## To stop the agent and ruler
   >spusa:$ sudo killall wr-dio-ruler
   >tornado:$ sudo killall wr-dio-agent
~~~~~~~~~~~~

### Interrupt generation, disabling and demo

Apart from the 5 physical channels (0-4), the [FMCDIO] implements a virtual channel
(ch5) that allows to generate interrupts, similarly to pulses. The interrupt
generation is controlled using the `wr-dio-cmd` as follows:

`wr-dio-cmd irq <when> [<period> <count>]`

where the parameters are used identically as in the `pulse` command.

~~~~~{.sh}
## Generate an interrupt now
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 irq now

## Observe that timestamp generated by the interrupt and recorded for ch5
>:$  sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 5

## Generate a train of 100 interrupts every 10 ms starting at the next second
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 irq +1 .01 100

## Generate an indefinite train of interrupts every 10 ms starting at  17:00 today
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 irq $(date +%s --date 17:00) .01 -1

## Stop generating interrupts
>:$  sudo wr-dio-cmd /dev/fmc-dio-1:0 irq +1 .01 0

## Observe that timestamps of the interrupts are recorded for ch5
>:$  sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 5

~~~~~~~~


It is also possible to disable interrupts from the physical channels, these
interrupts are by default generated each time a pulse is transmitted or received.

~~~~~{.sh}
## Enable PPS output on ch0
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 0 p

## Observe timestamps of PPS
>:$  sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 0

## Disable interrupts on channel 0
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mask_irq 0 y

## Observe that no new timestamps are available because the interrupts on ch0 are disabled
>:$  sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 0

## Enable interrupts in channel 0
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mask_irq 0 Y

## Observe that timestamps are available on ch0 again
>:$  sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 0

~~~~~~~~

> ***Note:*** The generation of interrupt can be observed by software such as `dstat`,
if you have/installed it, you can run it as follows: `dstat -t --top-int` and
you should observe interrupts coming from `spec-1:0-gpio-irq`.


The [FMCDIO] project includes an interrupt demo software that illustrates
operation of the interrupt channel. It can be found in `fmc-dio/sw/`.
This demo software disables the interrupts of all  the physical channels
but not the one dedicated to interrupts. By default, it configures generation of
a interrupt every 100 millisecond. The tool displays the total number of
interrupts and the number of interrupts per second, as well as some statistics
information. Each time interupt occurs, the WR time of the interrupt is generated
in the hardware, this time is made available to the software. The demo 
software registers the system time when the interrupt is detected and reads
the WR time associated with the interrupt. The software then calculates 
the difference between OS time and WR time. The mean value and standard deviation 
of this time difference is displayed by the software.

~~~~~{.sh}
## Go to the folder where the demo program is located (we assume you are in the root of the wr-starting-kit)
>:$ cd fmc-dio/sw/irq_demo

## Compile the demo software
>:$ make

## Generate one interrupt each 100 ms
>:$ sudo ./irq-demo -f /dev/fmc-dio-1:0

## Generate one interrupt each 10 ms
>:$ sudo ./irq-demo -f  /dev/fmc-dio-1:0 -p 10000000

## Generate one interrupt each 10 ms using MONOTONIC time instead of REALTIME (default)
>:$ sudo ./irq-demo -f  /dev/fmc-dio-1:0 -p 10000000 -c MONOTONIC
~~~~~~~~

> ***Note:*** See more information in the [FMCDIO] manual in `fmc-dio/doc/irq-demo`.


Advanced used
==================

<!--TODO: Heat & Auto-calibration chapter -->


Connect to the WRS
----------------------

If you have a White Rabbit Switch ([WRS]), you should try to see
the synchronization between the [WRS] and the two [SPEC+FMCDIO] boards.

First you need to set up the two [SPEC+FMCDIO]s as `slave` by changing the
mode in the UART of each board.

~~~~~{.sh}
wrc# ptp stop
wrc# mode slave
slave
wrc# ptp start
~~~~~~~

Then you should run the [WRS] in master or grandmaster mode as explained
 in (Resources) [wr-switch-guide.pdf].

Finally you need to connect the slave SFPs (blue) to the [SPEC]s and the
 master SFP (violet) to the switch.

The result of this experiment is illustrated by the next figure where
you can see the 3 PPS signals: yellow for [WRS], blue and violet for
[SPEC+FMCDIO].

![WRS PPS (yellow) vs WR Starting Kit PPS (blue & violet)](img/ssk_wrspps.jpg)

The skew between each PPS might be reduced by the improving the
calibration which is the subject of the next section.


Calibration
-----------------------

When you compare the PPS of two [SPEC+FMCDIO] board you will find that they
are almost always synchronized without the need to calibrate the
gateware. However if you compare these PPS with the one of the switch
and you do not obtain a delay similar as the one in the WRS figure above
you might need to calibrate the [WRS] and/or the [SPEC+FMCDIO].

The wr-starting-kit-v3.1 release has been calibrated for you for the SFPs
provided to you. The values of the calibration parameters that you need to use
with the wr-starting-kit-v3.1 can be found in the wiki page:
<https://ohwr.org/project/white-rabbit/wikis/calibration#sfp-database-for-wr-starting-kit>

If you are using a release version of the WRS, it has been also calibrated for
the provided SFPs. On the other hand, if you use a custom gateware or SFPs that
are not listed in the wiki page you should run yourself the [wr-calibration]
process in order to obtain the correct values for your setup.

> ***Warning:*** Cable length/type will add around ~5ns/meter to the delay
between the moment when the PPS was out of the board and when it was
captured by the oscilloscope.

> ***Note:*** Please check that you are using the values that correspond
to your gateware release and the hardware version of SPEC/FMCDIO. We reference
the gateware corresponding to this version of the document as
wr-starting-kit-v3.1. Specific hardware versions are 4.1 for SPEC and 2.0 for
FMCDIO. You can check the version of the gateware you are running with the command ver".

~~~~~{.sh}
wrc# ver
WR Core build: wr-starting-kit-v3.1
...

## List your actual sfp database
wrc# sfp show

## Erase the actual database if it was corrupted
wrc# sfp erase

## Add the parameters for the SFP provided in the WR Starting Kit
wrc# sfp add AXGE-1254-0531 180725 151965 67216940
wrc# sfp add AXGE-3454-0531 180795 151861 -68210600

## Readback value written in the database
wrc# sfp show
1: PN:AXGE-1254-0531   dTx:   180725 dRx:   151965 alpha:  67216940
2: PN:AXGE-3454-0531   dTx:   180795 dRx:   151861 alpha: -68210600

## Finally match SFP and load corresponding calibration value
wrc# sfp match
AXGE-1254-0531 or AXGE-3454-0531
...
~~~~~~~~~~~

Do not forget to also add the proper calibration values to your other
WR equipments (i.e, [WRS]) as explained in the wiki page.

You might also check the [wr-calibration] manual for extended explanations on how to run
yourself the calibration procedure.

SPEC+DIO as grandmaster
------------------------

In basic configuration your Master [SPEC] can use its internal
free-running oscillator as a time reference. However, you can also
discipline your Master [SPEC] with external 10 MHz and 1-PPS signal by
connecting them to appropiate LEMO connectors of [FMC-DIO] board (note: channel
0 is the first one):

* connected 10 MHz to channel 4.
* connected 1-PPS to channel 3.


![Grandmaster setup](img/ssk_grandmaster.jpg)

The requirement for the applied signals are:

* ~[1.5V to 3V][^aroundval] without termination
* ~[2.5V to 4V] with 50Ohm termination.
* PPS pulse width must be at least one 10MHz period (>100ns).

For more information about grandmaster mode you can take a look at: [wr_external_reference.pdf].
This document has been written for the WR switch, but timing/accuracy/stability requirements are
similar for the [SPEC+FMCDIO].

[^aroundval]: These values are given on an indicative basis because they depend on the source and
connection you are using.

### Using a GPS

The common way to use grandmaster mode is with a GPS source where the
input 10MHz signal can be square or sinusoidal.

When the GrandMaster is a [SPEC+FMCDIO] `PPS_in` pulse serves to *mark* which rising edge of the 10MHz signal is considered the first one valid for a new second: the `PPS_out` will be fixed on the `10MHz_in` and not the `PPS_in`. `PPS_in` should therefore arrives at least 8ns before the `10MHz_in` clock.

> ***Note:*** In the case when PPS_in and 10MHz_in are perfectly aligned at your source, you can use different lenghts of cables to provide proper skew between the two signals. As 1m is around 5ns of delay, you can use a cable of 1m length (5ns) to connect PPS_in and a cable of 3m lenght (15ns) to connect 10MHz.

The following figure shows what kind of signals need to be provided to plug to the GM [SPEC+FMCDIO], and with which fixed delay the `PPS_out` is produced (~8ns).

![Grandmaster signals from GPS with different cables length (PPS in=>Yellow,
10MHz CMOS=> Blue, 10MHz Sin => Green, PPS out=> pink)](img/ssk_gm-4m1m.jpg)

### Locking GM PLL to external 10MHz/PPS

To enable Grandmaster mode, in the wrc console of the [SPEC+FMCDIO] just execute the following commands:

~~~~~~{.sh}
wrc# ptp stop
wrc# mode gm
wrc# ptp start
~~~~~~~~~~~~~

And you should obtain:

~~~~~~{.sh}
PLL locking .................. LOCKED
~~~~~~~~~~~~~

By default, the channels of the FMCDIO are configured such that the SPEC+FMCDIO can act ask a GM. If you are not sure whether your current configuration is correct, please reset to default, see [Default Mode](#default-mode) Section. If you
want to run the [SPEC+FMCDIO] in standalone mode as grandmaster we also
recommand you to modify the startup script as explained in the [section below](#run-in-standalone)


Run in standalone
----------------------------

> **Note**: You need to have the Xilinx JTAG USB platform cable or similar
 to perform the next step.

You can run GM in a full standalone mode in order to transmit WR clock without the need of a PC.

First you need to flash the SPEC with the latest `wr_nic_dio.bit` bitstream you can find in the package:
[wr-starting-kit-v3.1_bin.tar.gz]

And you also need to download a pre-formatted sdbfs partition: [sdbfs-standalone-160812.bin]

Then you need to create a `mcs` image that contains both the SDBFS partition and FPGA bitstream.
This can be done by following the layout below:


| Address    | Binary file                     |
|------------|---------------------------------|
| `0x000000` | `wr_nic_dio.bit`                |
| `0x170000` | `sdbfs-standalone-160812.bin`   |


For example, to generate the `*.mcs` file for `M25P32` Flash on SPEC, the
following `promgen` (ISE Tool) parameters should be used:

~~~~~{.sh}
promgen -w -spi -p mcs -c FF -s 32768 -u 0 <your_bitstream>.bit -bd sdbfs-standalone-160812.bin start 0x170000 -o output.mcs
~~~~~~~~~~

Finally:

* The setup of the cable is exactly the same as above, and you do not need to update the configuration of DIOFMC inputs because it is correctly configured for GM by default at power up.
* Then you need to run grandmaster mode from wrc console, as above.
* (Optional) Finally, if you need to keep this configuration at power up, you can write a small script
on the EEPROM to boot in grandmaster mode.

> **Note**: In the case when you want to run on the SPEC a reference bitstream provided with a stable
WRPC release, you can simply program your Flash with \textit{spec\_top.mcs}
provided with the release binaries using for example Xilinx ISE Impact tool.

### Standalone GM with EEPROM boot script

By default the standalone SPEC+FMCDIO will be configured as slave and output a PPS on the first channel.

In case you need to used it as a standalone GM please connect the cables as in section 6.3 and connect the UART (section x.x) to reach the wrc console
and set the mode as gm.

~~~~~~{.sh}
wrc# ptp stop
wrc# mode gm
wrc# ptp start
~~~~~~

Finally, if you need to keep this configuration at power up, you can write an
initialization procedure on the EEPROM to boot in grandmaster mode:

~~~~~~{.sh}
wrc# init erase
wrc# init add ptp stop
wrc# init add sfp detect
wrc# init add sfp match
wrc# init add mode gm
wrc# init add ptp start
~~~~~~

Please refer to *Writing EEPROM and calibration* Section of the [wrpc.pdf] document.


\clearpage

Quick Start Guide For Developers
=================================


Introduction
--------------


This section has the purpose to give the developers a quick start guide to install
the tools and compile the different projects to obtain the binaries used previously.

If you need a more complete guide on how to create your own HDL for the [SPEC],
or you want to improve the structure of a project using the [OHWR] platform we recommend you
to look at the [spec-getting-started.pdf] guide.

The [WR Starting Kit][wr-starting-kit] is based on various projects:

* [spec-sw], The project that contains the PCIe driver and tools to access to the SPEC FPGA
* [spec-golden], A simple gateware in order to access to the EEPROM of the [FMC] (WB-I2C)
* [wr-nic], gateware (FPGA HDL) that includes the NIC capabilities and its corresponding kernel drivers
* [fmc-dio-5chttla], gateware (FPGA HDL) that includes the DIO capabilities (with NIC) and its corresponding kernel drivers
* [wrpc-sw], LM32 software in the White Rabbit PTP core for the synchronization
* [coht-vic], Kernel driver to handle interruption from FPGA such as the one from SPEC

> ***Note:*** Next sections describe manual procedures to generate software and
> FPGA firmware binary files. All relative path in their code listings are refered
> to main folder of the WR Starting Kit project. WR Starting Kit repository
> installation guidelines can be found in [Compile & Install the
> project](#wrsk-comp-inst) section.

In the following figure you can understand better how the different subproject are connected and included in the WR-starting-kit project.

![Structure of repositories](img/repos_structure.png)

If you want to get more information you can take a look at `doc/starting-kit_repos-v1.3.pdf` to get a
detailed view on how the subproject are structured.



Tools
----------------

In order to generate from scratch all the binaries provided by the starting-kit,
you must have installed the following tools:

* Xilinx ISE (>14.x)
* Hdl make (v3.2)
* Texinfo
* Git
* Build-essentials
* Autoconf/Automake
* Kernel source
* Lm32 cross compiler

> ***Notes***: Many of the tools migh be already installed or will be installed if you follow all the steps of this guide.
> However, we do no provide any instruction on how to install Xilinx ISE.

WRPC-SW (LM32 firmware)
-----------------------

> *Dep*: lm32 compiler

> ***Notes***: The steps in this section are not needed as we already provide the `wrc.ram` in the [wr-nic] repository.
However we provide here a simple resume of the steps required to compile the firmware specifically for the wr-nic.
You should also look at the [wrpc.pdf] to understand how to use it and
how to compile new firmware for other configurations.


You first need to install the **lm32** compiler as suggested in
[wrpc.pdf], and then you need to compile it using the specific configuration as bellow:

~~~~~~{.sh}
## Set up CROSS_COMPILE variable for this terminal
export CROSS_COMPILE="<your_path_to_lm32>/lm32/bin/lm32-elf-";

## Move to wrpc-sw inside wr-starting-kit
cd wrpc-sw

## Configuring the project for SPEC as NIC card
make wrnic_defconfig

## Compile
make
~~~~~~~~~~

You should obtain various files named wrc.bin, wrc.elf, wrc.vhd, wrc.ram

WR-SPEC-DIO-NIC (HDL-gateware)
----------------------

> *Dep*: hdlmake v3.2, Xilinx ISE 14

This step shows how to prepare the WR-SPEC-DIO-NIC bitstream ([SPEC+FMCDIO]) with
the wrpc-sw (`wrc.ram` file) embedded inside.

The first thing to do is to ensure that hdlmake is properly installed

~~~~~~{.sh}
$ hdlmake -v
hdlmake 3.2
~~~~~~

If it is not the case you should visit [hdlmake] project, and follows the
instruction to install by doing a checkout to version 3.2 (tag `3.2` or commit `#c4b5fce`)


~~~~~~{.bash}
## Initialize git submodule for fmc-dio project
$ cd fmd-dio
$ git submodule init
$ git submodule update

## Goto synthesis folder
$ cd hdl/syn/dio-nic

## Synthetize using hdlmake
$ hdlmake
$ make
~~~~~~~~~~~

This should produce a `.xise` project that you can synthetize with Xilinx ISE 14.

You should finally obtain the bitstream to import in your [FMC] driver folder.

Set Specific Kernel Version
===========================

This chapter describes how to set a specific (previous) kernel version if you have problems while compiling/loading the wr-starting-kit
drivers with newest/latest kernel[^ubuntultskernel].

> ***Note:*** We recommand to fix your kernel to `4.18.xxx`, as it has been the one that have reported less error during development.


[^ubuntultskernel]: We usually support the current Ubuntu LTS at the moment of writing this document, however during the LTS cycle there are
minor release that might contains a newer kernel incompatible with our drivers: <https://wiki.ubuntu.com/Kernel/Support>.


Find the ID of the submenu
-------------------------------------------------

We first need to obtain the `$menuentry_id_option` for the current submenu:

~~~~~~
>:$ grep submenu /boot/grub/grub.cfg
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
~~~~~~~~~~~

List available kernel entries
-----------------------


~~~~~~
>:$ grep -A100 submenu  /boot/grub/grub.cfg | grep menuentry
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
	menuentry 'Ubuntu, with Linux 5.0.0-32-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.0.0-32-generic-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
	menuentry 'Ubuntu, with Linux 5.0.0-32-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.0.0-32-generic-recovery-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
	menuentry 'Ubuntu, with Linux 4.18.0-17-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.18.0-17-generic-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
	menuentry 'Ubuntu, with Linux 4.18.0-17-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.18.0-17-generic-recovery-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
	menuentry 'Ubuntu, with Linux 4.15.0-46-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-46-generic-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
	menuentry 'Ubuntu, with Linux 4.15.0-46-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-46-generic-recovery-541c21ee-e687-4a1e-9f8c-c1b6285694da' {
~~~~~~~~~~




Create the default menu entry
-----------------------------

The parent entry id for Advanced options in this case is `gnulinux-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da`

Choose the entry for the kernel you want to use. For example, `gnulinux-4.18.0-17-generic-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da`

Concatenate both entries with `>` The result should be like:
`"gnulinux-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da > gnulinux-4.18.0-17-generic-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da"`


Edit Grub
----------

Edit the file `/etc/default/grub` and replace `GRUB_DEFAULT` with the previously created entry. The file would look like

~~~~~~{.sh}
GRUB_DEFAULT="gnulinux-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da > gnulinux-4.18.0-17-generic-advanced-541c21ee-e687-4a1e-9f8c-c1b6285694da"
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""
~~~~~~~~~~~

Update Grub and Reboot
----------------------

~~~~~~{.sh}
>:$ sudo update-grub
>:$ sudo reboot
~~~~~~~~~~~

Frequently Added Questions
===========================

Online & Updated
-----------------

The FAQ is updated on a regular basis on the ohwr.org website so
we strongly recommand you to visit the webpage:

* <http://www.ohwr.org/projects/wr-starting-kit/wiki/FAQ>

and you might also find information there:

* <http://www.ohwr.org/projects/spec/wiki/FAQ>
* <https://www.ohwr.org/project/wr-cores/wikis/home>


Resume
------------------


However, you will find a resume of the most asked questions below:

### I can't generate a PPS signal or any point in section 5 is not working

In order to generate any signals from section 5 you need to ensure that the main PLL has been locked. That means your card has to be either in the a valid Free-running Master, Grand Master or Slave Mode.

* When you have only one card (or some but unconnected) it should be working as Free Running Master, and you should always be locked to the main PLL.
* If you have two of them (or more), one of them has to work as master/grandmaster, the others as Slave and must be locked (TRACK_PHASE) to its WR master as explained en section [4.3](#configure-in-slave-master-mode).
* If you have configured your card in Grand Master Mode you need to be sure that you provide a valid external reference (10MHz/PPS).

If you want to check that the main PLL has been locked you should connect to the UART/VUART and run the following command

### I don't see the interface using ifconfig

Please check if you see them using `ifconfig -a`, if you see them by using the `-a` option you just need
to bring them up. This is explained in the [Check the network interface](#Check-the-network-interface) section.


### I can't detect any level on my inputs.

You should first try to desactivate the termination resistance that can
consume most of the current if this one is really low.

Otherwise you should configure the DAC to set up a different threshold, however this option is not actually implemented.


### The link is down, but everything is connected

*The SFPs and the optical fiber are plugged but the error message appears `Link down --`
and you can not see any green LED on the [SPEC].*

Please check that the SFP connector is totally plugged into the [SPEC]. The SFP cage of the [SPEC]
can be blocked sometimes by the PC back-panel. You must push the SFP
until hearing a "clicking noise".

### What is the latency in the WR-NIC

Many people ask us about the latency in the WR-NIC.

The answer depends really on what you want to measure: The only "relevant" delay might be the time between the SFP and the PCIe which is less than 20 cycles.
However the delay introduced by your nondeterministic OS is much more important.

### Why am I getting "Warning: tx timestamp never became available" message form the UART?

This warning seems to appear after uninstalling wr-nic driver if PTP has already been started. It can be easily solved by reinstalling the driver.

After that the device keeps working as was previously specified.

### No SFP in EEPROM database

When you try to match your SFPs and you obtain an error such as **Could not match to DB** you
need to add your own SFP parameters. Please refer to the [Calibration Section](#calibration)
to understand how to add correct SFPs parameters.


### Modprobe is not working

If you get and error like the following:

~~~~~~~{.sh}
>:$ sudo modprobe wr-nic
FATAL: Module wr_nic not found.
~~~~~~~~~

Be sure that you have run the `sudo depmod -a` command.
Otherwise you can still load the kernel modules with insmod using the
following commands:

~~~~~~~{.sh}
>:$ sudo insmod kernel/fmc.ko
>:$ sudo insmod kernel/htvic.ko
>:$ sudo insmod kernel/spec.ko
>:$ sudo insmod kernel/wr-nic.ko
>:$ sudo insmod kernel/wr-dio.ko
~~~~~~~~~~~~~

### The system hangs-up

Unfortunately, in some circumstances the system might hang during or
 slightly after firmware loading.
Please first check that you were not using the virtual UART while
 removing/inserting kernel modules.

Please help us to solve this error by sending us a log with the following information:

* What is your system & distribution  (`uname -a`)
* All the commands executed from login to the crash
* The serial log starting from the login (dmesg)



References
============

* [spec-sw.pdf] main documentation of the spec-sw
* [wrpc.pdf] White Rabbit Core documentation
* SFPs information <http://www.ohwr.org/projects/white-rabbit/wiki/SFP>
* [wr-calibration] White Rabbit Calibration
* [spec-2-spec]  SPEC-2-SPEC demo
* [wr_external_reference.pdf] How to use the grandmaster mode in the switch.
* [spec-getting-started.pdf] A tutorial to get ready to work with the (SPEC), including hardware deployment instructions, full required toolchain setup and a collection of step-by-step demonstrative tutorials.
* [spec-sw] Sub-project with  that contains the PCIe driver and tools to access to the SPEC FPGA
* [wr-nic] Sub-project with gateware (FPGA HDL) that includes the NIC capabilities and its corresponding kernel drivers
* [fmc-dio-5chttla] Sub-project with gateware (FPGA HDL) that includes the DIO capabilities (with NIC) and its corresponding kernel drivers
* [wrpc-sw] Sub-project with the LM32 software in the White Rabbit PTP core for the synchronization



<!-- List of links -->

[WR]:https://www.ohwr.org/project/white-rabbit/wikis/home
[WRS]: https://sevensols.com/index.php/products/white-rabbit-switch/
[WRSSK]: http://www.sevensols.com/en/products/wr-starting-kit.html
[SPEC+FMCDIO]: http://sevensols.com/index.php/kit-spec/
[SPEC]: http://sevensols.com/index.php/products/spec/
[FMC]: http://www.ohwr.org/projects/fmc-projects
[FMC-DIO]: http://sevensols.com/index.php/fmc-dio/
[FMCDIO]: http://sevensols.com/index.php/fmc-dio/
[FMC fine-delay]: https://sevensols.com/index.php/fmc-del/
[OHWR]: http://www.ohwr.org/projects/white-rabbit
[Seven Solutions]: http://www.sevensols.com
[IEEE's OUI]: https://standards.ieee.org/products-services/regauth/index.html

[wr-switch-guide.pdf]: https://sevensols.com/index.php/products/white-rabbit-switch/
[wr-nic]: http://www.ohwr.org/projects/wr-nic/
[wr-starting-kit]: http://www.ohwr.org/projects/wr-starting-kit/
[spec-sw]: http://www.ohwr.org/projects/spec-sw/
[wrpc-sw]: http://www.ohwr.org/projects/wrpc-sw/
[wrpc-core]: https://www.ohwr.org/project/wr-cores/wikis/wrpc-core
[fmc-dio-5chttla]: https://ohwr.org/project/fmc-dio-5chttla/
[coht-vic]: https://ohwr.org/project/coht-vic/
[hdlmake]: https://ohwr.org/project/hdl-make/wikis/home

[spec-2-spec]: http://www.ohwr.org/projects/wr-cores/wiki/spec-to-spec
[spec-getting-started.pdf]:http://www.ohwr.org/projects/spec-getting-started/wiki
[wr-nic.pdf]: http://www.sevensols.com/dl/wr-nic/doc/latest_stable.pdf
[fmc-bus.pdf]: http://www.ohwr.org/attachments/download/2685/fmc-bus-2014-02-release.pdf
[spec-sw.pdf]:https://www.ohwr.org/project/spec-sw/uploads/1893da4fbb480352be53888d535fd140/spec-sw-2014-02-release.pdf
[wr_external_reference.pdf]: http://www.ohwr.org/attachments/1647/wr_external_reference.pdf
[wrpc.pdf]: https://www.ohwr.org/project/wr-cores/uploads/ca546161d83fb3e48cd9f0266376377b/wrpc-user-manual-v4.2.pdf
[wrpc-v2.1.pdf]: http://www.ohwr.org/attachments/2559/wrpc-v2.1.pdf

[wr-starting-kit-v3.1_bin.tar.gz]: <https://ohwr.org/project/wr-starting-kit/wikis/uploads/0858d4af81f393459a1a2aa3f981c913/wr-starting-kit-v3.1_bin.tar.gz>
[spec-golden]: https://www.ohwr.org/project/spec-sw/uploads/636e7eaa2e1c3a884e2f9f694fdfd1b9/spec-init.bin-2015-09-18
[sdbfs-standalone-160812.bin]: <https://www.ohwr.org/project/wr-cores/uploads/9dad995e74a3d49b472b8654833be01a/sdbfs-standalone-160812.bin>

[wr-calibration]: <http://www.ohwr.org/documents/213>
