Q: What are the advantages of using external GPS signals or are there any disadvantages of not using them and leave the connectors open?
A: You mainly get two things:
A more stable oscillator for the master switch and therefore for all
A GPS/UTC/TAI-related time scale. This is important if you want to
relate your events to official time or if you want to avoid
re-starting your time scale at 1 Jan 1970 (or something like that)
every time you reboot the switch. Two events seen in a WR network at
very different times could have very similar time stamps if there
was a switch reboot in between. Syncing to GPS time on reboot avoids
Q: Is it possible to use White Rabbit systems in different locations and be able to have synchronised time via Satellite link?
A: Basically as long as the 10 MHz and PPS that you give to the
different White Rabbit systems are synchronised, WR will keep its
For example, you may synchronise two WR networks by at both ends letting
a GPS generate the PPS and 10MHz signals, but you will be limited by the
precision the GPS will give you on those two different locations. This
may be as good as 3 ns already or maybe better in combination with
Or you can synchronise your systems with Time and Frequency Transfer
using the phase of the GPS
Carrier, as CERN has done
with the CNGS experiment where we could, after the fact, synchronise
related events happening on both sites. The article Time transfer
techniques for the synchronisation between CERN and
LNGS explains this.
If you have Satellite links, you can use Two-Way Satellite Time Transfer
(TWSTT) and have on both sides a
system that will provide the PPS and 10 MHz signals. According to the
presentation Time transfer through optical
page 3 from Mrs. Amy-Klein from LPL, Paris, this can provide an accuracy
better than 1 ns.
Using Time Transfer by Laser Link (T2L2) would give you even a sub-ns
The NEAT-FT Workshop on Optical Networks for Accurate Time and
Frequency Transfer held in 2012 may
give you more ideas and you possibly can find partner institutes that
can help you with such a
Q: What is the required input level for the 10MHz input on the WRS as a grandmaster?
A: 0 dBm is fine for the logic inside the switch. It is better an higher
voltage if you have a sine wave due to better slew rate (noise
immunity). We used about 10 dBm sine wave with excellent results. The
input is terminated with a 50 Ohm, 1% resistor.
10 MHz Input power:
Q: What is the required input level for the PPS input on the WRS as a grandmaster?
A: The input is not terminated with a 50 Ohm resistor; it has around 5
kOhm input impedance.
PPS input levels:
PPS Input Minimum: 2.1V (on a 50-ohm load)
PPS Input Recommended: 2.5V-5V (on a 50-ohm load)
PPS Input Absolute Maximum: 7 V (on a 50-ohm load)
(Note that the SCB v3.3 has no 50 ohm termination. The SCB v3.4 has a 50
ohm termination that can be switched in with the appropriate gateware.
However, this gateware is not implemented (10 March 2017))
Q: When connecting the GPS-PPS or the 10 MHz GPS clock to the WhiteRabbit Switch, do I have to change/configure anything?
Q: How do I get the time-of-day (ToD) into the WRS Grandmaster?
A: The preferred way of synchronizing your WRS to an external reference
is to use 1-PPS & 10MHz from a GPSDO/Cesium/etc. and external NTP server
for time of day information. The WRS User
describes in section "3.4 Configuration Items that Apply at Run Time"
how you can set the IP address of the NTP
Q: We need to use 30m coaxial cable from the GPS to the WRS Grandmaster 10MHz/PPS signals and the cable will see strong temperature changes (15-35ºC).
Temperature effects are typically very slow phenomena. The first
question would be: do they care? Quite often people want to synchronize
nodes among themselves, to a common reference, but they don't mind too
much if that reference slips a bit with respect to for example GPS time.
As you can see in
thermal coefficient of delay (TCD) expressed in (ppm/deg C) is itself a
function of temperature. Taking a ballpark number of 100 ppm/degC for
coax, we are talking about delay changes in the order of 0.3 ns. Would
your client care if the reference slips slowly by 1 ns or so? Are they
using this reference for something else than feeding it into the
Grandmaster WR switch? If this phase slip is a problem, then you could
explore coax cables not using a teflon dielectric
"":http://przyrbwn.icm.edu.pl/APP/PDF/119/a119z4p17.pdf or even
active control of the temperature by enclosing the coax inside
double-wall tubing and filling the space within the walls with
Another possibility, if the temperature is kind of uniform along the
cable length, is to characterize the temperature dependency of the very
cable to be installed in a climatic chamber and then measuring
temperature in the real system and applying phase shifts digitally.
Management summary: LCF38-50J is most stable from the four cable
types characterised. Take care of the teflon knee.
The LCF38-50J is a rather thick (3/8") and possibly cumbersome
(because rigid) cable type. If you want a fixed cable in your
infrastructure it is recommendable though, since it is not
excessively expensive. However, it only comes with N-connectors,
fortunately those typically do have good phase stability.
Authors: Javier Serrano/CERN, Tjeerd Pinkert/VU
Q: The last paragraph of the Note on using WR Switch in Grandmaster mode mentions HAL messages and RT log. I find HAL messages but what is meant by RT log and where to find it?
A: By the RT (i.e. realtime) log mentioned in the Note on using WR
Switch in Grandmaster
we mean the messages generated by the LM32 CPU inside the Virtex FPGA.
They may provide useful information for debugging the timing subsystem.
They can be viewed by connecting a serial terminal to the the front left
USB port (up to v. 3.2 switches) or the "FPGA test" USB port in the rear
(newer switches). RS232 parameters are 115200,
Q: Is the WR Switch inter-operable with standard PTP (IEEE1588-2008) gear?
A: Yes, although you would completely lose the sub-ns timing resolution
of White Rabbit, a WR switch is interoperable with PTP and it has been
tested for inter-operability on ISCPS PlugFest, but
it is important to remember that WR devices use a very specific PTP
configuration (2-step, delay request-response mechanism, multicast,
Layer 2) which is not the most popular configuration and it is not
the default setting for standard-PTP devices (which use IP instead
of Layer 2). So it is not just plug&play, it's more like
1. buy PTP switch which supports fully the standard
2. configure it properly (change from IP to Layer 2)
3. plug &
Q: Can I connect a standard non-WR GbE NIC or Switch/router to the WR Switch?
A: Yes. You can use a WR-network for standard datatransfer, using
standard Ethernet hardware. It works fine since a WR switch is a
Q: Can you have a different vendor (non-WR) switch between it and another white rabbit device to distribute the time? Around a larger network for instance?
A: Yes, you can connect non-WR switches in a WR network. But it does not
make sense as in that case you completely lose the sub-ns timing
resolution of White Rabbit.
If the switch between the WR master (switch) and the WR end node is a
standard PTP one, your timing resolution will be reduced to the
relatively low PTP precision. The reason is that the White Rabbit
protocol is achieved by synchronizing each link between one end and the
other end of the fibre and the non-WR switch in the middle would not
support the WR protocol.
To say it in another way: you may mix non-WR and WR switches to build
your complete network (it will perfectly transfer your data) but to have
WR precision all switches between your WR timing master switch and your
WR nodes should be WR-compliant ones.
Ah, when you speak about different vendor, I assume you meant a
non-White Rabbit switch. But you may want to know that as the design of
the WR switch is open, there may in due time be more producers of White
Q: I was wondering if WR has plans to use SyncE as a standard for clock recovery, or if the custom clock recovery core in WR is as accurate (or better) for clock recovery? if WR meets the jitter SyncE specifications, what is the difference in the clock recovery method?
A: SyncE and WR are very similar in terms of clock recovery. What WR
does additionally is to implement a phase shift on the recovered clock
signal in the slave. This shift tracks changes in delay in the medium,
so as to guarantee constant (ideally zero) phase with respect to the
master clock signal. With SyncE, if the transmission delay in your fiber
changes, the phase of the clock signal you recover also drifts with
respect to that in your master. What's more, the SyncE standard
comprises of two elements:
slow protocol to configure the frequency distribution hierarchy in
specification of timing characteristics of synchronous Ethernet
equipment (e.g. ITU-T G.8262)
In WR, the protocol (1) is not used which is rather uncommon in SyncE
networks but it is allowed by SyncE. The timing characteristics (2) of
SyncE have a lot of legacy burden (compatibility with SDH).
Consequently, SyncE equipment must be tolerant to quite "bad" clock
input. WR is not tolerant to the corner-cases required by SyncE, see the
report on SyncE Characteristics of a White Rabbit
. Making WR PLL tolerant to clock characteristics required by ITU-T
G.8262 might result in deterioration of WR performance (e.g. requires
change in PLL bandwidth) or require a much more expensive oscillator.
ITU-T itself recognized that the SyncE specification is outdated and
most of SyncE equipment can do much better than the specifications.
Therefore, ITU-T is working on enhanced SyncE
There is some consultation between ITU-T and WR on the eEEC, though with
low priority, at the moment it seems that eEEC will not be directly
usable with WR.
Q: Can I increase the PTP packet rate in the WR Switch?
A: Yes. By default the Announce-message rate is 1 packet every 2 seconds
and the Sync-message rate is 1 packet/second. The packet rate is
calculated in the software using the formula 1/2^log_value (e.g. if
log_value is -4 the packet rate will be 16 pkts/s). To increase the
Sync-message rate you can modify the log_value of the Sync-message by
in the ppsi.conf file (e.g. sync-interval -4 to increase the
Sync-message rate to 18 pkts/s). The Announce-message rate is software
limited from accepting negative log_values. If you want to increase the
Announce-message rate you will have to comment out or modify lines 193
to 197 in wr-switch-sw/userspace/ppsi/lib/conf.c, recompile the PTP
and update the firmware on your WR Switch. After these steps you can add
in the ppsi.conf file (e.g. announce-interval -3 to increase the
Announce-message packet rate to 8 pkts/s). The log_value is software
limited to a minimum value of -4. Please note that these values are
global and all your WR Switch ports will have the packet rate modified.
A: Yes, the switch can handle frames larger than 1500 Bytes, so-called
Jumbo Frames. But you'll
have to enable it first. You would need to enable giant frames reception
on a WR switch port where you're sending frames larger than 1500 Bytes.
There is no software tool in the current release that will do this for
you, so you would need to manually write to a proper register using
devmem. Please see page 9 (for endpoint base addresses) and page 33 (for
bit A_GIANT) of the White Rabbit Switch HDL software interface
E.g. for port 3 (endpoint no. 2) you would write to address
KM3NeT uses the following commands to enable Jumbo frames on all
#Enable Jumbo frames on all ports (customising the MRU to 9000B) - In total 18 writesdevmem 0x10030008 32 0x00800002devmem 0x10030408 32 0x00800002devmem 0x10030808 32 0x00800002devmem 0x10030c08 32 0x00800002devmem 0x10031008 32 0x00800002...devmem 0x10034008 32 0x00800002devmem 0x10034408 32 0x00800002
Author: Greg Daniluk, Diego Real, Erik van der Bij,
Q: What is a possible problem if I see losses of Ethernet Frames when connecting to non-WR devices, even with reasonable data rates?
A: A possible reason for this can be the
that we are working on. You can use wrsw_pstats Command Line tool
(see FAQ: How can I see statistics about Ethernet Frames forwarded by
the WRS?) to see whether there are any errors. If you see that the
counter of "2:Riv-cd" is non zero, the chances are that this is the
. To mitigate the issue, try connecting an off-the-shelf Gigabit
Ethernet switch between the non-WR device and the WRS.
Author: Maciej Lipinski 31/10/18
Q: Is it possible to disable RTU learning on a switch?
A: Yes, from firmware v5.0 rtu_stat has a parameter "learning" which
can be used define on which port(s) to disable RTU learning. For all
(including older) firmwares the same result can be achieved by using
# Disable learning on port 10# First, select port 10 in "Port Select Register"wrs1# devmem 0x10060004 32 0xA# read the current value of "Port Control Register"wrs1# devmem 0x100600080x00000083# Clear bit 0 in the returned valuewrs1# devmem 0x10060008 32 0x82
Q: Can the WR Switch get its IP address on the Management Ethernet port from DHCP server?
A: Yes, the WR Switch has a DHCP client software. Please remember
though, if you modify the startup scripts to assign a static IP address,
the configuration received from the DHCP server will be overwritten.
Q: Is it possible to forward received UDP packets to the switch's Management (copper) Ethernet port?
A: It could be done, but is not supported. It is a dangerous idea which
is strongly discouraged as it can have very bad consequences including
switch crashing. Therefore, this is not supported and no information on
how to do it will be provided by the switch developers.
??Authors: Alessandro Rubini,
Gnudd, Maciej Lipinski,
Q: Why on older switches (software versions up to v3.1) it was possible to access (e.g.: ssh) WR switch through any port (wr0-wr17 and management) and in the newer software releases (from software version v3.3) it is not?
A: Accessing the WR switch for management (e.g. ssh) through a "normal"
port (e.g. wr17) instead of through the copper Ethernet management port
is one of the dangerous features that you won't find on "industrial
switches" - as it would open a serious security hole. This is why it
shall not be supported out-of-the-box on WR switches and it is strongly
advised by WR developers to avoid doing it.
Here are some dangers of doing so:
What it effectively does is enable any user in the network to break
the network completely by ssh-ing to the switch and doing whatever.
This is why only the management port should be used that should be
connected to a separate network which is limited-access only by the
administrator (we call it technical network at CERN). This is
especially true for "normal networks" with public access, less in
The management port will work regardless of FPGA binary and even
operating system/daemons on the switch. This means that if you
upgrade the FPGA binary or software on the WR switch and something
goes wrong for some reason, chances are that you can still access
the switch using the management port but you will surely have no
access through wrPorts - if you limit your access to the switch only
to wrPort (e.g. wr17) in a remote site... there are more chances you
will need to go there physically to fix something.
Using a wrPort (e.g. wr17) might degrade performance of frame
forwarding as well as the daemons (e.g. WRPTP timing) are running on
the CPU - remember that CPU is connected with all the wrPorts
through a single "virtual" port which is multiplexed. Your ssh
session (as well as scp and whatever you use over wr17) is shared
the with the daemons that make sure your network works properly. If
your management activities overwhelm the wrPort (and CPU), you might
end up managing a misbehaving switch.
To sum up, using a non-management port (e.g. wr17) for ssh and
management is possible on a WR switch but needs special configuration.
It is something you might do for some reasons in the lab or while
testing systems, but it should definitely be avoided in a deployed
Q: How can I see statistics about Ethernet Frames forwarded by the WRS?
A: If you want to see statistics of Ethernet Frames forwarded between
ports of WRS, you should use wrsw_pstats, one of the WRS Command-Line
Tools: it shows how many frames are forwarded between ports, errors,
dropped frames, etc.
Such commands as ifconfig or tcpdump will only show traffic (e.g.,
PTP, LLDP) that is destined to Linux CPU inside the switch. It will
not show traffic forwarded between WRS ports. The fields of
wrsw_pstats are described on page 14 of WRS HDL-SW interface
Q: Why WR Switch is not able to get path to dot-config via DHCP when dnsmasq is used as DHCP server?
A: To fix this problem please use "dhcp-no-override" in the dnsmasq
It looks like dnsmasq implementation is buggy (at least up to version
2.68). When switch explicitly requests "bootfile" option via DHCP, then
dnsmasq puts filename into the option 67 (bootfile name) in the
response. This is not compliant with rfc2132:
9.5 Bootfile nameThis option is used to identify a bootfile when the 'file' field inthe DHCP header has been used for DHCP options.
If situation described above applies, then dhcp response shall contain
also option 52 (which it doesn't):
9.3. Option Overload This option is used to indicate that the DHCP 'sname' or 'file' fields are being overloaded by using them to carry DHCP options. A DHCP server inserts this option if the returned parameters will exceed the usual space allotted for options. If this option is present, the client interprets the specified additional fields after it concludes interpretation of the standard option fields.[...]
Q: I can not connect to SSH after flashing the switch?
In v4.x, one of the last steps of the flashing procedure is to generate
new rsa/dss ssh keys.
If you unplug your switch while you are generating the key, you will
have an empty/invalid rsa/dsa file and it will be impossible to use SSH
to connect to this switch.
Please, try to wait about 2 minutes after flashing the switch before
Otherwise you need to re-generate the ssh keys by executing:
Q: Why my changes in /etc/* files are not visible after a reboot (after updating the firmware to v4.x)?
A: Since v4.0 we unpack the root filesystem from initramfs and the
/etc directory is copied at boot time from the /usr/etc. If you want
to make changes to the files inside the /etc, you should modify them
in the /usr/etc and reboot the device. You can find more information
about the code layout of the WR Switch in the Building and flashing
manual for v4.0
Q: When upgrading the 4.x switch to v5.0 I see an error "/wr/bin/sdb-read: can't load library 'libm.so.1'"
A: During the upgrade from pre-v5.0 version of firmware to v5.0 you can
observe on a console a following message being printed:
/wr/bin/sdb-read: can't load library 'libm.so.1'Creating SDB filesystem in /dev/mtd5cp: can't stat '/wr/etc/sdb-for-dataflash': No such file or directory done
Please ignore this printout. No real error occurs.
The printout comes from the fact that during the upgrade, the old
firmware is trying to execute the binaries (sdb-read and
sdb-for-dataflash) from the new firmware. Since both firmwares are using
different versions of libraries these programs fail to execute. Upgrade
of libraries' versions comes from the buildroot upgrade done between
v4.2 and v5.0 of wrs firmware. If there is a need to create hwinfo
partition, it will be created at the first boot of firmware v5.0.
Despite there is a message printed:
Creating SDB filesystem in /dev/mtd5
there is no change done to the hwinfo partition (/dev/mtd5).
Q: Is it possible to flash a switch via test interface using the scripts under wr-switch-sw/build/flash-wrs?
A: Yes, the procedure is described in the section "3.2 Flashing
Procedure" of the Developer's Manual .
If you have problems flashing a switch via USB, try to use USB 2.0 or
1.2 hub (I don't remember which version) in the middle. With some
versions of a switch there were problems to flash directly via USB.
Please note that there was a bug in the firmware v5.0 , which may
affect the procedure.
A: Yes, 18-port switch is a big complex design. If you want to speed up
your development you can synthesize the 8-port version of the gateware
to test new functionality. In that case you should make your synthesis
Q: ISE appears to have frozen during MAP/PAR.
A: Did you install a 64-bit version of ISE? It needs up to 4 GB of free
RAM during mapping/routing the design. 32-bit version can't use more
than 3 GBs and sometimes freezes instead of throwing an
Q: Firmware 4.2 (buildroot) does not compile on Ubuntu 16.04 (gcc problem)
A: The buildroot 2011.11 used in the Firmware 4.2 contains a
cross-compiler gcc 4.3.6, which does not compile with a host gcc version
>=5. To solve this problem please use a flag "-fgnu89-inline" for host
gcc. You can find more details about this issue in
If you see in the error log messages like the following, definitely you
are impacted by this
~/build/buildroot-2011.11/output/toolchain/gcc-4.3.6/gcc/toplev.c:545:1: error: redefinition of ‘floor_log2’~/build/buildroot-2011.11/output/toolchain/gcc-4.3.6/gcc/toplev.c:580:1: error: redefinition of ‘exact_log2’
In this case please use a branch wr-switch-sw-v4.2-fixes from
wr-switch-sw repo, which contains needed fixes.
Q: Should green and orange LEDs be lit when copper SFP is used even without Ethernet cable plugged in?
A: Yes, that is the normal behavior with most of the copper SFPs. Even
after unplugging Ethernet cable the copper SFP fakes an active
connection so green LED is ON and orange may be
Q: There are two fans. Is it correct that both of them push the air out of the box on backside?
A: Yes, it is correct that both fans push the air out of the box on
backside. The idea is that the cold air is just sucked in at the SFP
cages at the front. Tests have been made that shows that this is the
best way of cooling that is also compatible to racks that have a
front-to-back airflow. In the White Rabbit Switch Hardware
project one can find the
WRS - Stress Tests report.
Q: The output from the SMC connector labeled at 125MHz (OUT) is tuned at 62.5Mhz frequency?
A: In the initial version of the switch the reference frequency was set
at 125MHz, but the gateware have been modify during the production.
Therefore some old box are still labeled with 125 MHz. On the new box,
you will find this output labeled as CLK (OUT).
Q: Is it possible to generate 10MHz clock from WR Switch?
A: Yes, it is possible with latest v3.4 version of the White Rabbit
Switch hardware and coming v4.2 firmware release. By default SMC
connector CLK2 produces 10MHz signal synchronous to the White Rabbit
clock. The CLK2 frequency, ratio and delay to the 1-PPS output can be
configured by user to generate other clocks as well. Please check the WR
Switch manual for more details.
If you use one of the previous versions of the WR Switch hardware, you
can generate 10MHz using the
with a carrier (SPEC or
Q: How fast is the overall performance within the WR switch fpga? Is it really 18*1Gbps? As an example: When I establish a connection between port1 and port2 with 1 Gb/s is it possible for port3 also to communicate with port4 with 1Gb/s? Is it still possible for each connection to communicate at full speed?
A: A good question. It should be and it is meant to be but it's not yet
possible. The current gateware have the following issues:
It cannot handle well 100% load (so 1Gb/s) traffic even if you have
only two ports connected, so if port1 talks with port2, only. It is
ok up to ~90% of traffic load.
It has problems with handling small frames (<128 bytes) at high
If you use a large number of ports at the same time (more than 8
ports which create 4 pairs), the forwarding performance starts
dropping (the load of traffic that the switch can forward between
each directly connected pair of ports goes down as the number of
used ports goes up).
(as for July 2013, concerning v3.3 and below of gateware, intended to be
fixed for v4)
There is also an issue in
wr-switch-hdl project regarding this problem.
Q: Which FPGA is used inside the WR Switch, Virtex 6 LX130T or LX240T?
A: Initially WR Switch v3.3 was designed with LX130T FPGA. That is why
you can still find it in the schematic and PCB layout files. However,
over time we were adding more and more features implemented in HDL and
LX130T became to small. We have replaced the FPGA chip with LX240T which
is larger than LX130T and has the same pinout. Currently all WR Switches
are produced with larger, Virtex 6 LX240T FPGA.
Q: Has any kind of lifetime study been done on the White Rabbit Switch?
We are not aware of anyone studying this for the White Rabbit Switch. No
calculations or tests have been done for the lifetime. The switch has
been tested under rather harsh conditions, among others in ovens and in
and did not fail.
At CERN we are planning to track the number of hours each switch has
been running and at what operating temperature so as to collect useful
statistics to be able to do useful calculations of lifetime. This has
not yet been implemented in the switch. (March 2015)
Q: Is it possible to get ID-string of SFP on a switch?
A: In current release (4.1) it is not possible. However, `sfp detect`
command can be used on SPEC card to get SFP's ID-string. In next
release, HAL will report ID-strings of all not matched SFP's.
Additionally, wrs_dump_shmem will report ID's in fields calib.sfp.*
A: From the release (4.2.1) it is possible to use wrs_sfp_dump to read
content of SFPs memory.