Frequently Asked Questions
about the White Rabbit switch.

Connecting the switch to an external GPS receiver

Q: What are the advantages of using external GPS signals or are there any disadvantages of not using them and leave the connectors open?

A: You mainly get two things:

  • A more stable oscillator for the master switch and therefore for all the network.
  • A GPS/UTC/TAI-related time scale. This is important if you want to relate your events to official time or if you want to avoid re-starting your time scale at 1 Jan 1970 (or something like that) every time you reboot the switch. Two events seen in a WR network at very different times could have very similar time stamps if there was a switch reboot in between. Syncing to GPS time on reboot avoids that.

Q: When connecting the GPS-PPS or the 10 MHz GPS clock to the WhiteRabbit Switch, do I have to change/configure anything?

A: Yes, it is documented in the Note on using WR Switch in Grandmaster mode.

Q: We need to use 30m coaxial cable from the GPS to the WRS Grandmaster 10MHz/PPS signals and the cable will see strong temperature changes (15-35ºC).

Temperature effects are typically very slow phenomena. The first question would be: do they care? Quite often people want to synchronize nodes among themselves, to a common reference, but they don't mind too much if that reference slips a bit with respect to for example GPS time. As you can see in [1], the thermal coefficient of delay (TCD) expressed in (ppm/deg C) is itself a function of temperature. Taking a ballpark number of 100 ppm/degC for coax, we are talking about delay changes in the order of 0.3 ns. Would your client care if the reference slips slowly by 1 ns or so? Are they using this reference for something else than feeding it into the Grandmaster WR switch? If this phase slip is a problem, then you could explore coax cables not using a teflon dielectric [2] or even active control of the temperature by enclosing the coax inside double-wall tubing and filling the space within the walls with temperature-controlled water [3]. Another possibility, if the temperature is kind of uniform along the cable length, is to characterize the temperature dependency of the very cable to be installed in a climatic chamber and then measuring temperature in the real system and applying phase shifts digitally.

[1] http://ipnpr.jpl.nasa.gov/progress_report/42-99/99E.PDF
[2] http://przyrbwn.icm.edu.pl/APP/PDF/119/a119z4p17.pdf
[3] http://www.slac.stanford.edu/cgi-wrap/getdoc/slac-pub-6297.pdf

Note about [2]:
  • Management summary: LCF38-50J is most stable from the four cable types characterised. Take care of the teflon knee.
    The LCF38-50J is a rather thick (3/8") and possibly cumbersome (because rigid) cable type. If you want a fixed cable in your infrastructure it is recommendable though, since it is not excessively expensive. However, it only comes with N-connectors, fortunately those typically do have good phase stability.

Authors: Javier Serrano/CERN, Tjeerd Pinkert/VU Amsterdam

Q: The last paragraph of the Note on using WR Switch in Grandmaster mode mentions HAL messages and RT log. I find HAL messages but what is meant by RT log and where to find it?

A: By the RT (i.e. realtime) log mentioned in the Note on using WR Switch in Grandmaster mode we mean the messages generated by the LM32 CPU inside the Virtex FPGA. They may provide useful information for debugging the timing subsystem. They can be viewed by connecting a serial terminal to the the front left USB port (up to v. 3.2 switches) or the "FPGA test" USB port in the rear (newer switches). RS232 parameters are 115200, 8N1.

Q: Is it possible to use White Rabbit systems in different locations and be able to have synchronised time via Satellite link?

A: Basically as long as the 10 MHz and PPS that you give to the different White Rabbit systems are synchronised, WR will keep its promises.
For example, you may synchronise two WR networks by at both ends letting a GPS generate the PPS and 10MHz signals, but you will be limited by the precision the GPS will give you on those two different locations. This may be as good as 3 ns already or maybe better in combination with augmentation systems.
Or you can synchronise your systems with Time and Frequency Transfer using the phase of the GPS Carrier, as CERN has done with the CNGS experiment where we could, after the fact, synchronise related events happening on both sites. The article Time transfer techniques for the synchronisation between CERN and LNGS explains this.
If you have Satellite links, you can use Two-Way Satellite Time Transfer (TWSTT) and have on both sides a system that will provide the PPS and 10 MHz signals. According to the presentation Time transfer through optical fibers, page 3 from Mrs. Amy-Klein from LPL, Paris, this can provide an accuracy better than 1 ns.
Using Time Transfer by Laser Link (T2L2) would give you even a sub-ns accuracy.
The NEAT-FT Workshop on Optical Networks for Accurate Time and Frequency Transfer held in 2012 may give you more ideas and you possibly can find partner institutes that can help you with such a system.


Inter-operability

Q: Is the WR Switch inter-operable with standard PTP (IEEE1588-2008) gear?

A: Yes, although you would completely lose the sub-ns timing resolution of White Rabbit, a WR switch is interoperable with PTP and it has been tested for inter-operability on ISCPS PlugFest, but
  • it is important to remember that WR devices use a very specific PTP configuration (2-step, delay request-response mechanism, multicast, Layer 2) which is not the most popular configuration and it is not the default setting for standard-PTP devices (which use IP instead of Layer 2). So it is not just plug&play, it's more like
    1. buy PTP switch which supports fully the standard
    2. configure it properly (change from IP to Layer 2)
    3. plug & play

Q: Can I connect a standard non-WR GbE NIC or Switch/router to the WR Switch?

A: Yes. You can use a WR-network for standard datatransfer, using standard Ethernet hardware. It works fine since a WR switch is a standard GbE switch.

Q: Can you have a different vendor (non-WR) switch between it and another white rabbit device to distribute the time? Around a larger network for instance?

A: Yes, you can connect non-WR switches in a WR network. But it does not make sense as in that case you completely lose the sub-ns timing resolution of White Rabbit.

If the switch between the WR master (switch) and the WR end node is a standard PTP one, your timing resolution will be reduced to the relatively low PTP precision. The reason is that the White Rabbit protocol is achieved by synchronizing each link between one end and the other end of the fibre and the non-WR switch in the middle would not support the WR protocol.
To say it in another way: you may mix non-WR and WR switches to build your complete network (it will perfectly transfer your data) but to have WR precision all switches between your WR timing master switch and your WR nodes should be WR-compliant ones.

Ah, when you speak about different vendor, I assume you meant a non-White Rabbit switch. But you may want to know that as the design of the WR switch is open, there may in due time be more producers of White Rabbit switches.

Q: I was wondering if WR has plans to use SyncE as a standard for clock recovery, or if the custom clock recovery core in WR is as accurate (or better) for clock recovery? if WR meets the jitter SyncE specifications, what is the difference in the clock recovery method?

A: SyncE and WR are very similar in terms of clock recovery. What WR does additionally is to implement a phase shift on the recovered clock signal in the slave. This shift tracks changes in delay in the medium, so as to guarantee constant (ideally zero) phase with respect to the master clock signal. With SyncE, if the transmission delay in your fiber changes, the phase of the clock signal you recover also drifts with respect to that in your master. What's more, the SyncE standard comprises of two elements:
  1. slow protocol to configure the frequency distribution hierarchy in the network
  2. specification of timing characteristics of synchronous Ethernet equipment (e.g. ITU-T G.8262)

In WR, the protocol (1) is not used which is rather uncommon in SyncE networks but it is allowed by SyncE. The timing characteristics (2) of SyncE have a lot of legacy burden (compatibility with SDH). Consequently, SyncE equipment must be tolerant to quite "bad" clock input. WR is not tolerant to the corner-cases required by SyncE, see the report on SyncE Characteristics of a White Rabbit Switch . Making WR PLL tolerant to clock characteristics required by ITU-T G.8262 might result in deterioration of WR performance (e.g. requires change in PLL bandwidth) or require a much more expensive oscillator.
ITU-T itself recognized that the SyncE specification is outdated and most of SyncE equipment can do much better than the specifications. Therefore, ITU-T is working on enhanced SyncE (eEEC). There is some consultation between ITU-T and WR on the eEEC, though with low priority, at the moment it seems that eEEC will not be directly usable with WR.

Authors: Javier Serrano, Maciej Lipinski, CERN


Switch management

Q: Can the WR Switch get its IP address on the Management Ethernet port from DHCP server?

A: Yes, the WR Switch has a DHCP client software. Please remember though, if you modify the startup scripts to assign a static IP address, the configuration received from the DHCP server will be overwritten.

Author: Grzegorz Daniluk, CERN

Q: Is it possible to forward received UDP packets to the switch's Management (copper) Ethernet port?

A: It could be done, but is not supported. It is a dangerous idea which is strongly discouraged as it can have very bad consequences including switch crashing. Therefore, this is not supported and no information on how to do it will be provided by the switch developers.

Authors: Alessandro Rubini, Gnudd, Maciej Lipinski, CERN

Q: Why on older switches (software versions up to v3.1) it was possible to access (e.g.: ssh) WR switch through any port (wr0-wr17 and management) and in the newer software releases (from software version v3.3) it is not?

A: Accessing the WR switch for management (e.g. ssh) through a "normal" port (e.g. wr17) instead of through the copper Ethernet management port is one of the dangerous features that you won't find on "industrial switches" - as it would open a serious security hole. This is why it shall not be supported out-of-the-box on WR switches and it is strongly advised by WR developers to avoid doing it.
Here are some dangers of doing so:
  1. What it effectively does is enable any user in the network to break the network completely by ssh-ing to the switch and doing whatever. This is why only the management port should be used that should be connected to a separate network which is limited-access only by the administrator (we call it technical network at CERN). This is especially true for "normal networks" with public access, less in control&acquisition systems
  2. The management port will work regardless of FPGA binary and even operating system/daemons on the switch. This means that if you upgrade the FPGA binary or software on the WR switch and something goes wrong for some reason, chances are that you can still access the switch using the management port but you will surely have no access through wrPorts - if you limit your access to the switch only to wrPort (e.g. wr17) in a remote site... there are more chances you will need to go there physically to fix something.
  3. Using a wrPort (e.g. wr17) might degrade performance of frame forwarding as well as the daemons (e.g. WRPTP timing) are running on the CPU - remember that CPU is connected with all the wrPorts through a single "virtual" port which is multiplexed. Your ssh session (as well as scp and whatever you use over wr17) is shared the with the daemons that make sure your network works properly. If your management activities overwhelm the wrPort (and CPU), you might end up managing a misbehaving switch.

To sum up, using a non-management port (e.g. wr17) for ssh and management is possible on a WR switch but needs special configuration. It is something you might do for some reasons in the lab or while testing systems, but it should definitely be avoided in a deployed network.

Authors: Maciej Lipinski, CERN

Q: I want to modify some files in the switch but I can't because the filesystem is mounted as Read-Only! How can I set it as Read/Write?

A: Before v4.0 release we had our filesystem mounted read-only. Therefore if you still run an old firmware version (v3.3) and you want to modify files in your filesystem, please execute first:

wrfs_mnt.sh rw

Once you have applied your changes we recommend to go back to read only mode

wrfs_mnt.sh ro

Starting from v4.0 we use the ramdisk and UBIFS. The filesystem is mounted R/W and the commands above are no longer needed.

Authors: Benoit Rat, Seven Solutions

Q: What is the ssh root password to access WR switch through the management port ?

A: Currently there is no password at all, you just press ENTER. This will change in the future.

Q: There is a new leap second announced, what shall I do?

A: When a new leap second is announced then a leap-seconds.list file shall be updated on the grand master or free running master using the following procedure:

Remove the old file with the leap second information:

rm /usr/etc/leap-seconds.list

Download a new file:

wget -P /usr/etc http://www.ietf.org/timezones/data/leap-seconds.list

Then reboot a switch.

If your switch does not have an access to the Internet, download the leap-seconds.list file on a different machine and copy it via e.g. ssh.

Author: Adam Wujek, CERN


Updating firmware

Q: I have bought WR switch v3.x is it compatible with the latest update of firmware?

A: You can look at the different version compatibility on the WRS_versions page, you will also find which firmware version is the best to use according to your hardware.

Author: Benoit Rat, Seven Solutions

Q: I can not connect to SSH after flashing the switch?

In v4.x, one of the last steps of the flashing procedure is to generate new rsa/dss ssh keys.
If you unplug your switch while you are generating the key, you will have an empty/invalid rsa/dsa file and it will be impossible to use SSH to connect to this switch.

Please, try to wait about 2 minutes after flashing the switch before unplugging it.

Otherwise you need to re-generate the ssh keys by executing:

dropbearkey  -t dss -f dropbear_dss_host_key
dropbearkey  -t rsa -f dropbear_rsa_host_key

Author: Benoit Rat, Seven Solutions

Q: Why my changes in /etc/* files are not visible after a reboot (after updating the firmware to v4.x)?

A: Since v4.0 we unpack the root filesystem from initramfs and the /etc directory is copied at boot time from the /usr/etc. If you want to make changes to the files inside the /etc, you should modify them in the /usr/etc and reboot the device. You can find more information about the code layout of the WR Switch in the Building and flashing manual for v4.0 release

Author: Grzegorz Daniluk, CERN

Q: When upgrading the 4.x switch to v5.0 I see an error "/wr/bin/sdb-read: can't load library 'libm.so.1'"

A: During the upgrade from pre-v5.0 version of firmware to v5.0 you can observe on a console a following message being printed:

/wr/bin/sdb-read: can't load library 'libm.so.1'
Creating SDB filesystem in /dev/mtd5
cp: can't stat '/wr/etc/sdb-for-dataflash': No such file or directory
 done

Please ignore this printout. No real error occurs.

The printout comes from the fact that during the upgrade, the old firmware is trying to execute the binaries (sdb-read and sdb-for-dataflash) from the new firmware. Since both firmwares are using different versions of libraries these programs fail to execute. Upgrade of libraries' versions comes from the buildroot upgrade done between v4.2 and v5.0 of wrs firmware. If there is a need to create hwinfo partition, it will be created at the first boot of firmware v5.0.

Despite there is a message printed:

Creating SDB filesystem in /dev/mtd5

there is no change done to the hwinfo partition (/dev/mtd5).

Author: Adam Wujek, CERN

Q: After upgrade a switch to firmware v5.0, it does not boot via NFS

A: Please update "bootargs" variable in your boot script.
With the kernel 2.6.39 we suggested use of "root=nfs", but this convention is no more supported in Linux, please use "root=/dev/nfs".

Author: Adam Wujek, CERN


Building firmware

Q: It takes very long to synthesize.

A: Yes, 18-port switch is a big complex design. If you want to speed up your development you can synthesize the 8-port version of the gateware to test new functionality. In that case you should make your synthesis inside syn/scb_8ports/

Q: ISE appears to have frozen during MAP/PAR.

A: Did you install a 64-bit version of ISE? It needs up to 4 GB of free RAM during mapping/routing the design. 32-bit version can't use more than 3 GBs and sometimes freezes instead of throwing an error.

Q: Firmware 4.2 (buildroot) does not compile on Ubuntu 16.04 (gcc problem)

A: The buildroot 2011.11 used in the Firmware 4.2 contains a cross-compiler gcc 4.3.6, which does not compile with a host gcc version >=5. To solve this problem please use a flag "-fgnu89-inline" for host gcc. You can find more details about this issue in stackoverflow
If you see in the error log messages like the following, definitely you are impacted by this error.

~/build/buildroot-2011.11/output/toolchain/gcc-4.3.6/gcc/toplev.c:545:1: error: redefinition of ‘floor_log2’
~/build/buildroot-2011.11/output/toolchain/gcc-4.3.6/gcc/toplev.c:580:1: error: redefinition of ‘exact_log2’

In this case please use a branch wr-switch-sw-v4.2-fixes from wr-switch-sw repo, which contains needed fixes.

Author: Adam Wujek, CERN

Q: Firmware 4.2 (buildroot) does not compile on Ubuntu 16.04 (perl problem)

A: Using perl >= 5.21.x on a host system can lead to errors when compiling the kernel. The warning (for perl < 5.21.x) becomes an error (for perl >= 5.21.x):

defined(@array) is deprecated at kernel/timeconst.pl line 373.
  (Maybe you should just omit the defined()?)

This error pops up on newer systems. It is not related with the change of the buildroot, but with the host version of perl on a host system.

In this case please use a branch wr-switch-sw-v4.2-fixes from wr-switch-sw repo, which contains needed fixes.

Author: Adam Wujek, CERN


General questions

Q: Should green and orange LEDs be lit when copper SFP is used even without Ethernet cable plugged in?

A: Yes, that is the normal behavior with most of the copper SFPs. Even after unplugging Ethernet cable the copper SFP fakes an active connection so green LED is ON and orange may be blinking.

Q: There are two fans. Is it correct that both of them push the air out of the box on backside?

A: Yes, it is correct that both fans push the air out of the box on backside. The idea is that the cold air is just sucked in at the SFP cages at the front. Tests have been made that shows that this is the best way of cooling that is also compatible to racks that have a front-to-back airflow. In the White Rabbit Switch Hardware project one can find the WRS - Stress Tests report.

Author: Benoit Rat, Seven Solutions

Q: Does the switch comply to CE regulations (EMC)?

A: It is. The company Seven Solutions has performed extensive EMC tests on the switch. The full report can be found in White Rabbit Switch Hardware documentation.

Q: The output from the SMC connector labeled at 125MHz (OUT) is tuned at 62.5Mhz frequency?

A: In the initial version of the switch the reference frequency was set at 125MHz, but the gateware have been modify during the production. Therefore some old box are still labeled with 125 MHz. On the new box, you will find this output labeled as CLK (OUT).

Author: Benoit Rat, Seven Solutions

Q: Is it possible to generate 10MHz clock from WR Switch?

A: Yes, it is possible with latest v3.4 version of the White Rabbit Switch hardware and coming v4.2 firmware release. By default SMC connector CLK2 produces 10MHz signal synchronous to the White Rabbit clock. The CLK2 frequency, ratio and delay to the 1-PPS output can be configured by user to generate other clocks as well. Please check the WR Switch manual for more details.
If you use one of the previous versions of the WR Switch hardware, you can generate 10MHz using the FineDelay FMC with a carrier (SPEC or SVEC).

Author: Grzegorz Daniluk, CERN

Q: in the repository there are different branches. Which one should I take?

A: All the correct files and used for manufacturing of the WRS are in the Master branch of http://www.ohwr.org/projects/wr-switch-hw/repository.
Other branches such as CTI_modifications contain files that are not valid for production and may not be correct.

Q: Could the WR switch be used as Open Compute Project open source switch?

A: The article Facebook aims to knock Cisco down a peg with open network hardware shows the need for an open source switch.
The OCP open source switch project seems to be all about being able to run any OS on the switch and run software-defined networking (SDN) (a kind of remote management) on it.
It opens some questions:
  1. Would one be able to run "any OS" on our switch?
    • Yes. We have an ARM processor inside the switch that runs Linux, and any OS with ARM support may be run on it.
  2. Is our API to the firmware documented?
  3. Is the hardware well documented?
    • Everything we have is available on the White Rabbit switch page.
      Although most information is available, unfortunately it is not yet enough documented. Some description is in the Gateware-Software document and the hardware documentation should be completed (schematics, PCB layout, production documentation and mechanics) and it should be checked if everything is up-to-date. This is on the list of things to do already.

Likely the WR switch is not good enough for a final OCP product as with its size of only 18-ports it is less than what computer centers are used to, like the 1 Gbps speed instead of the now common 10 Gbps. The early implementation of the High Accuracy Option/Profile of IEEE1588 is a nice plus showing innovation for which Facebook or others may find good use, for example in Spanner.

So the WR switch project possibly could be used as a prototype platform. It is completely open and already commercially available and may quickly be used to port OpenFlow and OpenDaylight on it to prove the principle.

Author: Erik van der Bij, CERN

Q: How fast is the overall performance within the WR switch fpga? Is it really 18*1Gbps? As an example: When I establish a connection between port1 and port2 with 1 Gb/s is it possible for port3 also to communicate with port4 with 1Gb/s? Is it still possible for each connection to communicate at full speed?

A: A good question. It should be and it is meant to be but it's not yet possible. The current gateware have the following issues:
  1. It cannot handle well 100% load (so 1Gb/s) traffic even if you have only two ports connected, so if port1 talks with port2, only. It is ok up to ~90% of traffic load.
  2. It has problems with handling small frames (<128 bytes) at high loads
  3. If you use a large number of ports at the same time (more than 8 ports which create 4 pairs), the forwarding performance starts dropping (the load of traffic that the switch can forward between each directly connected pair of ports goes down as the number of used ports goes up).

(as for July 2013, concerning v3.3 and below of gateware, intended to be fixed for v4)

There is also an issue in wr-switch-hdl project regarding this problem.

Author: Maciej Lipinski, CERN

Q: Which FPGA is used inside the WR Switch, Virtex 6 LX130T or LX240T?

A: Initially WR Switch v3.3 was designed with LX130T FPGA. That is why you can still find it in the schematic and PCB layout files. However, over time we were adding more and more features implemented in HDL and LX130T became to small. We have replaced the FPGA chip with LX240T which is larger than LX130T and has the same pinout. Currently all WR Switches are produced with larger, Virtex 6 LX240T FPGA.

Author: Grzegorz Daniluk, CERN

Q: Has any kind of lifetime study been done on the White Rabbit Switch?

We are not aware of anyone studying this for the White Rabbit Switch. No calculations or tests have been done for the lifetime. The switch has been tested under rather harsh conditions, among others in ovens and in Siberia and did not fail.
At CERN we are planning to track the number of hours each switch has been running and at what operating temperature so as to collect useful statistics to be able to do useful calculations of lifetime. This has not yet been implemented in the switch. (March 2015)

Q: Is it possible to get ID-string of SFP on a switch?

A: In current release (4.1) it is not possible. However, `sfp detect` command can be used on SPEC card to get SFP's ID-string. In next release, HAL will report ID-strings of all not matched SFP's. Additionally, wrs_dump_shmem will report ID's in fields calib.sfp.*
A: From the release (4.2.1) it is possible to use wrs_sfp_dump to read content of SFPs memory.

Author: Adam Wujek, CERN


See also:

Frequently Asked Questions about White Rabbit

Frequently Asked Questions about the White Rabbit PTP Core


22 February 2017

wr_external_reference.pdf (75.8 kB) Tomasz Wlostowski, 2012-10-08 11:47