Frequently Asked Questions
about White Rabbit.
-
Frequently Asked Questions
-
White Rabbit general questions
- Q: What's the easiest way to start playing with White Rabbit?
- Q: I am considering using White Rabbit in my next project. Where can I get support?
- Q: How can I test White Rabbit switch equipment? Or can I buy some test equipments from your company?
- Q: If I don’t participate in WR workshop meetings, is the presentation material available?
- Q: How can I trust White Rabbit?
- Q: How important is White Rabbit?
- Q: Who to contact if I cannot find the right information about White Rabbit?
-
White Rabbit use cases
- Q: I have many GPS receivers on our company site. Any advantage to use White Rabbit here?
- Q: Can I use copper SFPs in WR network for connecting nodes that don't require WR timing?
- Q: Are WR devices compatible with 100 Mbit/s copper SFP?
- Q: Will synchronization between WR devices in a WR network be equally accurate with and without a Grandmaster connected to a good reference (10MHz, 1PPS) ?
- Q: Is it possible/foreseen to allow more than one Grandmaster in a WR Network?
- Q: White Rabbit will give very low time differences between the devices connected to the network, in the low nano seconds. But what about UTC?
-
White Rabbit technical questions
- Q: What is the WR performance? Accuracy and precision.
- Q: Can I get links longer than 10 km?
- Q: How much of the 1 Gbps bandwidth can I use?
- Q: Where can I put additional delays coming e.g. from optical amplifiers to be able to compensate them?
- Q: What limits the timing performance for the long-distance WR links?
- Q: May we use optical patch panels in the network or do you need direct connections?
- Q: Any restriction on which fibre type should be used?
- Q: We have existing fibres already available which are meant for 10 Gb/s networks. Can I use these?
- Q: WR uses a single single-mode fibre, but would a short (~5m) multi-mode fibre link work too?
- Q: What is the accuracy and resolution of the round-trip time that is calculated?
- Q: What about start-up variations in shown cRTT values?
- Q: How could one create a performant wire tap on a port of a WR Switch ? / How to create a stable mirroring of a given port of a WR Switch to its another port?
- See also:
-
White Rabbit general questions
White Rabbit general questions
Q: What's the easiest way to start playing with White Rabbit?
A: For starting to use WR technology a WR starting kit is available. The manual you can find there shows clearly what is possible with just two PCIe plug-in cards. Later on you can expand your kit with a White Rabbit switch.
Q: I am considering using White Rabbit in my next project. Where can I get support?
A: White Rabbit (WR) is open source and commercial. That means that you can get the best of both worlds. You can avoid vendor lock-in because the technology is open source, so you can switch vendors if you are unhappy with the service a particular vendor provides. Because it is open source, provided your team has adequate know-how, you can even decide to not rely on commercial companies for support. But if you want the convenience of paying somebody for guaranteed support, you can also have that. In general, each WR user has a particular problem to solve, and it would be hard to design a one-size-fits-all WR solution. WR starter kits illustrate the basic capabilities of WR and provide a good basis to build upon. There is often a need for local design customization, and that is why we recommend that WR users have either the backing of a local design team or a commercial company. There is also the [white-rabbit-dev](https://forums.ohwr.org/c/white-rabbit-dev/344 discourse forum, where you can ask questions. But for obvious reasons, experts in white-rabbit-dev cannot guarantee they will answer all your questions in a timely manner, or even at all. That would not be scalable. Answers on white-rabbit-dev are therefore provided on a good-will basis. The same is true for email you send privately to individual core WR developers. If you need guaranteed support, a local team or a company is the way to go.
Q: How can I test White Rabbit switch equipment? Or can I buy some test equipments from your company?
A: CERN is not a company, but a publicly funded intergovernemental
organization. We develop the technology but don't produce the equipment,
we buy WR switches from commercial vendors. Anyone can produce a WR
switch as all the sources are on our
pages (just have a careful
look at the wiki, sub-projects, ets... )
For example, there is wiki page about the WR switch on which
you can find the companies that are involved in switch development and
the companies from which you can buy a WR switch (Commercial producers,
at the bottom). On the wiki page about SPEC
card (an example of WR Node)
you will find companies producing SPEC
cards.
Q: If I don’t participate in WR workshop meetings, is the presentation material available?
A: All we do is public. All information from the workshops is directly accessible.
Q: How can I trust White Rabbit?
A: Most White Rabbit equipment is Open Hardware and Open Software. You
have full access to all design data and can inspect and recompile the
source code yourself to verify anything you like.
Many tests have been done by independent institutes to verify the timing
synchronisation and data transmission performance. Some institutes
indeed had found issues that have been solved since then. Other issues
have been found and are being worked on.
Q: How important is White Rabbit?
A1: Have a look at the many institutes and companies who are using White
Rabbit at the list of Users of White Rabbit Technology.
A2: That's a hard question. It is important enough to have been
specifically mentioned in many job offers at different institutes and
companies:
- Embedded Systems Developer (f/m/d) B.Sc., M.Sc. or Diploma in Electronics, Computer-Science or equivalent (GSI, 03/07/2019)
- https://www.ohwr.org/project/white-rabbit/wikis/News#04-02-2019-job-vacancy-at-gsi (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#24-01-2019-wr-job-vacancy-at-besancon-observatory- (FEMTO-ST)
- https://www.ohwr.org/project/white-rabbit/wikis/News#02-08-2018-job-vacancy-at-gsi (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#23-02-2018-wr-job-vacancy-at-gsi (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#19-06-2017-wr-job-vacancy-at-cern (CERN)
- https://www.ohwr.org/project/white-rabbit/wikis/News#26-04-2017-wr-job-vacancy-at-opnt---nl (OPNT)
- https://www.ohwr.org/project/white-rabbit/wikis/News#23-02-2017-wr-job-vacancy-at-cern (CERN)
- https://www.ohwr.org/project/white-rabbit/wikis/News#13-06-16-a-job-with-white-rabbit (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#04-09-2014-job-offer-at-gsi (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#7-5-2013-job-offer-for-developer (GSI)
Q: Who to contact if I cannot find the right information about White Rabbit?
A: You may contact Erik van der Bij - CERN or you may ask your question in the [white-rabbit-dev](https://forums.ohwr.org/c/white-rabbit-dev/344 discourse forum.
White Rabbit use cases
Q: I have many GPS receivers on our company site. Any advantage to use White Rabbit here?
A: The cost of the GPS receiver, although even those cost over 10 K (well above the price of a simple WR installation), may not be your problem. It's the antenna installation and big cable for it that make it all a hassle to manage GPS installations. Everyone can nowadays install fibres and likely they are already installed in buildings, so within a day you may add WR timing capabilities to other buildings or offices. GPS receivers give you only the time at one single point.
Likely you can also manage the whole thing better, including backup GPS receivers on completely other places in the network as WR can do automatic switchover between timing sources. Redundancy is taken into account in the design, much in the same way as implemented by the Ethernet standards that it follows. Also for data transfer we make sure that high priority packets 'never' get lost.
Q: Can I use copper SFPs in WR network for connecting nodes that don't require WR timing?
A: Yes. You can plug copper SFP into the Switch and use it to connect your node. You will be able to transmit Ethernet traffic, but you won't get WR timing for that node through copper SFP. However, you will be able to receive PTP time from WR Switch and synchronize to it. Of course it won't be sub-ns accuracy any more. See also Copper SFP modules.
Actually it is really not worth bothering using copper SFPs as fibre-optic SFPs are cheap (a pair is less than 100 Euro, cables are not expensive either), you are using standard equipment that most WR adapts are using and the delay parameters of the ones shown in [2] are already known by the White Rabbit switch and other equipment so that you have an out-of-the-box good performance not requiring a calibration.
See also the WR Switch FAQ question: Q: Should green and orange LEDs be lit when copper SFP is used even without Ethernet cable plugged in?
Q: Are WR devices compatible with 100 Mbit/s copper SFP?
A: No. WR switches/nodes support only 1GbE (copper/fiber). If you want to connect a non-1GbE device to a WR device, you need to do it through a switch that can speak both (most of 1GbE switches should do 10/100M)
Q: Will synchronization between WR devices in a WR network be equally accurate with and without a Grandmaster connected to a good reference (10MHz, 1PPS) ?
A: You can have your grandmaster switch free running completely, or connected only to 1PPS+10MHz but not the source of UTC. The synchronization accuracy in a WR network is guaranteed between the grandmaster and any WR device, regardless of the external source. However, having a good external frequency source is generally a good idea.
Q: Is it possible/foreseen to allow more than one Grandmaster in a WR Network?
A: It will be possible in the future using the seamless redundancy tools that are currently in a proof-of-concept state, see slide 7-11 of Redundancy presentation from WR Workshop. These mechanisms allow to connect as many Grandmasters as you want to a single switch downstream, provided the Grandmasters are synchronized at picoseconds (or tens of ps) level.
The logic to detect failure is very simple. If there is only one backup (e.g. 2 GMs), the switchover is triggered by: (1) physical failure, (2) notification from the master (i.e. if it looses the source of time). If there are two or more backups (e.g. 3 and more GMs), there is a third trigger of switchover: a kind-of majority voting based on comparison of all sources.
Byzantine faults and other more difficult problems are not handled. The main problem with that faults is that it takes considerably more time to detect them. So, even if you do detect them reliably, you are already off by much more than 1ns, so your timing does not meet the specification (sub-ns synchronization), so why bother. The first step to any attempt of solving this problem is much more stable local oscillator on the switch, so you have more time to decide and you can actually use the oscillator as a third source of time, and fall back to the majority voting...
Q: White Rabbit will give very low time differences between the devices connected to the network, in the low nano seconds. But what about UTC?
A: The time in WR network is transferred like in PTP (TAI plus
information about leap seconds). The usual scenario is that you connect
the grandmaster WR switch to 1PPS+10MHz from a GPS or Cesium clock and
you get the UTC information from
NTP.
Actually the WR system uses the PTP standard. And this standard uses TAI
and not UTC. Using UTC in WR (and also in PTP) is a dangerous strategy
as the behavior during a leap second jump is uncertain.
White Rabbit technical questions
Q: What is the WR performance? Accuracy and precision.
We understand accuracy as mean offset, and precision as standard deviation of that offset between PPS output signals. These definitions are quite simplistic for high-end time/frequency transfer.
-
For "standard" WR switch (current release v.5.0.1), the old numbers are quite accurate for accuracy/precision. A newer and more complete figure can be found in Maciej's thesis, page 50, Figure 4.5, it is also in the WR repo. In general, for this case, in constant temperature, the accuracy/precision for a single link are <100ps/10ps. Accuracy depends on calibration, it should be <100ps with proper calibration and this is what we typically see. It cannot be guaranteed it to be better than that because of bitslide problem. The source of inaccuracy and imprecision in WR and WRSv5.0.1 are summarized in Maciej's presentation from EFTS 2019, slides 22-30 (see White Rabbit lecture at European Frequency and Time Seminar, including the bitslide problem and relevant references. See also this article. Regarding the WR Switch, two improvements are addressed by:
- Low Jitter Daughtboard - improves precision
- Low Phase Drift Calibration (LPCD) - improves accuracy
-
WR switch with Low Jitter Daughtboard (LJD)
- See https://www.ohwr.org/project/wrs-low-jitter
- This is an additional feature, one must buy a WRS with LJD
- Related article that described the improvements and in general limits of WR switch time/frequency transfer : "White Rabbit Clock Synchronization: Ultimate Limits on Close-In Phase Noise and Short-Term Stability Due to FPGA Implementation".
- Regarding performance of LJD, the improvement in precision is not
characterized with standard deviation (to simplistic metric) but rather jitter RMS and Modified Allan Deviation, which improve as follows
- GM: 9ps to 1ps RMS 1Hz-100kHz
2E-12 to < 5E-13 τ =1s ENBW 50Hz - BC: 11ps to <2ps RMS 1Hz-100kHz
4E-12 to <7E-13 τ =1s ENBW 50H
- GM: 9ps to 1ps RMS 1Hz-100kHz
-
WR with Low Phase Drift Calibration (LPCD)
- as of next WR switch release (V6.0), this feature will work by default on ports 1-12 of the WR switch (we do not have sufficient number of clocking resources in the FPGA to make it available for all ports)
- It mitigates imprecision of bitslide measurement (~+/-35ps for GTX) which limited the accuracy to 100ps on a single link.
- what we know so far is that it allows to keep the offset between PPS of the master and a slave to within 10ps after replugging the link and restarting switches.
- the expectation is that it will improve the accuracy of synchronization on a single link to <10ps (so tenfolds)
- We need proper tests to confirm this.
Note that all the above is for "standard" WR links (<10km, single mode bidirectional fibers). (ML, March 2020)
Answer from May 2016:
A: A WR Switch can easily achieve a jitter (measured from 1Hz to 100kHz)
less than 20ps with two levels of hierarchy. Most of the jitter power
(99+%) is confined in the 1Hz-2kHz bandwidth. If you are using a
WR-enabled SPEC with a digital I/O card, the jitter performance may
deteriorate but still will be better than 50ps RMS. The accuracy is
specified better than 1ns, but this specific offset will be very stable
having only the mentioned jitter on top of it. Temperature effects will
give a very slow drift.
The White Rabbit low jitter
project may show more
detailed results and shows ways of how the jitter (or phase-noise) may
be improved (May 2016)
Q: Can I get links longer than 10 km?
A: If you want to build a larger distance network, you may want to know
that there are several meteorological institutes working on extending
the 10 km range. One may replace the standard SFP's by longer distance
ones (see Non-compliant SFP types), and there may
be ways of multiplexing specific wavelengths on a dark channel in a
Telecom network. Also light amplifiers may be used. See for example the
data of Vrije Universiteit Amsterdam and VSL at Users of WR
technology. At the moment of writing (April 2013) no links
longer than 16 km have been made though. The Finnish Metrology
Institute MIKES has set up a White Rabbit link between two PCs
that are connected with a 1000 (one thousand!) kilometer link.
The NEAT-FT Workshop on Optical Networks for Accurate Time and Frequency Transfer held in 2012 may give you more ideas and you possibly can find partner institutes that can help you with such a system. In 2016, other links, spanning well over 100 km have been presented at the Ninth White Rabbit Workshop.
Q: How much of the 1 Gbps bandwidth can I use?
A: We tested synchronization with a load of +95% bandwidth. In principle, PTP/WR requires much less than 1% of bandwidth (i.e. each second: 4 x min size frame + 1 frame of less than 100 bytes). If your network design is such that you plan to use over 99% of the bandwidth, it seems a rather bad design. If you leave a reasonable 10-20% of bandwidth margin, this is more than enough for WR/PTP to work. What is more, WR synchronization relies heavily on L1 (physical) syntonization which is independent from traffic (the frequency is transferred in the carrier with and without traffic), an incidental loss of PTP frames due to temporary peak of traffic load is acceptable by the PTP and will not break WR sub-ns synchronization which is maintained constantly by L1 syntonization.
Q: Where can I put additional delays coming e.g. from optical amplifiers to be able to compensate them?
A: For compensating additional (constant) assymetry you can modify deltaTx and deltaRx parameters in the SFP database stored in EEPROM (for WR PTP Core) or in a config file (for Wr Switch). Please take a look on our calibration procedure draft (https://www.ohwr.org/project/white-rabbit/wikis/Documents/White-Rabbit-calibration-procedure) and the WR PTP Core User's Manual (https://www.ohwr.org/documents/192). Basically what you have to do is to use "sfp add" command to specify Tx and Rx delays for particular SFP transceiver. In fact our delay model says that those Tx/Rx delays are the sum of constant delays of transmission/reception path so you can add your correction value there.
Those values are loaded by the PTP daemon only at the initialization time, so if one wants to update them, restarting the daemon is required.
Q: What limits the timing performance for the long-distance WR links?
A: Basically, the timing performance of a WR network can be limited by any link asymmetry not taken into account in the White Rabbit link delay model:
- If, instead of WDM in a single fiber, two separate fibers are used for the transmission in both directions the latency of both links has to be measured and compensated (asymmetry caused by not equal fibers' length). Moreover, if those two fibers are not part of the same cable, their operation temperature will differ causing additional link asymmetry not foreseen in the WR link delay model.
- If optical amplifiers are used, their delays have to be measured and taken into account. Asymmetry caused by different operation temperature of each amplifier are not taken into account in WR link delay model.
- Constant delays (e.g. delays from optical amplifiers) can be taken into account during the calibration and summed together with Tx and Rx hardware delays (check White Rabbit Calibration).
- Large values of delays or asymmetry are not a problem. The PTP software uses 64-bit variables as well as 64-bit mathematical operations.
Other effects are:
- Temperature effects on the fibre itself. Over a large temperature range and 10km, this can vary up to 200ps.
- Restarts of links may give an internal difference of up to 200ps. (see also another FAQ question)
So at the end of the day, just count on it being better than 1ns for links up to 10km. Only with much effort one can reach much better than that.
Q: May we use optical patch panels in the network or do you need direct connections?
A: There is no need to have direct connections between WR nodes. You can use patch panels in a WR network without any problem. Just make sure you have enough optical budget as each patch attenuates a bit of the light. This usually should not be a problem, but it is good to calculate this anyway.
Q: Any restriction on which fibre type should be used?
A: In 2010 we recommended to use a G652 type, that is optimized for dispersion. I am not sure if this requirement is still needed. (EB, 19/1/16)
Q: We have existing fibres already available which are meant for 10 Gb/s networks. Can I use these?
A: White Rabbit uses a 1 Gb/s transmission over a single mode fibre. So
as long as the fibre is for single-mode transmission, any fibre is OK.
The speed of data that you transfer over a fibre is not important.
So you will get the full White Rabbit performance with the fibre you
have installed.
If you look at the datasheet of the fibre you have, it will say
somewhere 9 um (mode field diameter). This specifies that the light can
only follow a straight line through it, i.e. single mode.
Multi-mode fibres have a field diameter of 50 um or 62.5 um, meaning
that the light can follow many different routes through it, including
ones that reflect via the sides. This makes that a short light pulse
will be spread out during the transmission. That’s why you cannot run
long distances with multi-mode
fibres.
Q: WR uses a single single-mode fibre, but would a short (~5m) multi-mode fibre link work too?
A: Multi-mode fibre links work fine. However you should redo timing calibration for your particular setup. Alpha (i.e. fiber asymmetry) is now caused by the difference in fibre length, not by chromatic dispersion. So for precise timing you need to calibrate each individual link.
Q: What is the accuracy and resolution of the round-trip time that is calculated?
A: The accuracy of the round-trip time measurement will be mostly determined by the accuracy of the frequency source you feed to the Grandmaster (the switch that is assigned to distribute the time, in many cases connected to a GPS or other clock). The resolution is approximately (1/62.5MHz)*(1/2^14) = 1 ps. However, also the implementation of the calculation can make a big difference: the implementation in the WRS is better than 1 ps, while the calculation in the end nodes may introduce an error in the order of 100 ps for a 10 km link.
The measurement of round-trip time is done by counting ticks of a
62.5MHz clock (which is ultimately derived from whatever clock signal
you feed to the GM) and enhancing that result (integer number of ticks)
with a fractional part using a DDMTD phase detector. The DDMTD gives you
a zooming effect of 2^N, with N=14 for the current switch gateware.
More details can be found in the document White Rabbit clock
synchronization: ultimate limits
...
Note that the synchronization is achieved by calculating master-to-slave
delay based on the round-trip measurement and calculation. This is quite
a tricky calculation because it involves large and small values (i.e.
round-trip and alpha parameter). The implementation of this calculation
can influence accuracy of synchronization on very long links (e.g.
quantization errors and limited number of bits when doing the
calculations in fixed-point arithmetic). The current implementations of
this calculations for the WR switch and node are explained in the a
short, unpublished,
write-up
(these might change for the PTP-HA High Accuracy implementation in
future, contact Maciej
Lipinski
if you need more information).
The calculation on the WRS and on a node (using WRPC) is done
differently. The calculation on the WRS is more precise that the
calculation on WRPC. This is because on a WRS the ARM processor can use
floating point calculations that cannot be used in the WRPC that has
more limited resources (notably RAM for storing the program). The
synchronization error due to limited calculation precision on WRPC (e.g.
quantization error) may be in the order of 100 ps for a 10 km link (to
be further characterized). While White Rabbit switches will remain very
precisely synchronized, an error (in for example the PPS and 10 MHz
outputs) on an end node can be caused by this calculation. This is work
requires more study and may be improved in the PTP-HA implementation.
(CERN, 22/11/18)
Q: What about start-up variations in shown cRTT values?
A: Every time a link is restarted, the round trip time is somewhat different. On a SPEC measurements show that it can be around 100 ps (A.Wallin), while a restart of the WR Switch may show variations of 10 ps (A.Wallin) to 30 ps (J.Diaz).
This start-up variations are caused by some internal functionality in the gigabit serialisers and is being worked on (Nov.2018). Actually a part of it may be caused by the precision of the calculations (see previous Q&A).
Update (March 2020):
WR with Low Phase Drift Calibration (LPCD)
- as of next WR switch release (V6.0), this feature will work by default on ports 1-12 of the WR switch (we do not have sufficient number of clocking resources in the FPGA to make it available for all ports)
- It mitigates imprecision of bitslide measurement (~+/-35ps for GTX) which limited the accuracy to 100ps on a single link.
- What we know so far is that it allows to keep the offset between PPS of the master and a slave to within 10ps after replugging the link and restarting switches.
- The expectation is that it will improve the accuracy of synchronization on a single link to <10ps.
Q: How could one create a performant wire tap on a port of a WR Switch ? / How to create a stable mirroring of a given port of a WR Switch to its another port?
A: As of v6.0.0, WR Switch supports mirroring of ports (separately tx and rx can be set), see options of the command rtu_stat:
ctdwa-774-cbt#rtu_stat -h
usage: rtu_stat <command> <values>
help: Show this message
list: List the routing table (same as empty command)
[..cut..]
mirror ingress <src_port> <dst_port>: Enable mirroring of ingress traffic
from src_port to dst_port
mirror egress <src_port> <dst_port>: Enable mirroring of egress traffic
from src_port to dst_port
mirror off: Turn off port mirroring
See also:
Frequently Asked Questions about the White Rabbit Switch
Frequently Asked Questions about the White Rabbit PTP Core
17 September 2021