@@ -66,7 +66,7 @@ that WR will continue to proliferate in scientific applications and should soon
\vspace{-0.1cm}
\section{Introduction}
White Rabbit (WR) \cite{biblio:whiteRabbit} is an innovative technology that provides sub-nanosecond
accuracy and \textcolor{red}{tens of} picoseconds precision of synchronization as well as deterministic and
accuracy and tens of picoseconds precision of synchronization as well as deterministic and
reliable data delivery for large distributed systems.
% \textcolor{gray}{
% The project with the same name
...
...
@@ -88,13 +88,13 @@ to meet the requirements of WR applications. A WR network
is a Bridged Local Area Network (IEEE 802.1Q) that uses Ethernet
(IEEE 802.3) to interconnect network elements and
Precision Time Protocol (PTP, IEEE 1588) to synchronize
them. \textcolor{red}{The WR network elements, i.e. 802.1Q bridges and end stations,
are called WR switches and WR nodes, respectively, and}
them. The WR network elements, i.e. 802.1Q bridges and end stations,
are called WR switches and WR nodes, respectively, and
implement WR enhancements:
% A WR network consists of \textcolor{red}{(802.1Q bridges and end stations, called switches and nodes respectively,)} and nodes
% \textcolor{red}{(802.1Q end stations)} that implement WR enhancements:
\begin{enumerate}
\item\textbf{Synchronization with \textcolor{red}{sub-nanosecond} accuracy and \textcolor{red}{tens of} picoseconds precision}\textcolor{red}{among} all
\item\textbf{Synchronization with sub-nanosecond accuracy and tens of picoseconds precision} among all
WR switches/nodes. Such synchronization is provided by the WR extension to PTP (WR-PTP,
\cite{biblio:WRPTP}) and its supporting hardware \cite{biblio:ISPCS2011}\cite{biblio:TomekMSc}\cite{biblio:WRproject}.
\item\textbf{Deterministic and low-latency communication} between WR nodes provided by
...
...
@@ -113,19 +113,19 @@ manner ensuring at most a single failure per year for a network of 2000 WR nodes
% \end{figure}
Since its conception in 2008, the number of WR applications has grown beyond
any expectations. The \textcolor{blue}{WR Users website \cite{biblio:WRusers}} attempts to keep
any expectations. The WR Users website \cite{biblio:WRusers} attempts to keep
track of WR applications. Apart from the suitable synchronization performance, the reasons for such a proliferation of WR applications
are the open nature of the WR project \textcolor{red}{and} the fact that the WR technology is based
are the open nature of the WR project and the fact that the WR technology is based
on standards. The former encourages collaboration,
reuse of work and adaptations that also prevent vendor lock-in. The latter allows using
off-the-shelf solutions with WR networks and catalyzes
collaboration with companies. \textcolor{blue}{
collaboration with companies.
What started as a project to renovate one of the most critical systems at CERN,
GMT \cite{biblio:GMT}\cite{biblio:GMTJavierPres}, is now a multilaboratory,
multicompany and multinational collaboration developing a technology that is
commercially available, used all over the world, and being incorporated into
the original PTP standard \cite{biblio:P1588}\cite{P1588-HA-enhancements}.
}
% This article attempts at providing a snapshot of the various WR applications,
% the ongoing work on enhancing WR and evolution of WR into IEEE 1588.
The first application of WR was in the second run of the CERN Neutrinos to Gran
Sasso (CNGS) experiment \cite{biblio:wr-cngs} and required timestamping of
events at the extraction and detection of neutrinos. Two WR
...
...
@@ -495,13 +495,12 @@ WR switch connected to the time reference \cite{biblio:PolaRx4e}\cite{biblio:CS4
a WR switch in the underground cavern and a number of WR nodes timestamping
input signals. The measured timestamping performance of the deployed system over 1 month of
operation was 0.517 ns accuracy and 0.119 ns precision.
}
The most demanding WR applications in terms of timestamping are cosmic ray and
neutrino detectors that record the time of arrival of particles in individual
detector units distributed over \textcolor{red}{distances up to several} kilometers.
detector units distributed over distances up to several kilometers.
Based on the difference
in the \textcolor{red}{times of arrival of the same particles detected by different unit}, the trajectories of these particles are calculated. For these
in the times of arrival of the same particles detected by different unit, the trajectories of these particles are calculated. For these
applications, a high precision and accuracy is required in harsh
environmental conditions due to their locations \cite{biblio:TAIGA-WR-harsh-env}.
...
...
@@ -515,7 +514,6 @@ calibrated using a portable calibrator \cite{biblio:LHAASO-WR-calibrator}.
These methods have proved to work in a prototype installation that has been running
since 2014 (50 WR nodes, 4 WR switches in 4 layers, \cite{biblio:LHAASO-WR-prototype}).
\textcolor{blue}{
The Cubic Kilometre Neutrino Telescope (KM3NeT)\cite{biblio:KM3NeT} is a research
infrastructure housing the next generation neutrino telescopes located at the
bottom of the Mediterranean Sea, off-shore France and Italy. The needed angular
...
...
@@ -525,18 +523,17 @@ few 100~ps precision. 4140 DOMs at 3500~m depth 100~km off-shore of Italy and
2070 DOMs at 2475~m depth 40~km off-shore France will be synchronized with an on-shore
reference using WR network \cite{biblio:WR-KM3NeT-Letter}\cite{biblio:WR-KM3NeT-presentation}.
Initial tests have been successfully performed with 18 DOMs off-shore France and Italy to validate the system.
}
Other applications of WR that use timestamping include
The fixed-latency data transfer is used in the BTrain-over-WhiteRabbit (WR-BTrain)
\cite{biblio:WR-Btrain} system that distributes in real-time the value of the magnetic field in CERN accelerators.
...
...
@@ -691,29 +688,28 @@ the CERN accelerators, except LHC, should be running WR-BTrain operationally \ci
% WR switches and 2-5 WR nodes.
% }
\textcolor{blue}{
Fixed-latency data transfer is considered for the operation of the
Nuclotron-based Ion Collider Facility (NICE) at the Joint Institute for Nuclear
Research (JINR) \cite{biblio:JINR} that already uses WR as the main clock
and time distribution system \cite{biblio:JINR-WR}.
}
\section{Radio-Frequency Transfer (RF)}
\label{sec:RFoverWR}
\subsection{\textcolor{blue}{Basic Concept}}
\subsection{Basic Concept}
Radio-frequency (RF) transfer \textcolor{red}{over WR network} allows \textcolor{red}{the digitization of} periodic input signals in a WR master node,
\textcolor{red}{the sending of}
their digital form over a WR network, and \textcolor{red}{the subsequent regeneration of the} signal coherently with a fixed delay in many
WR slave nodes. \textcolor{red}{Such a digital RF transfer provides a number of advantages over an analogue transmission of RF signals. For example, it is
Radio-frequency (RF) transfer over WR network allows the digitization of periodic input signals in a WR master node,
the sending of
their digital form over a WR network, and the subsequent regeneration of the signal coherently with a fixed delay in many
WR slave nodes. Such a digital RF transfer provides a number of advantages over an analogue transmission of RF signals. For example, it is
scalable and allows transmission of multiple RF signals from multiple sources over a single WR network
whereas analogue transmission typically requires dedicated network per source and signal. It also allows easy and automatic phase-alignment of the output
RF signals with compensation for temperature changes of transmission cables whereas such alignment and compensation in analogue transmission is very challenging.}
RF signals with compensation for temperature changes of transmission cables whereas such alignment and compensation in analogue transmission is very challenging.
% Lastly, the digital RF transfer over WR network minimizes bandwidth of transmitted data allowing to use WR network also for other purposes.
In \textcolor{red}{the RF transfer over WR Network} schema, depicted in Figure~\ref{fig:RFoverWR} and detailed in \cite{biblio:WR-LIST},
In the RF transfer over WR Network schema, depicted in Figure~\ref{fig:RFoverWR} and detailed in \cite{biblio:WR-LIST},
\begin{figure}[!ht]
\centering
\vspace{0.2cm}
...
...
@@ -731,30 +727,28 @@ that is sent over WR network. Each of the receiving WR slave nodes recreates the
by using the received tuning words to control the local DDS with a fixed delay.
In such way, the WR slave nodes produce RF outputs that are syntonized with
the RF input, phase-aligned among each other, and delayed with respect to the RF input --
all with \textcolor{red}{sub-nanosecond} accuracy and \textcolor{red}{tens of} picoseconds precision.
\textcolor{red}{Such a performance is possible because the noise of the DDS and the reference clock signal provided by WR to digitize/synthesize
all with sub-nanosecond accuracy and tens of picoseconds precision.
Such a performance is possible because the noise of the DDS and the reference clock signal provided by WR to digitize/synthesize
the RF signal are much lower than the required characteristics
of the digital RF signal transmission, thus negligible.}
of the digital RF signal transmission, thus negligible.
taking care for the effects described in the next section.
Links longer than 80~km require active amplifiers and/or unidirectional fibers.
This deteriorates accuracy due to an unknown asymmetry.
...
...
@@ -1253,7 +1247,7 @@ However, the variation of fiber temperature results in changes of the actual
while the variation of WR nodes/switches temperature result in laser wavelength
variation (e.g. 17~ps/nm km for 1550 nm). These and other
effects analyzed in \cite{biblio:SKA-80km} are significant on long links and
can amount to over \textcolor{blue}{3~ns} inaccuracy for \textcolor{blue}{80 km} bidirectional link using 1490/1550~nm
can amount to over 3~ns inaccuracy for 80 km bidirectional link using 1490/1550~nm
and exposed to 50 degrees Celsius temperature variation. The Square Kilometre Array (SKA) \cite{biblio:SKA}
radio telescope mitigates these effects to achieve $<$1~ns accuracy on 80~km links
by using DWDM SFP on ITU channels C21/C22 (1560.61/1558.98~nm) and combining them
...
...
@@ -1275,9 +1269,9 @@ on a single fiber via a simple DWDM channel filter, as described in \cite{biblio
\label{sec:}
The accuracy of WR depends greatly on the calibration of hardware delays.
\textcolor{blue}{WR has been using}
WR has been using
procedures for relative calibration of these delays \cite{biblio:wrCalibration}.
With relative calibration, \textcolor{red}{sub-nanosecond} accuracy can be achieved provided that the
With relative calibration, sub-nanosecond accuracy can be achieved provided that the
synchronized WR devices are calibrated against the same "golden calibrator".
% This is
% because the obtained values of hardware delays are estimates that are biased. The bias
...
...
@@ -1286,8 +1280,8 @@ synchronized WR devices are calibrated against the same "golden calibrator".
% and needs to be repeated each time a composing elements changes.
\textcolor{blue}{The recently completed work on absolute calibration \cite{biblio:WR-calibration}\cite{biblio:WR-CALIB-ABSOLUTE}\cite{biblio:WR-CALIB-ABSOLUTE-2}} allows
\textcolor{red}{the precise measurement of the} actual value of hardware delays and their different contributors.
The recently completed work on absolute calibration \cite{biblio:WR-calibration}\cite{biblio:WR-CALIB-ABSOLUTE}\cite{biblio:WR-CALIB-ABSOLUTE-2} allows
the precise measurement of the actual value of hardware delays and their different contributors.
With such calibration, a "golden calibrator" will not be required and adding a new type of component
(e.g. SFP) to a WR network will not necessitate a time-consuming calibration of all
devices with this component.
...
...
@@ -1296,7 +1290,7 @@ devices with this component.
\label{sec:WRin1588}
The P1588 Working Group \cite{biblio:P1588} is revising the
IEEE 1588 standard, due to \textcolor{red}{be finished} in 2019. This group has been studying
IEEE 1588 standard, due to be finished in 2019. This group has been studying
WR in order to incorporate its generalized
version into the standard \cite{P1588-HA-enhancements}.
This resulted in a third Default PTP Profile, High Accuracy,
...
...
@@ -1307,14 +1301,14 @@ additions are functionally equivalent to WR and allow the support of WR hardware
Along with the new features, informative annexes are added with a "standardized"
description of the WR calibration procedures \cite{biblio:wrCalibration} and
an example implementation of the High Accuracy profile that achieves
\textcolor{red}{sub-nanosecond} synchronization, a.k.a White Rabbit. The mapping between WR and High Accuracy
sub-nanosecond synchronization, a.k.a White Rabbit. The mapping between WR and High Accuracy
is described in \cite{biblio:WRin1588}.
% \newpage
\section{Conclusions}
\label{sec:conclusions}
WR is an innovative solution to provide \textcolor{red}{sub-nanosecond} accuracy and \textcolor{red}{tens of} picoseconds
WR is an innovative solution to provide sub-nanosecond accuracy and tens of picoseconds
precision of synchronization over large distances.
The number of WR applications and their specifications have exceeded the original
