Skip to content

W
White Rabbit
Commit 9eb3dd97 authored by Maciej Lipinski's avatar Maciej Lipinski
Browse files
Options

implemented feedback (most) from Erik/Javier + added missing citations to table

parent 25e59929
Hide whitespace changes
Inline Side-by-side
Showing with 144 additions and 138 deletions
papers/ISPCS2018/wrApplicationsOverview.bib
View file @ 9eb3dd97
......@@ -509,4 +509,12 @@ year = "2016",
@Misc{biblio:WRin1588,
title = "{White Rabbit integration into IEEE15880-20XX as High Accuracy}",
howpublished = {\url{www.ohwr.org/projects/wr-std/wiki/wrin1588}},
}
\ No newline at end of file
}
@Misc{biblio:WRusers,
title = "{White Rabbit Users}",
howpublished = {\url{www.ohwr.org/projects/white-rabbit/wiki/WRUsers}},
}
@Misc{biblio:WRcompanies,
title = "{Companies producing White Rabbit Devices}",
howpublished = {\url{www.ohwr.org/projects/white-rabbit/wiki/wrcompanies}},
}
papers/ISPCS2018/wrApplicationsOverview.tex
View file @ 9eb3dd97
......@@ -66,29 +66,29 @@ White Rabbit (WR) extends the Precision Time Protocol (PTP)
\section{Introduction}
White Rabbit (WR) \cite{biblio:whiteRabbit} is a
multilaboratory, multicompany and multinational collaboration to
develop new technology providing a versatile solution for control and data acquisition
systems. This new technology is also called White Rabbit and provides sub-nanosecond
accuracy and picoseconds precision of synchronization as well and deterministic and
develop new a technology providing a versatile solution for control and data acquisition
systems. With the same name, this new technology provides sub-nanosecond
accuracy and picoseconds precision of synchronization as well as deterministic and
reliable data delivery for large distributed systems.
WR is based on well-established networking standards, extending them if needed,
WR is based on well-established networking standards, extending them when needed,
to meet the requirements of WR applications. A WR network
% , depicted in Figure~\ref{fig:WRN},
is a Bridged Local Area Network (IEEE 802.1Q \cite{biblio:802.1Q}) that uses Ethernet
(IEEE 802.3 \cite{biblio:IEEE802.3}) to interconnect network elements and
Precision Time Protocol (PTP, IEEE 1588-2008 \cite{biblio:IEEE1588}) to synchronize
these network elements. A WR network consists of WR switches and WR nodes
them. A WR network consists of WR switches and WR nodes
that implement WR enhancements:
\begin{enumerate}
\item \textbf{Synchronization with sub-ns accuracy and picoseconds precision} between all
WR switches/nodes, such synchronization is provided by the WR extension to PTP (WR-PTP,
WR switches/nodes. Such synchronization is provided by the WR extension to PTP (WR-PTP,
\cite{biblio:WRPTP}) and its supporting hardware \cite{biblio:ISPCS2011}\cite{biblio:TomekMSc}\cite{biblio:WRproject}.
\item \textbf{Deterministic and low-latency communication} between WR nodes provided by
a purposely customized and open design of the WR switch, described in
\cite{biblio:MaciekPhD}.
\end{enumerate}
Studies \cite{biblio:MaciekPhD}\cite{biblio:CesarPhD}\cite{biblio:JosePhD} has shown
that both of the above enhancements can be further extended to operate in highly reliable
that both of the above enhancements can be further extended to operate in a highly reliable
manner ensuring at most a single failure per year for a network of 2000 WR nodes.
% \begin{figure}[!ht]
% \centering
......@@ -99,11 +99,11 @@ manner ensuring at most a single failure per year for a network of 2000 WR nodes
% \end{figure}
Since its conception in 2008, the number of WR applications has grown beyond
any expectations. The reasons for such a proliferation of applications
any expectations \cite{biblio:WRusers}. The reasons for such a proliferation of applications
are the open nature of the WR project and the fact that the WR technology is based
on well-know and well-established standards. The former encourages collaboration,
reuse of work, adaptations and prevents vendor locking. The latter allows using
of-the-shelf solutions (hardware/software) with WR networks and catalyzes
on well-known and well-established standards. The former encourages collaboration,
reuse of work and adaptations that also prevent vendor lock-in. The latter allows using
off-the-shelf solutions (hardware/software) with WR networks and catalyzes
collaboration with companies.
% What started as a project to renovate one of the most critical systems at CERN,
......@@ -113,16 +113,16 @@ collaboration with companies.
% the original PTP standard \cite{biblio:P1588WG}\cite{P1588-HA-enhancements}.
This article attempts at providing a snapshot of the various WR applications,
ongoing work on enhancing WR and WR's evolution into IEEE1588.
The article briefly describes in Section~\ref{sec:wrElements} portfolio of
the ongoing work on enhancing WR and evolution of WR into IEEE1588.
The article briefly describes in Section~\ref{sec:wrElements} the portfolio of
readily available WR network elements. It then explains in
Sections~\ref{sec:time-and-freq}-\ref{sec:RFoverWR} different types of
WR applications, their basic concept and examples WR-based system,
WR applications, their concept and use examples,
summarized in Table~\ref{tab:applications}. Some of the described applications
requires enhancements of WR performance, this enhancements are described in
require enhancements of WR performance. These enhancements are described in
Section~\ref{sec:WRenhancements}. Finally, in Section~\ref{sec:WRin1588} we
briefly describe the about-to-complete integration of WR in the upcoming revision of the
IEEE1588 standard and conclude in Section~\ref{sec:conclusions}.
IEEE1588 standard and we conclude in Section~\ref{sec:conclusions}.
\begin{table}[!t]
\caption{Non-exhaustive list of White Rabbit applications}
\centering
......@@ -130,7 +130,7 @@ IEEE1588 standard and conclude in Section~\ref{sec:conclusions}.
\begin{tabular}
{| p{0.9cm} | p{1cm} | p{0.6cm} | p{0.51cm} | p{0.9cm} | p{0.9cm} | p{1.1cm} |} \hline
& & & & \multicolumn{2}{c |}{\textbf{ Network Size}} & \\
\textbf{Facility}&\textbf{Location}&\textbf{Type}&\textbf{Link} & \textbf{in 2018}& \textbf{$>$2020} &\textbf{Ref.} \\
\textbf{Facility}&\textbf{Location}&\textbf{Type}&\textbf{Link} & \textbf{in 2018}& \textbf{$>$2020} &\textbf{Reference} \\
& & &\textbf{Len} & N / S / L & N / S / L & \\
& & & (max) & & & \\ \hline
......@@ -139,9 +139,9 @@ IEEE1588 standard and conclude in Section~\ref{sec:conclusions}.
CERN & Switz. & TF & 10km & 0/2/1 & 0/2/1 & \\ \hline
CERN & Switz. & FL & 1km & 6/2/1 & 20/8/1 & \cite{biblio:wr-streamers}\cite{biblio:WR-Btrain}\cite{biblio:WR-Btrain-MM} \cite{biblio:WR-BTrain-RF}\cite{biblio:WR-Btrain-status}\\ \hline
CERN & Switz. & TD & 10km & & & \cite{biblio:WR-LIST}\cite{biblio:WR-LIST-2}\cite{biblio:WRXI} \\ \hline
CERN & Switz. & RF & 10km & & & \\ \hline
CERN & Switz. & TC & & & & \\ \hline
GSI & Germany & TC & 1km & & & \\ \hline
CERN & Switz. & RF & 10km & & & \cite{biblio:WR-LIST} \\ \hline
CERN & Switz. & TC & 10km & & & \\ \hline
GSI & Germany & TC & 1km & & & \cite{biblio:WR-GSI}\cite{biblio:FAIRtimingSystem} \\ \hline
JINR & Russia & TS & 1km & 50/5/3 & & \cite{biblio:JINR-WR} \\ \hline
JINR & Russia & TS,TD & 1km & & 200/15/- & \cite{biblio:JINR-WR} \\ \hline
......@@ -162,20 +162,19 @@ SBN & USA & TS,TD & 1km & 6/1/1
LHAASO & China & TF,TS & 1km & 40/4/4 & 6734/564/4 & \cite{biblio:LHAASO}\cite{biblio:LHAASO-WR-temp}\cite{biblio:LHAASO-WR-calibrator} \cite{biblio:LHAASO-WR-prototype}\\ \hline
HiSCORE & Russia & TS & & & & \\ \hline
CTA & Spain/Chile & TF,TS & few km & 32/3/2 & 220/10/2 & \cite{biblio:CTA-WR-timestamps}\\ \hline
SKA & Australia/ Africa& TF & 80km & 2/1/1 & 233/15/3 & \\ \hline
\multicolumn{7}{|c|}{\textbf{National Time Laboratories}} \\ \hline
MIKES & Finland & TF & 950km & 10/few/2 & & \\ \hline
NE-SYRTE & France & TF & 125km & 4/2/4 & & \\ \hline
VLS & Nederland & TF & 137km & & & \\ \hline
NIST & USA & TF & 10km & & & \\ \hline
MIKES & Finland & TF & 950km & 10/few/2 & & \cite{biblio:MIKES-50km}\cite{biblio:MIKES+VSL} \\ \hline
LNE-SYRTE & France & TF & 125km & 4/2/4 & & \cite{biblio:SYRTE-LNE-25km}\cite{biblio:SYRTE-LNE-500km} \\ \hline
VLS & Nederland & TF & 137km & & & \cite{biblio:MIKES+VSL} \\ \hline
NIST & USA & TF & 10km & & & \cite{biblio:WR-NIS} \\ \hline
NLP & UK & TF & & & & \\ \hline
INRIM & Italy & TF,TS & 400km & & & \\ \hline
INRIM & Italy & TF,TS & 400km & & & \cite{biblio:WR-INRIM}\cite{biblio:WR-INRIM-400km} \\ \hline
\multicolumn{7}{|c|}{\textbf{Other Applications}} \\ \hline
SKA & Australia/ Africa& TF & 80km & 2/1/1 & 233/15/3 & \cite{biblio:SKA-80km} \\ \hline
DLR & Germany & TS & 1km & & & \cite{biblio:ELI-BEAMS-WR} \\ \hline
ELI-ALPS & Hungry & TS & 1km & & & \cite{biblio:ELI-ALP-WR} \\ \hline
ELI-BEAMS & Czech & TF,TS, TD,TC& 1km & 70/16/2 & & \cite{biblio:ELI-BEAMS-WR} \\ \hline
......@@ -203,8 +202,8 @@ WR network elements, nodes and switches, are openly available on the Open Hardwa
\cite{biblio:OHWR} and can be purchased from companies. %, see Figure~\ref{fig:WRN}.
While all of the WR networks use the same design of
the WR switch \cite{biblio:wr-switch},
the design of WR nodes depends on application. Thus, WR node design is available
as open-source IP core \cite{biblio:wr-node} that can be easily used in one of
the design of WR nodes depends on the application. Therefore the WR node design is made available
as an open-source IP core \cite{biblio:wr-node} that can be easily used in one of
the supported boards or integrated into a custom design. WR-compatible boards
are available on OHWR in various form factors, including:
% Peripheral Component Interconnect Express (PCIe) \cite{biblio:spec},
......@@ -219,11 +218,11 @@ AMC \cite{biblio:AFC}\cite{biblio:AFCK},
FMC \cite{biblio:cute-wr-dp},
cRIO \cite{biblio:crio} and
PXI \cite{biblio:spexi}.
All of these boards are commercially available. Morover, more and more
companies integrate WR into their products, e.g.
All of these boards are commercially available \cite{biblio:WRcompanies}.
Morover, more and more companies integrate WR into their products,
\cite{biblio:STRUCK}\cite{biblio:sundance}\cite{biblio:spdevices}.
Such a variety of WR nodes facilitaties
implementations of WR applications described in the following sections
implementations of WR applications as described in the following sections
% \begin{figure}[!ht]
......@@ -257,12 +256,12 @@ The most basic application of WR is the transfer of time and frequency from the
Grandmaster WR switch/node (Grandmaster) to all other WR switches/nodes in
the WR network. WR ensures that the Pulse Per Second (PPS) outputs of all the
WR switches/nodes in the WR network are aligned to the PPS output of the
Grandmaster with sub-ns accuracy and picoseconds precision. WR switches and
nodes use and can output clock signal (e.g. 10MHz, 125MHz) that is traceable to that
Grandmaster with a sub-ns accuracy and picoseconds precision. WR switches and
nodes use and can a output clock signal (e.g. 10MHz, 125MHz) that is traceable to that
of the Grandmaster.
In most applications, Grandmaster is connected to a clock reference. Typically,
it is a Cesium or Rubidium oscillator disciplined by a global
In most applications, the Grandmaster is connected to a clock reference. Typically,
this is a Cesium or Rubidium oscillator disciplined by a global
navigation satellite system (GNSS).
In such case, the time and frequency transferred by WR are traceable to
the International Atomic Time (TAI).
......@@ -279,21 +278,21 @@ built on top of it and described in the subsequent sections.
Time and frequency transfer is used by National Time Laboratories to
disseminate official UTC time and compare clocks. Laboratories in
disseminate the official UTC time and to compare clocks. Laboratories in
Finland (VTT MIKES), Netherlands (VSL), France (LNE-SYRTE), UK (NLP),
USA (NIST), Italy (INRIM) and South Korea (KRIS) have WR installations. MIKES and INRIM
use WR to provide their realization of UTC to clients, e.g. UTC(MIKE) over 50~km
to Metsähovi Observatory \cite{biblio:MIKES-50km} for applications in geodesy,
USA (NIST) and Italy (INRIM) have WR installations. MIKES and INRIM
use WR to provide their realization of UTC to clients, e.g. UTC(MIKE) over a 50~km
link to the Metsähovi Observatory \cite{biblio:MIKES-50km} for applications in geodesy,
% (International GNSS Service,
% Very-long-baseline interferometry and satellite laser ranging)
UTC(INRIM) over
400~km to the financial district of Millano. NIST uses WR to distribute UTC(NIST)
within their campus. All
laboratories are studying WR with different types and lenghts of fiber links and attempt to
400~km to the financial district of Millano. NIST and LNE-SYRTE use WR to distribute
within their campus UTC(NIST) and UTC(OP), respectively. All
laboratories are studying WR with different types and lengths of fiber links and attempt to
increase its performance. These studies showed that the stability (at tau=1s) of the off-the-shelf
WR switch is 1e-11 (Alan Deviation, ADEV, similar to typical frequency counter e.g. Keysight 53230A)
and can be improved. This prompted development of the Low-Jitter Daughterboard (LJD)
\cite{biblio:WR-LJD} that improves 1e-12 performance of the WR switch without any
WR switch is 1e-11 (Alan Deviation, ADEV, similar to a typical frequency counter e.g. Keysight 53230A)
and can be improved. This prompted the development of the Low-Jitter Daughterboard
\cite{biblio:WR-LJD} that improves the performance of the WR switch to 1e-12 without any
modifications to the WR-PTP Protocol, see
\cite{biblio:MIKES-50km}\cite{biblio:SYRTE-LNE-500km}\cite{biblio:WR-ultimate-limits}.
The improved WR Switches are now commercially available \cite{biblio:WR-LJD-switch}.
......@@ -306,7 +305,7 @@ The results from time laboratories studies are summarized in Table~\ref{tab:time
\begin{tabular}{
| p{0.7cm} | p{0.75cm} | p{2.3cm} | p{0.5cm} | p{1.5cm} | c | } \hline
\textbf{Time} & \textbf{Link} & \textbf{Link } & \textbf{Time } & \textbf{Time} & \textbf{Ref} \\
\textbf{Lab} & \textbf{Len } & \textbf{Type } & \textbf{Error} & \textbf{Stability} & \textbf{} \\ \hline
\textbf{Lab} & \textbf{Length } & \textbf{Type } & \textbf{Error} & \textbf{Stability} & \textbf{} \\ \hline
VTT & 950km & unidir. in DWDM & $\pm$2ns & 20ps@1000s & \cite{biblio:MIKES+VSL} \\ \cline{2-6}
MIKES & 50km & bidir. on adjacent ITU DWDM channels & $<$1ns & ~2e-12@1s (*) & \cite{biblio:MIKES-50km} \\ \hline
VSL & 2x137km & bidir. on CWDM (1470\&1490nm)(\#) & $<$8ns & 10ps@1000s & \cite{biblio:MIKES+VSL} \\ \hline
......@@ -380,17 +379,17 @@ INRIM & 70k m & bidir. in WDM
Many accelerators, synchrotrons and spallation sources are controlled by triggering
events in a pre-configured sequence of actions. In fact, it is a very convenient
way to control beams of particle that move at very large speeds, often close
way to control beams of particles that move at very high speeds -- often close
to the speed of light -- faster than the propagation of control signals.
In the \textit{time-triggered control} schema, a sequence of actions is determined
by a controller and distributed to controlled devices in advance. These actions
are scheduled to be executed by spatially-distributed devices at a particular time.
The responsiveness of such system greatly depends on the latency of
The responsiveness of such systems greatly depends on the latency of
delivering the control-information from the controller to the accelerator devices.
The precision of such system depends on synchronization quality between
these devices and the controller. WR provides precise/accurate synchronization and guarantees
upper-bound latency through the network to enable implementation of time-triggered control
The precision of such systems depends on the synchronization quality between
these devices and the controller. WR provides precise and accurate synchronization and guarantees an
upper-bound in latency through the network to enable the implementation of a time-triggered control
for accelerators.
\subsection{Example Applications}
......@@ -404,22 +403,22 @@ accelerators and will control GSI's new Facility for Antiproton and Ion Research
(FAIR) \cite{biblio:GSI-WR-GMT}. Control of GSI and FAIR requires that the
control-information is delivered from a common controller to any of the controlled
subsystems in any of the accelerators within 500$\mu s$. The most demanding of
these subsystems require accuracy of 1-5ns. The controller, called Data Master,
these subsystems require an accuracy of 1-5ns. The controller, called Data Master,
is a WR node. The subsystems are either WR Nodes or have a direct interface with WR Nodes.
All these WR nodes are connected to a common WR Network that provides synchronization,
delivers control-information from the Data Master to all subsystems as well as
between subsystems, and allows diagnostics.
When FAIR is completed in 2025, the WR network at GSI and FAIR will include
2000-3000 WR nodes connected to 300 WR switches in 5 layers. The WR-based
2000-3000 WR nodes connected to 300 WR switches in five layers. The WR-based
GMT has been operational at GSI since 2015. First, it was used to control
a small CRYRING accelerator build purposely to test WR-based GMT
\cite{biblio:GSI-WR-GMT-CRYRING} and consisting of 30 WR nodes in 3 layers of
a small CRYRING accelerator built purposely to test the WR-based GMT
\cite{biblio:GSI-WR-GMT-CRYRING} and consisting of 30 WR nodes in three layers of
WR switches. Then, the GMT system that had been used so far was replaced with WR-based GMT
that consists of 35 WR switches and it is commissions for operation, first beam in
that consists of 35 WR switches and it is commissioned for operation, with a first beam in
June 2018.
A WR-based GMT to control CERN accelerators has been the reasson for WR's
Although a WR-based GMT to control CERN accelerators has been the reason for WR's
conception and it is yet to be implemented at CERN.
Both, at CERN and GSI, the same WR network that is used for time-triggered control
can provide to subsystems precise time and frequency which can be used,
......@@ -435,14 +434,14 @@ section.
\section{Precise Timestamping (TS)}
\label{sec:timestamping}
\subsection{Basic Concept}
In a great numbers of applications, time and frequency are transferred in order
In a great number of applications, time and frequency are transferred in order
to timestamp accurately and/or precisely incoming signals. Such incoming signals
can be either discrete pulses timestamped with time-to-digital converters
can be either discrete pulses that are timestamped with time-to-digital converters
or analogue signals that are sampled (digitized) with the distributed frequency
and associated with the distributed time. Timestamps are usually produced to
measure time of flight (ToF) or correlate events between distributed systems. In
such case, accurate and precise synchronization between these systems is required.
If timestamps are used to measure duration of events detected by distributed subsystems,
measure the time of flight (ToF) or correlate events between distributed systems. In
such cases, accurate and precise synchronization between these systems is required.
If timestamps are used to measure the duration of events detected by distributed subsystems,
precision and frequency stability are required, accuracy is not so important.
......@@ -455,21 +454,21 @@ events at the extraction and detection of neutrinos. Two WR
networks were installed in parallel with the initial timing system: one at CERN and one in Gran Sasso. Each WR network consisted of a Grandmaster
WR switch connected to the time reference \cite{biblio:PolaRx4e}\cite{biblio:CS4000},
a WR switch in the underground cavern and a number of WR nodes timestamping
inputs signal. The measured performance of the deployed system over 1 month of
input signals. The measured performance of the deployed system over 1 month of
operation was 0.517 ns accuracy and 0.119 ns precision.
}
The most demanding WR applications in terms of timestamping are cosmic ray and
neutrino detectors that record the time of arrival of particles in individual
detector units distributed over up to tens of kilometers. Based on the difference
in the time of arrival, the trajectory of particles are calculated. For this
applications, high precision and accuracy is required in very harsh
environmental condition.
detector units distributed over up to kilometers. Based on the difference
in the time of arrival, the trajectory of particles are calculated. For these
applications, a high precision and accuracy is required in very harsh
environmental conditions due to their locations.
The Large High Altitude Air Shower Observatory (LHAASO), located at 4410m above
sea level in China (Tybert), requires 500~ps RMS \cite{biblio:LHAASO} alignment
sea level in China (Tibert), requires a 500~ps RMS \cite{biblio:LHAASO} alignment
of timestamps produced by 7000 WR nodes distributed over $1~km^2$ and exposed to
day-night variation of -10 to +55 Celsius degree. To meet such requirements, active
day-night variation of -10 to +55 degrees Celsius . To meet such requirements, active
compensation of temperature-related hardware delays has been implemented
\cite{biblio:LHAASO-WR-temp} and each of the WR nodes will be individually
calibrated using a portable calibrator \cite{biblio:LHAASO-WR-calibrator}.
......@@ -478,7 +477,7 @@ since 2014 (50 WR nodes, 4 WR switches in 4 layers, \cite{biblio:LHAASO-WR-proto
The Cubic Kilometre Neutrino Telescope (KM3NeT)\cite{biblio:KM3NeT} is a research
infrastructure housing the next generation neutrino telescopes located at the
bottom of the Mediterranean Sea, off-shore France and Itally. The needed angular
bottom of the Mediterranean Sea, off-shore France and Italy. The needed angular
resolution of 0.1 degree means that the submerged \textit{digital optical modules} (DOMs),
which constitute KM3NeT, must be synchronized with 1~ns accuracy and a
few 100~ps precision. 4140 DOMs at 3500~m depth 100~km off-shore of Italy and
......@@ -487,26 +486,26 @@ reference using WR network \cite{biblio:WR-KM3NeT-Letter}\cite{biblio:WR-KM3NeT-
Initial tests have been succesfully performed with 18 DOMs off-shore France and Italy to validate the system.
\textcolor{gray}{
Other applications of WR that use timestamping include Cherenkov Telescope Array
to be build in Chile and Spain \cite{biblio:CTA-WR-timestamps},
Extreme Light Infrastructures in Hungary \cite{biblio:ELI-ALP-WR} and Czech
Other applications of WR that use timestamping include the Cherenkov Telescope Array
to be built in Chile and Spain \cite{biblio:CTA-WR-timestamps},
the Extreme Light Infrastructures in Hungary \cite{biblio:ELI-ALP-WR} and Czech Republic
\cite{biblio:ELI-BEAMS-WR}, Satellite Laser Ranging at German Aerospace Center
or Power Industry and Smart Grid studied at Swiss Federal
Institute of Technology Lausanne (EPFL) \cite{biblio:EPFL-WR-PMU}.
}
\section{Triggers Distribution (TD)}
\section{Trigger Distribution (TD)}
\label{sec:triggers-distribution}
\subsection{Basic Concept}
Triggers distribution combines, to some extend, the time-triggered control and precise timestamping
described before. In this application, an input trigger signal is timestamped by a WR node and sent over
Trigger distribution combines, to some extend, the time-triggered control and precise timestamping
described before. In this application, an input trigger signal is timestamped by a WR node and sent over the
WR network to many WR nodes that act upon the received message simultaneously, at
a precise delay with respect to the input signal.
The input trigger can be either a pulse or an analogue signal exceeding a treshold.
Once the trigger occurs, the information about the trigger (e.g. ID), along with
the timestamp, is sent over WR network to other WR nodes, usually as a broadcast.
The deterministic characteristics of the WR network allow to determine the
the timestamp, is sent over the WR network to other WR nodes, usually as a broadcast.
The deterministic characteristics of the WR network allows to determine the
upper-bound latency for the message to reach all the WR nodes.
In order to make sure that all the "interested" nodes act upon the trigger
simultaneously, the delay between the input trigger and the time of execution
......@@ -514,11 +513,11 @@ is set to be greater than the upper-bound latency.
\subsection{Example Applications}
The \textit{trigger distribution} schema has been used at CERN since 2015 in the
The trigger distribution schema has been used at CERN since 2015 in the
WR Trigger Distribution (WRTD) system for instability
diagnostics of the LHC \cite{biblio:WR-LIST}\cite{biblio:WR-LIST-2}.
In the WRTD, there is a number of instruments capable of detecting
LHC instabilities and continuously acquiring data in round buffers. Upon detection of instabilities, such a device generates a
LHC instabilities and continuously acquiring data in circular buffers. Upon detection of instabilities, such a device generates a
pulse that is timestamped by a Time-to-Digital Converter (TDC) integrated in
a WR Node, as depicted in Figure~\ref{fig:WRTD}.
The timestamp produced by the TDC is broadcast over the WR network,
......@@ -546,13 +545,13 @@ The concept that has proven to work in WRTD is now being generalized to
provide trigger distribution for CERN's Open Analog Signals Information System
(OASIS) \cite{biblio:OASIS}. OASIS is a gigantic distributed oscilloscope that
provides hundreds of input channels and spans all CERN's accelerators except LHC.
Triggers in this system are currently distributed via coax cables without
delay compensation and multiplex using analogue multiplexers. In order to use
Triggers in this system are currently distributed via coax cables that may be 1~km long without
delay compensation and multiplexed using analogue multiplexers. In order to use
OASIS to diagnose LHC and to improve its performance, the distribution of triggers
is being upgraded to use WR. The WR-based trigger distribution in OASIS is meant
to be generic and reusable. It is developed within the White Rabbit eXtensions
for Instrumentation (WRXI) project \cite{biblio:WRXI} that is based on an existing LAN
eXtensions for Instrumentation (LXI) standard, extending it if necessary. The WRXI
for Instrumentation (WRXI) project \cite{biblio:WRXI} that is based on the existing LAN
eXtensions for Instrumentation (LXI) standard, extending it when necessary. The WRXI
for OASIS is meant to be operational in 2019.
\textcolor{gray}{
......@@ -564,7 +563,7 @@ experiment instrumentation at China Spallation Neutron Source (CSNS)
\section{Fixed-Latency Data Transfer (FL)}
\label{sec:fixed-latency}
\subsection{Basic Concept}
Fixed-latency data transfer provides a well-know and precise latency of data
Fixed-latency data transfer provides a well-known and precise latency of data
transmitted between WR nodes in the WR network. It uses very similar
principles to the trigger distribution described in Section~\ref{sec:triggers-distribution}.
The time of data transmission is timestamped and this timestamp
......@@ -572,26 +571,26 @@ is sent in the Ethernet frame with the data. When the data is received, a progra
delay is added to the transmission timestamp and the associated data is provided
to an application precisely at the delayed time. Such a functionality is implemented
by the so-called "WR Streamers" IP Core \cite{biblio:wr-streamers} which adds a
data transmission layer on top of WR and act as an fixed-latency FIFO over Ethernet.
data transmission layer on top of WR and acts as a fixed-latency FIFO over Ethernet.
By providing such functionality to the application, the application does not need
to be aware of time but rather processes data as it comes, knowing that all the
WR nodes in the WR network will execute the same action at the same time. In
order to take advantage of the precise fixed-latency data transfer, the
application needs to be integrated with WR node.
application needs to be integrated with a WR node.
\subsection{Example Applications}
The fixed-latency data transfer is used in the BTrain-over-WhiteRabbit (WR-BTrain)
\cite{biblio:WR-Btrain} system that distributes the value of magnetic field in
\cite{biblio:WR-Btrain} system that distributes in real-time the value of the magnetic field in
real-time in CERN accelerators.
In circular accelerators, the acceleration of beam by radio-frequency (RF) cavities
In circular accelerators, the acceleration of the beam by radio-frequency (RF) cavities
needs to be synchronized with the increase of magnetic field (B-field) in the
bending magnets. BTrain is the system at CERN that measures and distributes the
value of the magnetic field (B-value) in real-time to the RF cavities,
power converters and beam instrumentation.
While the RF cavities
simply follow the ramp of magnetic field, the power converters adjust the current
simply follow the ramp of the magnetic field, the power converters adjust the current
of the magnets such that the intended B-value is obtained, closing a control
loop. BTrain is essential to the operation of most of CERN accelerators, i.e.
Booster, PS, SPS, LEIR, AD, and ELENA.
......@@ -605,18 +604,17 @@ Booster, PS, SPS, LEIR, AD, and ELENA.
The original BTrain system uses coaxial cables to distribute pulses that indicate
increase and decrease of the B-value. This method is now being upgraded to a
WR-based distribution of the absolute B-value and other additional information
\cite{biblio:WR-Btrain-MM}. In this upgraded system, WR-BTrain, B-values are transmitted
at 250kHz (every $4\mu s$) from the measurement WR node to all the other WR nodes
WR-based distribution of the absolute B-value and additional information
\cite{biblio:WR-Btrain-MM}. In this upgraded system B-values are transmitted
at 250kHz (every $4\mu$s) from the measurement WR node to all the other WR nodes
that are integrated with RF cavities, power converters and beam instrumentation. In the most
demanding accelerator, SPS, the data must be delivered over 2 hops (WR switches)
with latency of $10\mu s\pm 8ns$.
The WR-BTtrain has been successfully evaluated in the PS accelerators where it has
been running operationally since 2017 \cite{biblio:WR-BTrain-RF}. It is now being
installed in the remaining accelerators. By 2021, all
been running operationally since 2017 \cite{biblio:WR-BTrain-RF}. By 2021, all
the accelerators should be running WR-BTrain operationally \cite{biblio:WR-Btrain-status}.
For each accelerator, a separated WR-Btrian system is installed consisting of 1-2
For each accelerator, a separated WR-BTrian system is installed consisting of 1-2
WR switches and 2-5 WR nodes.
\textcolor{gray}{
......@@ -631,7 +629,7 @@ and time distribution system \cite{biblio:JINR-WR}.
\label{sec:RFoverWR}
\subsection{Basic Concept}
Radio-frequency transfer allows to digitize periodic input signals in a WR master node, send
their digital form over WR network, and then regenerate them coherently with a fixed delay in many
their digital form over a WR network, and then regenerate this signal coherently with a fixed delay in many
WR slave nodes. In such schema, depicted in Figure~\ref{fig:RFoverWR} and detailed in \cite{biblio:WR-LIST},
\begin{figure}[!ht]
\centering
......@@ -643,7 +641,7 @@ WR slave nodes. In such schema, depicted in Figure~\ref{fig:RFoverWR} and detail
a digital direct synthesis (DDS) based on the WR reference clock
signal (125MHz) is used to generate an RF signal. The generated RF signal is then compared by a phase detector to the
input RF signal. The error measured by the phase detector is an input to a
loop filter (e.g. Integral-Proportional controller) that steers the DDS to produce signal identical
loop filter (e.g. Integral-Proportional controller) that steers the DDS to produce a signal identical
to the RF input - effectively the DDS is locked to the input signal.
The tuning words of the DDS are the digital form of the RF input
that is sent over WR network. Each of the receiving WR nodes recreate the RF input signal
......@@ -656,28 +654,28 @@ the RF input, phase-aligned among each other with sub-ns accuracy and picosecond
The WR-based radio-frequency transfer is being implemented in the European Synchrotron Radiation
Facility (ESRF) \cite{biblio:ESRF}. The operation of the ESRF accelerator facility
is controlled by a "Bunch Clock" system that delivers to accelerator subsystems
is controlled by a "Bunch Clock" system that delivers to accelerator subsystems a
$\approx$352 MHz RF signal and triggers initiating sequential actions synchronous
to the RF signal, such as "gun trigger", "injection trigger", "extraction trigger".
to the RF signal, such as "gun trigger", "injection trigger" or "extraction trigger".
The jitter of the output RF signal is required to be below 50~ps. The RF signal is continuously
trimmed around the 352~MHz value as the tuning parameter in the "fast orbit feedback"
process. Apart from 352~MHz signal, other frquencies are distributed, such as
355~kH Storage Ring revolution frequency or 10~Hz Injection sequence.
process. Apart from the 352~MHz signal, other frequencies are distributed, such as the
355~kHz Storage Ring revolution frequency or the 10~Hz Injection sequence.
The current ESRF "Bunch Clock" system is being refurbished to use WR \cite{biblio:ESRF-WR}.
The solution has passed 6-months validation tests in 2015. In 2016, a prototype system
The solution has passed a 6-months validation tests in 2015. In 2016, a prototype system
consisting of a WR switch, a WR master node and a WR slave node successfully injected
bunches in the storage ring providing $<$10ps jitter. A system consisting
of 1 master and 7 slaves is expected to be operational in July 2018. It
of one master and seven slaves is expected to be operational in July 2018. It
will be expanded to 40 WR slaves nodes and 4-5 WR switches by 2020.
The ESRF "Bunch Clock" system not only distributs a number of RF frequencies,
The ESRF "Bunch Clock" system not only distributes a number of RF frequencies,
it also provides timestamps and triggers that can be synchronous with these RF
frequencies.
The radio-frequency transfer in CERN's Super Proton Synchrotron (SPS)
accelerator is also being upgraded to use WR. The SPS requires distribution
of a ???~MHz RF signal with 0.25ps RMS jitter (100Hz to 100kHz) and accuracy of $\pm$10ps.
of a 200~MHz RF signal with 0.25ps RMS jitter (100Hz to 100kHz) and an accuracy of $\pm$10ps.
These requirements necessitate enhancements of WR performance.
% \section{WR Applications outside CERN}
......@@ -1076,7 +1074,7 @@ These requirements necessitate enhancements of WR performance.
\section{Performance Enhancements}
\label{sec:WRenhancements}
The growing number of applications provides needs and means for constant improvements
The growing number of applications catalyzes constant improvements
of WR performance. These are summarized in this section.
......@@ -1102,21 +1100,21 @@ achieving jitter of sub-100fs RMS from 100Hz to 20MHz \cite{biblio:SPS-WR-LLRF}.
\subsection{Temperature Compensation}
\label{sec:}
The studies \cite{biblio:LHAASO-WR-temp} has showed that temperature variation
The studies \cite{biblio:LHAASO-WR-temp} have shown that the temperature variation
of WR nodes and switches degrades synchronization performance, still
maintaining sub-ns accuracy over the measured 45 Celsius temperature range.
maintaining sub-ns accuracy over the measured 45 degrees Celsius temperature range.
This degradation and its sources have been carefully characterized
\cite{biblio:wr-cngs} showing that its major contributor is the variation of hardware
delays (\textit{fixed delays}), considering links below 10~km (see next section).
These delays are usually calibrated for WR devices
\cite{biblio:wrCalibration} in a room temperate and assumed constant throughout
\cite{biblio:wrCalibration} at a room temperature and assumed constant throughout
operation. Their variation however is linear with temperature and so an online
correction can be applied. Such real-time correction was developed for LHAASO
correction can be applied. Such real-time correction was developed for the LHAASO
experiment \cite{biblio:wr-cngs}. For temperatures between
-10 and 50 Celsius degrees, this method has show to reduce the synchronization
peak-to-peak variation to $<$150ps with standard deviation below 50 ps while
without the compensation the measured variation is 700ps. This online compensation
is used in the LHAASO to ensure 500ps (rms)
-10 and 50 degrees Celsius, this method has shown to reduce the synchronization
peak-to-peak variation from 700~ps to $<$150ps with a standard deviation below 50 ps.
This online compensation
is used in LHAASO to ensure 500ps (rms)
synchronization of 7000 WR nodes exposed to harsh environmental conditions
\subsection{Long-haul Link}
......@@ -1126,29 +1124,29 @@ Experiments have shown that WR can successfully provide sub-ns accuracy on bidir
\cite{biblio:WR-INRIM}\cite{biblio:WR-INRIM}\cite{biblio:SYRTE-LNE-25km}\cite{biblio:MIKES-50km}\cite{biblio:SKA-80km}
taking care for the effects described in the next section.
Links longer than 80~km require active amplifiers and/or unidirectional fibers.
This deteriorates accuracy due to unknown asymmetry.
On the 137~km bidirectional link in Netherlands \cite{biblio:MIKES+VSL},
dedicated optical amplifiers that work with bidirectional fibers is used in
an attempt to overcome this limitations. The tests so far has shown $<$8ns accuracy
and further improvements are studied.
This deteriorates accuracy due to an unknown asymmetry.
On the 137~km bidirectional link in the Netherlands \cite{biblio:MIKES+VSL},
dedicated optical amplifiers that work with bidirectional fibers are used in
an attempt to overcome these limitation. The tests so far have shown $<$8ns accuracy
and further improvements are being studied.
On the 950~km unidirectional link in Finland, GPS precise point positioning
(PPP) was used to calibrate asymmetry and achieve $\pm$2~ns accuracy. This
method requires re-calibration after any disruption of the network. Laboratory
tests of 500km WR connection using 5 cascaded WR devcies and four 125km unidirectional
link showed 2.5~ns peak-to-peak time error \cite{biblio:SYRTE-LNE-500km}.
tests of a 500km WR connection using five cascaded WR devicies and four 125km unidirectional
links showed a 2.5~ns peak-to-peak time error \cite{biblio:SYRTE-LNE-500km}.
\subsection{Link Asymmetry}
\label{sec:linkAsym}
WR estimates and compensates asymmetry of bidirectional links knowing the relation
between the wavelengths in the two directions. This relation, called \textit{alpha}
between the wavelengths in the two directions. This relation, characterized by the \textit{alpha}
parameter, is calibrated at room temperature and assumed constant.
However, variation of fiber temperature results in changes of the actual
\textit{alpha} (-0.12~ps/km K for 1310/1490~nm and -0.05~ps/km~K for 1490/1550~nm)
while variation of WR nodes/switches temperature result in laser wavelength
However, the variation of fiber temperature results in changes of the actual
\textit{alpha} (-0.12~ps/km/K for 1310/1490~nm and -0.05~ps/km/K for 1490/1550~nm)
while the variation of WR nodes/switches temperature result in laser wavelength
variation, e.g. 17~ps/nm km for 1550 nm \cite{biblio:SKA-80km}. These and other
effects analyzed in \cite{biblio:SKA-80km} are significant on long links and
can ammount to over 5ns inaccuracy for bidirectional link using 1490/1550nm
can amount to over 5ns inaccuracy for bidirectional link using 1490/1550nm
and exposed to 50 Celsius degrees temperature variation. The Square Kilometre Array (SKA) \cite{biblio:SKA}
radio telescope mitigates these effects to achieve $<$1ns accuracy on 80~km links
by using DWDM SFP on ITU channels C21/C22 (1560.61/1558.98~nm) and combining them
......@@ -1171,16 +1169,16 @@ on a single fiber via a simple DWDM channel filter, as described in \cite{biblio
The accuracy of WR depends greatly on the calibration of hardware delays. WR uses
procedures for relative calibration of these delays \cite{biblio:wrCalibration}.
With relative calibration, sub-ns accuracy can be achieve provided that the
With relative calibration, sub-ns accuracy can be achieved provided that the
synchronized WR devices are calibrated against the same "golden calibrator". This is
because the obtained values of hardware delays are estimates that are biased. The bias
cancels out out only when WR devices calibrated to the same calibrator are connected.
cancels out only when WR devices calibrated to the same calibrator are connected.
% Relative calibration is performed for a complete WR device (e.g. a given version of WR switch and SFPs)
% and needs to be repeated each time a composing elements changes.
An ongoing work on absolute calibration \cite{biblio:WR-calibration} will allow
An ongoing work on absolute calibration \cite{biblio:WR-calibration} allows
to measure precisely actual value of hardware delays and their different contributors.
With such calibration, "golden calibrator" will not be required and adding new type of component
(e.g. SFP) to a WR network will not necessitate time-consuming calibration of all
With such calibration, a "golden calibrator" will not be required and adding a new type of component
(e.g. SFP) to a WR network will not necessitate a time-consuming calibration of all
devices with this component.
\section{WR Standardization in IEEE1588 (PTP)}
......@@ -1188,16 +1186,16 @@ devices with this component.
The P1588 Working Group \cite{biblio:P1588} is revising the
IEEE1588 standard, due to finish by the end of 2019. This group has been studying the
protocol extensions implemented in WR in order to incorporate their generalized
protocol extensions implemented in WR in order to incorporate a generalized
version into the standard \cite{P1588-HA-enhancements}.
As a result, one of the additions to the standard
is a third Default PTP Profile: High Accuracy.
This profile mandates a number of IEEE1588's new optional features that
are functionally equivalent to WR-PTP and allow support of WR hardware.
Along with the new features, informative annexes are added with "standardized"
are functionally equivalent to WR-PTP and that allow the support of WR hardware.
Along with the new features, informative annexes are added with a "standardized"
description of the WR calibration procedures \cite{biblio:wrCalibration} and
an example implementation of the High Accuracy profile implementation that achieves
sub-ns synchronization, a.k.a White Rabbit. The maping between WR and High Accuracy
sub-ns synchronization, a.k.a White Rabbit. The mapping between WR and High Accuracy
is described in \cite{biblio:WRin1588}.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment