Commit 7cfb6027 authored by Maciej Lipinski's avatar Maciej Lipinski

WIP: restructured according to Javiers advise

parent c7815469
......@@ -52,6 +52,17 @@
\begin{abstract}
%\boldmath
White Rabbit (WR) extends the Precision Time Protocol (PTP)
\\
\\
\\
\\
\\
\\
\\
\\
\\
\end{abstract}
\section{Introduction}
......@@ -86,8 +97,8 @@ that both of the above enhancements can be further extended to operate in highly
manner ensuring maximum a single failure per year for a network of 2000 WR nodes.
\begin{figure}[!ht]
\centering
% \vspace{0.1cm}
\includegraphics[width=0.4\textwidth]{network/wr_network-enhanced_pro.pdf}
% \vspace{0.2cm}
\includegraphics[width=0.3\textwidth]{network/wr_network-enhanced_pro.pdf}
\caption{White Rabbit Network.}
\label{fig:WRN}
\end{figure}
......@@ -107,25 +118,43 @@ commercially available, used all over the world, and being incorporated into
the original PTP standard \cite{biblio:P1588WG}\cite{P1588-HA-enhancements}.
\section{WR Network Elements}
WR Network consists of WR Switches and WR Nodes.
\\
\\
\\
\\
\\
\\
\\
\begin{figure}[!ht]
\centering
\vspace{0.1cm}
\includegraphics[width=0.4\textwidth]{misc/zoo-v2.jpg}
\caption{White Rabbit Network.}
\label{fig:WRN}
\end{figure}
\section{Types of White Rabbit Applications}
This section describes different ways in which WR is used in end-
applications. Such applications are described in the subsequent sections and
include:
accelerators, synchrotrons and spallation sources (Section~\ref{sec:accelerators}),
detectors of cosmic rays and neutrinos (Section~\ref{sec:detectors}),
extreme laser experiments (Section~\ref{sec:lasers}),
national metrology institutes (Section~\ref{sec:timelabs}), and
power industry (Section~\ref{sec:power}.
% \section{Types of White Rabbit Applications}
%
% This section describes different ways in which WR is used in end-
% applications. Such applications are described in the subsequent sections and
% include:
% accelerators, synchrotrons and spallation sources (Section~\ref{sec:accelerators}),
% detectors of cosmic rays and neutrinos (Section~\ref{sec:detectors}),
% extreme laser experiments (Section~\ref{sec:lasers}),
% national metrology institutes (Section~\ref{sec:timelabs}), and
% power industry (Section~\ref{sec:power}.
\subsection{Time and Frequency Transfer}
\newpage
\section{Time and Frequency Transfer (TF)}
\label{sec:time-and-freq}
\subsection{Basic Concept}
The most basic application of WR is a transfer of time and/or frequency. The time
is provided as an output Pulse Per Second (PPS) signal and an information about the
number of seconds since the epoch. The frequency is provided as an output clock
......@@ -144,8 +173,28 @@ frequency. Yet, in practice mostly national time laboratories
from functionalities that are built using the transferred time and/or frequency.
Such functionalites are described in the following subsections.
\subsection{Time-Triggered Control (TC)}
\subsection{Example Applications}
Time and Frequency distribution for AD\\
Finland (MIKES)\\
Netherlands (VSL)\\
France (LNE-SYRTE)\\
UK (NLP)\\
Italy (INFRIM)\\
White Rabbit Industrial Timing Enhancement (WRITE))\\
\\
\\
\\
\\
\\
\\
\newpage
\section{Time-Triggered Control (TC)}
\label{sec:time-triggered-ctrl}
\subsection{Basic Concept}
Many accelerators, synchrotrons and spallation sources are controlled by triggering
events in a pre-configured sequence of actions. In fact, it is a very convenient
way to control beams of particle that move at very large speeds, often close
......@@ -158,9 +207,16 @@ minimum advance with which an event can be scheduled. By having precise time
and frequency, events can be scheduled with sub-ns accuracy and picoseconds
precision.
\subsection{Example Applications}
GSI Helmholtz Centre for Heavy Ion Research \\
China Spoliation Neutron Source (CSNS)\\
General Machine Timing (GMT) and Beam Synchronous Timing (BST)\\
\subsection{Precise Timestamping (TS)}
\newpage
\section{Precise Timestamping (TS)}
\label{sec:timestamping}
\subsection{Basic Concept}
In a great numbers of applications, time and frequency are transferred in order
to timestamp accurately and/or precisely incoming signals. Such incoming signals
can be either discrete pulses timestamped with time-to-digital converters
......@@ -176,9 +232,32 @@ experiments. Being able to collerate signals in accelerators allows to recreate
the sequence of events when something goes wrong. Correlating samples in detectors
allows to coherently recreate experimental data.
\subsection{Example Applications}
The first operational application of WR was in the second run of the
CERN Neutrinos to Gran Sasso (CNGS) experiment \cite{biblio:wr-cngs}. Two WR
networks were installed in parallel with the initial timing system, one WR
network at CERN and one in Gran Sasso. Each WR network consisted of a Grandmaster
WR Switch connected to the time reference (Septentrio PolaRx4TR
\cite{biblio:PolaRx4e} and Symmetricom Cs4000 \cite{biblio:CS4000}), a WR
switch in the underground cavern and a number of SPEC boards with FMC-DEL that
performed timestamping of inputs signal. The measured performance of the deployed
system over 1 month of operation was 0.517 ns accuracy and 0.119 ns precision with
MTIE below 1.05 ns and only 0.0003\% of values exceeding the ±0.5 ns range.
\subsection{Triggers Distribution (TD)}
Large High Altitude Air Shower Observatory (LHAASO) \\
Hundred Square km Cosmic ORigin Explorer (HiSCORE)\\
Square Kilometre Array (SKA)\\
Cherenkov Telescope Array (CTA)\\
Joint Institute for Nuclear Research (JINR)\\
ELI Attosecond Light Pulse Source (ELI-ALPS) \& ELI Beamline Facility (ELI-BEAMS) \\
The German Aerospace Center (DLR)\\
Power Industry and Smart Grid\\
\newpage
\section{Triggers Distribution (TD)}
\label{sec:triggers-distribution}
\subsection{Basic Concept}
Triggers distribution combians time-triggered control and precise timestamping
described above. In this application, an input trigger is timestamped, sent over
WR network to many WR nodes that act upon the received message simultaneously, at
......@@ -192,8 +271,15 @@ the trigger simultaneously in the WR nodes, the worst case latency with some
margin of error is applied. Such trigger distribution has been used to diagnose
LHC since \cite{biblio:WR-LIST} since 2013, see Section~\ref{sec:CERN-LIST}.
\subsection{Fixed-Latency Data Transfer (FL)}
\subsection{Example Applications}
Trigger Distribution for LIST and OASIS
\newpage
\section{Fixed-Latency Data Transfer (FL)}
\label{sec:fixed-latency}
\subsection{Basic Concept}
Fixed-latency data transfer provides a well-know and precise latency of data
transfer between WR nodes in the WR network. It uses very similar
principles to the trigger distribution in Section~\ref{sec:triggers-distribution}.
......@@ -210,7 +296,45 @@ The "WR Streamers" are used at CERN in the WR-BTrain system
(Section~\ref{sec:CERN-wr-btrain}) that distributes the value of magnetic field
in real-time.
\subsection{Radio-Frequency Transfer (RF)}
\subsection{Example Applications}
In circular accelerators, the acceleration of beam by radio-frequency (RF) cavities
needs to be synchronized with the increase of magnetic field (B-field) in the
bending magnets. At CERN, BTrain is a system that measures and distributes the
value of the magnetic field (B-value) in real-time to the RF cavities,
power converters and beam instrumentation, as depicted in Figure~\ref{fig:WR-BTrain}.
While the RF cavities
simply follow the ramp of magnetic field, the power converters adjust the current
of the magnets such that the intended B-value is obtained and closing a control
loop. BTrain is essential to the operation of most of CERN accelerators, i.e.
Booster, PS, SPS, LEIR, AD, and ELENA.
% \begin{figure}[!ht]
% \centering
% \vspace{0.5cm}
% \includegraphics[width=0.5\textwidth]{applications/CERN/BTrainUpgrade.jpg}
% \caption{BTrain over White Rabbit.}
% \label{fig:WR-BTrain}
% \end{figure}
The original BTrain system uses coaxial cables to distribute pulses which indicate
increase or decrease of magnetic filed. This old-BTrain is now being upgraded to
use WR network \cite{biblio:WR-Btrain-MM} for distribution of the absolute value of the
magnetic field (i.e. B-value) and other additional information. This upgraded system
is called WR-BTrain \cite{biblio:WR-Btrain}. In WR-BTrain, the B-values are transmitted
at 250kHz
(every $4\mu s$) from the measurement WR node to all the other nodes. In the most
demanding accelerator, SPS, the data must be delivered over 2 hops (WR switches)
with latency of $10\mu s\pm 8ns$.
The WR-BTtrain has been succesfully evaluated in the PS accelerators where it has
been running operationally since 2017. It is now being installed in the remaining
accelerators. By 2021, all the accelerators should be running WR-BTrain operationally
\cite{biblio:WR-Btrain-status}.
For each WR-BTrain, a separated WR network is installed that consists of 1-2
WR switches and 2-5 WR nodes.
\newpage
\section{Radio-Frequency Transfer (RF)}
\subsection{Basic Concept}
Radio-frequency (RF) transfer allows to digitize periodic signals in a WR node, send
their digital form over WR network to other WR nodes, and then regenerate them
coherently in many WR nodes with a fixed delay. Such schema is depicted in
......@@ -234,227 +358,398 @@ accuracy and picoseconds precision and that are delayed with respect to the RF
input. This schema is used currently tested in ESRF (Sectoin~\ref{sec:ESFR-GMT})
and at CERN (Section~\ref{sec:CERN-RF})
\section{Applications in Accelerators, Synchrotrons and Spallation Sources}
\label{sec:accelerators}
\subsection{Example Applications}
\subsection{European Organization for Nuclear Research (CERN)}
\label{sec:CERN}
While it was initiated within the framework of renovating CERN's General
Machine Timing \cite{biblio:GMTJavierPres}, WR found many other applications
and it is one of recommended fieldbuses at CERN.
\subsubsection{CERN Neutrinos to Gran Sasso (CNGS)}
\label{sec:CERN}
The first operational application of WR was in the second run of the
CERN Neutrinos to Gran Sasso (CNGS) experiment \cite{biblio:wr-cngs}. Two WR
networks were installed in parallel with the initial timing system, one WR
network at CERN and one in Gran Sasso. Each WR network consisted of a Grandmaster
WR Switch connected to the time reference (Septentrio PolaRx4TR
\cite{biblio:PolaRx4e} and Symmetricom Cs4000 \cite{biblio:CS4000}), a WR
switch in the underground cavern and a number of SPEC boards with FMC-DEL that
performed timestamping of inputs signal. The measured performance of the deployed
system over 1 month of operation was 0.517 ns accuracy and 0.119 ns precision with
MTIE below 1.05 ns and only 0.0003\% of values exceeding the ±0.5 ns range.
European Synchrotron Radiation Facility (ESRF)
\subsubsection{BTrain over White Rabbit (WR-BTrain)}
\label{sec:CERN-wr-btrain}
In circular accelerators, the acceleration of beam by radio-frequency (RF) cavities
needs to be synchronized with the increase of magnetic field (B-field) in the
bending magnets. At CERN, BTrain is a system that measures and distributes the
value of the magnetic field (B-value) in real-time to the RF cavities,
power converters and beam instrumentation, as depicted in Figure~\ref{fig:WR-BTrain}.
While the RF cavities
simply follow the ramp of magnetic field, the power converters adjust the current
of the magnets such that the intended B-value is obtained and closing a control
loop. BTrain is essential to the operation of most of CERN accelerators, i.e.
Booster, PS, SPS, LEIR, AD, and ELENA.
\begin{figure*}[!ht]
\centering
\vspace{0.5cm}
\includegraphics[width=0.9\textwidth]{applications/CERN/BTrainUpgrade.jpg}
\caption{BTrain over White Rabbit.}
\label{fig:WR-BTrain}
\end{figure*}
% \section{WR Applications outside CERN}
% \label{sec:GSI-GMT}
The original BTrain system uses coaxial cables to distribute pulses which indicate
increase or decrease of magnetic filed. This old-BTrain is now being upgraded to
use WR network \cite{biblio:WR-Btrain-MM} for distribution of the absolute value of the
magnetic field (i.e. B-value) and other additional information. This upgraded system
is called WR-BTrain \cite{biblio:WR-Btrain}. In WR-BTrain, the B-values are transmitted
at 250kHz
(every $4\mu s$) from the measurement WR node to all the other nodes. In the most
demanding accelerator, SPS, the data must be delivered over 2 hops (WR switches)
with latency of $10\mu s\pm 8ns$.
The WR-BTtrain has been succesfully evaluated in the PS accelerators where it has
been running operationally since 2017. It is now being installed in the remaining
accelerators. By 2021, all the accelerators should be running WR-BTrain operationally
\cite{biblio:WR-Btrain-status}.
For each WR-BTrain, a separated WR network is installed that consists of 1-2
WR switches and 2-5 WR nodes.
% \subsection{Accelerators, Synchrotrons and Spallation Sources}
% \label{sec:GSI-GMT}
% White Rabbit (WR) is used as the basis for timing system at GSI Helmholtz Centre
% for Heavy Ion Research (GSI)\cite{biblio:GSI}. GSI is a heavy ion
% laboratory that performs basic
% and applied research in physics and related science disciplines. It is located in
% Darmstadt, Germany. GSI’s complex of accelerators is being extended with a new
% Facility for Antiproton and Ion Research (FAIR) \cite{biblio:FAIRtimingSystem}.
% FAIR will be one of the
% largest and most complex accelerator facilities in the world with its biggest
% accelerator of 1100m circumference. The operation of the FAIR facility requires
% time-triggered actions in different sub-systems of its accelerators. This is the
% responsibility of a timing system called General Machine Timing (GMT). Operation
% of FAIR requires that a central controller triggers within 500us an action in any of
% the 2000-3000 sub-systems. While the vast majority requires only about 1us accuracy,
% many subsystems like RF, beam instrumentation and experiments require 1-5 ns accuracy
% and 10 ps precision. This is achieved using WR network \cite{biblio:WR-GSI}\cite{biblio:MathiasPhD}
% that consists of 300
% WR switches in 5 layers and 2000 WR nodes integrated with accelerator sub-system.
% The WR nodes and WR switches are connected with fibers of up to 2km length. The
% required reaction time of the system is 1ms which translates into an upper-bound network
% latency of 500us from the central controller to any node. The WR-based GMT timing
% system already replaced the old timing system at the existing GSI facility and has
% started operation in 2017. The first major beam time using the new timing system
% starts in June 2018 \cite{biblio:GSI-schedule}.
% \subsection{Joint Institute for Nuclear Research (JINR)}
% \label{sec:JINR-GMT}
\subsubsection{Trigger Distribution for LIST and OASIS}
\label{sec:CERN-OASIS}
% The Joint Institute for Nuclear Research (JINR) [1] is an international research center for nuclear sciences in Dubna, Russia. It operates a collection of experimental physics facilities, including superconducting accelerator of nuclei and heavy ions – Nuclotron and proton accelerator – Phasotron. These facilities are now upgraded into a Nuclotron-based Ion Collider Facility (NICE) [2] to study properties of dense baryonic matter. Two NICE’s experiments, the Baryonic Matter at the Nuclotron (BM@N) and the MultiPurpose Detector (MPD) at the NICE Collider use White Rabbit as the main clock distribution network. Other NICE sub-systems consider WR for timing purposes. The WR network at BM@N was operational in 2015. It consists of 6 WR Switches connecting 12 detector subsystems (nodes) [4]. At BM@N, White Rabbit is used for timestamping of the detected signals and triggering of acquisition.
%
\subsubsection{RF distribution at SPS}
\label{sec:CERN-RF}
% \subsection{European Synchrotron Radiation Facility (ESRF)}
% \label{sec:ESFR-GMT}
\subsubsection{General Machine Timing (GMT) and Beam Synchronous Timing (BST)}
\label{sec:CERN-GMT-BST}
% The European Synchrotron Radiation Facility (ESRF) [1] is a joint research facility situated in Grenoble, France, and supported by 22 countries. Research at the ESRF focuses, in large part, on the use of X-ray radiation in fields as diverse as protein crystallography, earth science, paleontology, materials science, chemistry and physics. The ESRF accelerator system consists of a “gun” that generates electron bunches, a Booster that accelerates these bunches, and a Storage Ring that stores the bunches serving experimental beamlines. The operation of the ESRF accelerator facility is controlled by a “Bunch Clock” system that delivers to accelerator sub-systems ~352 MHz Radio Frequency (RF) signal and triggers initiating sequential actions synchronous to the RF signal, such as “gun trigger”, injection trigger”, “extraction trigger”. The jitter of the RF signal is required to be below 50ps. The RF is continuously trimmed around the 352 MHz value as the tuning parameter in the “fast orbit feedback” process. The current “Bunch Clock” system becomes obsolete while being inflexible and enable to meet new needs of the evolving accelerator complex and experiments. Thus, it has been decided to refurbish the current system, basing the new “Bunch Clock” system on White Rabbit (WR) [2].The new “Bunch Clock” uses the WR-wide common notion of time and frequency. Using the WR time and frequency, it digitizes the input 352 MHz RF signal, sends its digital version over WR Network, and synthesizes a delayed and in-phase version of the input RF signal in the WR Node. The same WR Network is also used to distribute information about the triggers from the “master module”. The WR Nodes can allow for precise timestamping which is required for diagnostics and by experimental beamlines using the accelerator facility. The WR-based new “Bunch Clock” system was validated by transmitting a real-life RF signal over 6 months without failure and within specification (what sepcs). Phase noise measurements of the RF signal transmitted over WR shows <10ps jitter. As part of the validation, the new system has been successfully used to inject bunches in the storage ring. It is to partially replace parts of the old system by the end of 2018. It will fully replace the old system by 2020.
\subsection{GSI Helmholtz Centre for Heavy Ion Research (GSI)}
\label{sec:GSI-GMT}
% \subsection{China Spoliation Neutron Source (CSNS)}
% \label{sec:CSNS-GMT}
White Rabbit (WR) is used as the basis for timing system at GSI Helmholtz Centre
for Heavy Ion Research (GSI)\cite{biblio:GSI}. GSI is a heavy ion
laboratory that performs basic
and applied research in physics and related science disciplines. It is located in
Darmstadt, Germany. GSI’s complex of accelerators is being extended with a new
Facility for Antiproton and Ion Research (FAIR) \cite{biblio:FAIRtimingSystem}.
FAIR will be one of the
largest and most complex accelerator facilities in the world with its biggest
accelerator of 1100m circumference. The operation of the FAIR facility requires
time-triggered actions in different sub-systems of its accelerators. This is the
responsibility of a timing system called General Machine Timing (GMT). Operation
of FAIR requires that a central controller triggers within 500us an action in any of
the 2000-3000 sub-systems. While the vast majority requires only about 1us accuracy,
many subsystems like RF, beam instrumentation and experiments require 1-5 ns accuracy
and 10 ps precision. This is achieved using WR network \cite{biblio:WR-GSI}\cite{biblio:MathiasPhD}
that consists of 300
WR switches in 5 layers and 2000 WR nodes integrated with accelerator sub-system.
The WR nodes and WR switches are connected with fibers of up to 2km length. The
required reaction time of the system is 1ms which translates into an upper-bound network
latency of 500us from the central controller to any node. The WR-based GMT timing
system already replaced the old timing system at the existing GSI facility and has
started operation in 2017. The first major beam time using the new timing system
starts in June 2018 \cite{biblio:GSI-schedule}.
% The China Spallation Neutron Source (CSNS) [1] is an accelerator-based pulsed spallation neutron source located in south China, forth of the kind in the world. A "super microscope" for looking into the microstructure of materials, the CSNS has a wide range of application prospects, including in life sciences, physics, chemistry, resources and the environment, and new energy. Completed in March 2018 [2], the facility includes a powerful linear proton accelerator, a rapid circling synchrotron, a target station and three neutron instruments: the General-Purpose Powder Diffractometer (GPPD), Small-Angle Neutron Scattering instrument (SANS), and multi-purpose reflectometer (MR).
% The instrument control system of CSNS is based on White Rabbit network that provides synchronization and real-time control [4]. The experimental control system of CSNS is in charge of target and instrument control. In CSNS, the precise time (T0) of proton hitting the target needs to be measured. The measured time is broadcast to the target stations and the neutron instruments so that these equipment can work relative to T0. This time is also needed to measure the neutron time of flight. The required precision of T0 is 10ns while its transmission to all neutrino instruments must be below 5us [3]. While WR has proven to provide sub-ns accuracy of synchronization, thus meet the 10ns requirement, the tests in [3] confirmed WR’s suitability for the real-time controls (delay below 5us with jitter below <500ns). WR network is used also to synchronize “standard” IEEE1588 devices (NI, BeckHoff PLCs) [4]. The WR Network is synchronized to GPS receiver and rubidium clock. It is composed of 3 WR Switch in 2 layers and 7 WR Nodes
%
\subsection{Joint Institute for Nuclear Research (JINR)}
\label{sec:JINR-GMT}
% \subsection{Neutrinos Detectors}
% \label{sec:detectors}
%
\subsection{European Synchrotron Radiation Facility (ESRF)}
\label{sec:ESFR-GMT}
% \subsection{Cubic Kilometre Neutrino Telescope (KM3NeT)}
% \label{sec:}
% The Cubic Kilometre Neutrino Telescope (KM3NeT) [1] is a research infrastructure housing the next generation neutrino telescopes located at the bottom of the Mediterranean Sea. Once completed, the telescopes will have detector volumes between megaton and several cubic kilometres of clear sea water. Located in the deepest seas of the Mediterranean, KM3NeT will aim at the discovery and subsequent study of high-energy neutrino sources in the Universe and the determination of the mass hierarchy of neutrinos. When completed, the KM3NeT infrastructure will consist of 3 detectors located off shore the cost of France, Italy and Greece. Each detector will consist of the same building blocks: pressure-resistant "digital optical modules" (DOMs) detectors attached to strings. Holding 18 detectors, each string will be anchored to the sea floor and supported by floats. Equally spaced strings will detect Cherekov light generated by the charged particles from neutrino induced interactions inside or close to the telescope. By correlating detection time from different DOMs, the properties and trajectory, thus source, of neutrinos will be studied. The required angular resolution of the measurement is 0.1 degree which translates into synchronization of all the DOMs with 1 ns accuracy and a few 100 ps of jitter. Such synchronization must be performed between the on-shore reference and all the DOMs submerged in the see. The ARCA [2] installation is located at a depth of 3500, 100km off-shore of Italy. The ORCA [3] installation is located at a depth of 2475 m, 40 km off-shore France. While currently one string of DOMs (18) is installed at each site, once completed [4], ARCA will consist of 4140 DOMs with a volume of 1 Gton of seawater (1km3) and ORCA will consist of 2070 DOMs spanning with a volume of about 3.7 Mton. White Rabbit [5] is used in KM3NeT as synchronization system. It synchronizes DOMs (WR Nodes) with the on-shore clock reference to allow timestamping the arrival of photons using FPGA-based Time-to-Digital converters with 1ns granularity. Additionally, WR is used to synchronize a “start of run” at a predefined UTC/TAI time. WR has been successfully operating with the already installed DOMs. In the final installation of ARCA and ORCA to be completed around 2020, around few hundred WR Switches will be deployed to synchronize over 6000 DOMs. It is yet to be decided whether some of the switches will be submerged or all of the will be located on-shore.
\subsection{China Spoliation Neutron Source (CSNS)}
\label{sec:CSNS-GMT}
\section{Detectors of Cosmic Rays and Neutrinos}
\label{sec:detectors}
\subsection{CERN Neutrinos to Gran Sasso (CNGS)}
\label{sec:}
% \subsection{Cherenkov Detectors in Mine PitS (CHIPS)}
% \label{sec:}
\subsection{Cubic Kilometre Neutrino Telescope (KM3NeT)}
\label{sec:}
\subsection{Cherenkov Detectors in Mine PitS (CHIPS)}
\label{sec:}
\subsection{Deep Underground Neutrino Experiment (DUNE)}
\label{sec:}
% \subsection{Deep Underground Neutrino Experiment (DUNE)}
% \label{sec:}
\subsection{Short Baseline Neutrino Physics program (SBN)}
\label{sec:}
\subsection{Large High Altitude Air Shower Observatory (LHAASO) }
\label{sec:}
% In the framework of the Deep Underground Neutrino Experiment (DUNE) [1] to be developed over two decades, a number of Neutrino detectors are installed in Fermilab and at CERN. These detectors contribute to evaluating technologies and procedures to be used in DUNE. The DUNE will consist of two detectors. One detector will record particle interactions near the source of the beam, at the Fermi National Accelerator Laboratory in Batavia, Illinois. A second detector, the largest of its type ever build, will be installed more than a kilometer underground at the Sanford Underground Research Laboratory in Lead, South Dakota — 1,300 kilometers downstream of the source. The Conceptual Design Report [2] proposes White Rabbit as a timing system. As such, WR is being evaluated in the currently developed and deployed neutrino detectors that are part of the DUNE initiative. These detectors are built in Fermilab and at CERN. While the accuracy required currently by the neutrino experiments is at a sub-microsecond level (~400ns accuracy, page 36 of [4]), the goal is to provide a nanosecond synchronization. The ICARUS detector, a part of the Short-Baseline Neutrino (SBN) Program, is the first to use White Rabbit in North America [3]. In this detector, White Rabbit is to be used to time-tag the neutrinos from their production at the beam source through to the detector at the end of the experiment. On the SBN ICARUS detector, White Rabbit is also expected get an extremely accurate tagging of unwanted cosmic particles that come from space and get in the way of the experiment, potentially hiding the neutrino signatures. WR is also to be used in GLACIER detector (page 62 of [4], also 400ns accuracy/alignment, page 64 of [4]). WR is used in GLACIER for timestamping and gating measurements. WA105 detector is installed at CERN [5]. The currently installed WR Network has XXX WR Nodes and YYY WR Switches in ZZZ Layers. The size of the final installation in 20XX is planned to be
\subsection{Hundred Square km Cosmic ORigin Explorer (HiSCORE)}
\label{sec:}
% \subsection{Short Baseline Neutrino Physics program (SBN)}
% \label{sec:}
%
% \subsection{Detectors of Cosmic Rays and Neutrinos}
% \label{sec:detectors}
\subsection{The German Aerospace Center (DLR)}
\label{sec:}
\subsection{Cherenkov Telescope Array (CTA) }
\label{sec:}
% \subsection{Large High Altitude Air Shower Observatory (LHAASO) }
% \label{sec:}
\subsection{Square Kilometre Array (SKA)}
\label{sec:}
% Large High Altitude Air Shower Observatory (LHAASO) [1] will be one of the world’s largest and most sensitive cosmic-ray facilities, once competed in 2020. Located at about 4410 m above sea level in the Haizi Mountain in Sichuan Province in southwest China, the observatory will attempt to understand the origins of high-energy cosmic rays. Cosmic rays are particles that originate in outer space and are accelerated to energies higher than those that can be achieved in even the largest man-made particle accelerators. These high-energy protons and atomic nuclei create air showers of particles such as protons and muons. Where cosmic rays come from has remained a mystery since they were first spotted some 100 years ago. LHAASO aims to detect cosmic rays using a Cherenkov water detector, covering a total area of 80 000 m2. The facility will also consist of a 1.3 km2 array of 6000 scintillation detectors that will study electrons and photons in the air showers, while an overlapping 1.3 km2 underground array of 1200 underground Cherenkov water tanks will detect muons. Aiming to high sensitivity and wide spectrum of cosmic ray detection, the 1 km2 complex array consists of over 7000 detectors of different types. To reconstruct the air shower events with high angular resolution (0.5 degree), timestamps of all detector electronics and digitizers should be aligned better than 500 ps RMS while the synchronization of Analog-to-Digital converters requires < 100ps skew [2]. Such synchronization must be maintained in harsh and remote environment with large and rapid temperature variation in the range -10 to +55 Celsius degree. The synchronization of around 7000 detectors will be performed using White Rabbit Network consisting of 583 WR Switches in 4 layers connecting WR Nodes embedded in each detector.
% To meet LHAASO’s stringent synchronization performance requirements in the harsh environmental conditions, two dedicated developments were performed. First, the dependency on temperature of the WR Link Model [3] parameters was studied [4]. These parameters, the fixed delays and alpha, are assumed to be constant in normal operation of WR while the degradation of synchronization performance due to their variation with temperature can be up to 700 ps under ambient temperature between 10 and 55◦C. The studies [4] showed the parameters’ linear dependency on temperature which allows easy correction. An online real-time correction method was applied based on the result which reduced the synchronization variation below 150ps with standard deviation below 50 ps under a varying temperature between -10 and 50 Celsius degree. Second, to guarantee the overall synchronization precision, individual calibration of each WR node is essential to compensate device deviation. To facilitate calibration of over 7000 WR nodes, portable calibration node (PCN) was developed that allows auto-calibration [5]. It has been shown that the autocalibration procedure can achieve calibration of 300-ps accuracy.
% A small prototype based on the White Rabbit (WR) network was built at Yangbajin, Tibet, China in 2014 [6]. This prototype contains 4 WR switches and 50 WR customized nodes (48 electron detectors and 2 muon detectors). The four WR switches are used to build a four layer hierarchy WR network. All the WR nodes were calibrated using PCN and perform online temperature compensation. The prototype was operational for over a year performing detection of air showers and providing performance benchmark. It showed that the long term synchronization of the White-Rabbit network is promising and 500 ps overall synchronization precision is achievable with individual calibration and temperature correction. With this promising results, the construction of LHAASO detectors startd in 2016 and is due to finish in 2020. 1/4 of LHAASO facility is to be operation and produce physical data in 2018.
\section{Extereme Laser Experiments}
\label{sec:lasers}
\subsection{ELI Attosecond Light Pulse Source (ELI-ALPS) }
\label{sec:}
% \subsection{Hundred Square km Cosmic ORigin Explorer (HiSCORE)}
% \label{sec:}
\subsection{ELI Beamline Facility (ELI-BEAMS) }
\label{sec:}
\section{National Metrology Institutes}
\label{sec:timelabs}
% The Tunka experiment, called “Tunka Advanced Instrument for cosmic ray physics and Gamma Astronomy” [1], measures air showers, which are initiated by charged cosmic rays or high energy gamma rays. It is situated in Siberia in the Tunka valley close to lake Baikal. One of the five detectors installed in Tunka is the Hundred Square km Cosmic ORigin Explorer, HiSCORE [2]. HiSCORE is a non-imaging atmospheric Cherenkov light-front sampling array (20 TeV to few PeV), build of many optical detector stations, located at typical distances of 100-200 m. The detector is under construction, it will cover initially an area of 1km2, and up to 100km2 in the final phase. The HiSCORE aims at achieving a good reconstruction of the air shower arrival direction. For a pointing resolution of ∼ 0.1 ◦, all array stations need to be synchronized with sub-ns time precision. Synchronization of HiSCORE stations uses White Rabbit Network to perform time stamping and WR-triggering.
% WR was evaluated through field tests in Tunka between 2012 and 2015 [4]. The first HiSCORE-3 prototype array (winter season 2012/13) with three stations was composed of 1 WR Switch and 6 WR Nodes. It allowed methodical tests, study of fiber-delay compensation in varying temperature and gaining confidence in WR. The 9-station array HiSCORE-9 (winter season 2013/14) arranged on a regular grid of 3 x 3 stations with 150 m distance and an instrumented area of 0.09 km2 was the first astroparticle physics setup composed of 1 WR Switch and 9 WR Nodes. It focused on an end-to-end functional test of the WR time-synchronization. HiSCORE-9 was upgraded in 2014 to HiSCORE-28 with 28station at 100 m spacing forming a super-cell structure, on a total area of 450 m × 600 m (0.25km2). As described in [4], laboratory tests, long-term field tests and direct cross-verification with an alternative timing system, LED-calibration runs, and operation with Air Showers, confirmed < 0.5ns precision of WR and its suitability for application at Tunka. The first-phase HiSCORE detector of 1km2 is to be operational in 20xx with YY WR nodes and ZZZ WR Switches in NN layers. The 100km2 installation is foreseen in 20xx with YY WR nodes and ZZZ WR Switches in NN layers.
% An unexpected discovery was made, while routinely operating the HiSCORE array with the sub-nsec precision timing mode. The International Space Station (ISS) passes over the TAIGA HiSCORE detectors at a height of 400 km a few times per year and has a laser beam (CATS-LIDAR) directed at earth. The full HiSCORE array is illuminated by the ISS/CATS-LIDAR ultra-short laser pulses at the same moment. This laser beam turns out to be a unique calibration tool, and proves the stable WR-operation.
\subsection{Finland (MIKES)}
\label{sec:}
\subsection{Netherlands (VSL)}
\label{sec:}
% \subsection{Cherenkov Telescope Array (CTA) }
% \label{sec:}
\subsection{France (LNE-SYRTE)}
\label{sec:}
\subsection{UK (NLP)}
\label{sec:}
% The Cherenkov Telescope Array (CTA) [1] is one of the major future facilities in the field of astroparticle physics and high-energy astrophysics, exploring the high-energy universe with gamma rays (20 GeV to 300 TeV). The CTA observatory is planned to start full operation in the early 2020's. It will consist of more than 100 telescopes distributed over two sites, one on La Palma (Spain) and one in Paranal (Chile) and the telescopes on each site will be located up to several kilometers away from each other. The telescopes consist of tessellated mirrors which focus the few nanoseconds long Cherenkov light ashes from air showers initiated by cosmic gamma rays in the Earth’s atmosphere onto fast-recording, pixelated cameras. In order to properly combine the spatial and temporal information of these short light ashes from all the telescope cameras and accurately reconstruct the properties of the observed air shower, precise knowledge of the timing is mandatory for CTA. It is therefore required, that the relative timing precision between different cameras is better than two nanoseconds with less than one nanosecond jitter (RMS).
% In order to achieve such a high time precision, CTA will use a unified timing system [2] which is based on a hierarchical White Rabbit Ethernet network. A mock setup of a Timing System similar to the final CTA Timing System was built in the lab in Amsterdam [2]. It consisted of 3 WR Switches and 32 WR Nodes and allowed validation of system performance providing synchronization at -175 ± 12 picoseconds and therefore well within all of the CTA requirements. The final installation will consist of XXX WR nodes and YYY WR switches in ZZZ layers. It is to be completed in 202y.
\subsection{Italy (INFRIM)}
\label{sec:}
\subsection{White Rabbit Industrial Timing Enhancement (WRITE))}
\label{sec:}
% \subsection{Square Kilometre Array (SKA)}
% \label{sec:}
% The Extreme Light Infrastructure (ELI) [1] is the first civilian large-scale high-power laser research facility to be realized with trans-European cooperation and the worldwide scientific community. Hungary, the Czech Republic and Romania, with a coordinated management and research strategy, will simultaneously implement the project through the construction of the three laser facilities with the respective mission in the attosecond, beamline and photonuclear applications. The main objective of ELI Attosecond Light Pulse Source (ELI-ALPS) is the establishment of a unique attosecond facility which provides ultrashort light pulses between THz (1012 Hz) and X-ray (1018-1019 Hz) frequency range with high repetition rate for developers and end-users. Experimental projects demanding ultrahigh intensity light, like laser particle acceleration or laser generated x-ray radiation will be primarily developed at the Beamline Facility (ELI-BEAMS) in Prague, Czech Republic, while the photoinduced nuclear experiments will be performed at the research institute to be built in Magurele (ELI-NP), near Bucharest, Romania. Two of the infrastructures, ELI-ALPS and ELI-BEAMS, are two use White Rabbit [2][3].
% \subsection{Finland (MIKES)}
% \label{sec:}
% The Centre for Metrology and Accreditation (MIKES) [1] is the national metrology institute of Finland. One of its mandates is a nation-wide dissemination of the official time in Finland, UTC (MIKES). MIKES’s customers include both Finnish and international companies as well as the public sector. MIKES has been evaluating application of White Rabbit for dissemination of UTC and comparison of clocks over 1)long-distance fiber links. Since 2013, it has been operating 950 km WR link ever [2][3] – the longest ever – to study application of WR for long-haul time transfer. Since recently, it has been also operating a 50km WR link [5] to study mid-distance low-jitter and low-asymmetry time transfer.
% The 950km WR-PTP link [2][3] uses unidirectional paths in a dark channel of the Finnish University and Research Network (FUNET) between Espoo and Kajaani, Finland. The long-haul link uses identical SFPs on ITU-T DWDM channel \#60 (196.00 THz) in both Espoo and Kajaani. It consists of 11 fiber spans between 15 and 140 km in length with amplifiers and dispersion compensation spools. Independent verification of WR time transfer between the two ends of the long-haul link was performed using GPS PPP postprocessing to determine the WR fiber asymmetry. The same method was used to verify WR time transfer, while the WR asymmetry parameter was held constant. The results presented in 2016 [4] from 4-month operation of the 950km WR link show the time transfer error within ±2 ns over the entire interval. The stability of the time transfer difference between WR and GPS PPP is shown in Fig. 6 of [4] expressed as time deviation. A minimum of 20 ps is reached at an averaging time of 1000 s.
% The 50km WR-PTP link [5] is established between laboratories operating active hydrogen masers. It is used to provide the official time of Finland, UTC(MIKES), to the Metsähovi Observatory [6]. The link uses optical DWDM transceivers on adjacent ITU DWDM channels (optical carrier spacing 100 GHz). This limits the systematic error due to chromatic dispersion to $<$700 ps without any calibration. The link takes advantage of the recently developed Low Jitter Daughterboard (LJD) that enhances the performance of the WR Switch without any modifications to the WR-PTP protocol. The results show performance at the 1e-12 level (at 1s, with 0.5 Hz BW). It also demonstrates that bidirectional optical link using adjacent 100 GHz DWDM channels is an economical and simple solution for reducing systematic errors on long links below 1 ns.
%
% \subsection{Netherlands (VSL)}
% \label{sec:}
% \subsection{France (LNE-SYRTE)}
% \label{sec:}
% The LNE-SYRTE [1] assumes the role of national metrology institute. It SYRTE [2] is a leader in time and frequency metrology and is also specialized in Earth rotation, celestial reference frame and history of Astronomy.
% LNE-SYTRE performs experiments with White Rabbit for time and frequency dissemination over long distance optical fiber links. Experiments with two setups and lengths of fiber links were performed over years 2015-2017.
% First, WR time transfer over 25km fiber spools was studied [3]. In this study, synchronization performance of a connection composed of two unidirectional links using single wavelength of about 1541 nm was compared with a connection composed of a bidirectional link using one 25km fiber and two wavelengths 1310 and 1490 nm. For both connections, time deviation of about a 1-2 ps at 1000 seconds of integration time and peak-peak fluctuations of about 150 ps was reported. With 25-km fiber spools, the expected contribution of the chromatic dispersion to the instability of is about 5-7 ps @ 4 s. From the study, it is clear that chromatic dispersion plays an important and limiting role for long haul fiber links with SFPs, and that care must be taken on the frequency stability of the emitters for links longer than 100 km. This is confirmed with the second study.
% Second, WR transfer over WR Network spanning 500km and composed of 4 cascaded WR switches, a WR Node (WR-ZEN) board was studied [4]. This setup utilizes a 2 × 125 km uni-directional fiber links and long range SFPs in the C-band or OSC channels close to the C-Band. The performance of the WR Switches and the WR Node was enhanced. In particular, the input stage of the Grandmaster WR switch was improved, the bandwidth of the Soft PLLs in the remaining WR Switches and WR Node was increased to 70Hz, and the rate of PTP messages was increased to 15 per second. The study reported frequency transfer stability at the level of 2 × 10−15 over one day of integration time. Its results showed the time stability at one second of integration time to be 5.5 ps, and it reached a minimum of 1.2 ps at 20 seconds of integration time. The peak to peak fluctuations were about 2.5 ns. The limitations for the performance were confirmed to be due to chromatic dispersion (optical emitter stability) at short integration time and fiber thermal noise for long integration time.
% \subsection{UK (NLP)}
% \label{sec:}
% \subsection{Italy (INFRIM)}
% \label{sec:}
% \subsection{White Rabbit Industrial Timing Enhancement (WRITE))}
% \label{sec:}
\section{Power Industry and Smart Grid}
\label{sec:power}
%
% \subsection{Other applications}
% \label{sec:lasers}
%
% \subsection{ELI Attosecond Light Pulse Source (ELI-ALPS) }
% \label{sec:}
% The ELI-ALPS requires ns-to-ps accuracy for timestamping, ps accuracy for Radio Frequency (RF) systems and fs accuracy for pulsed optical synchronization [4]. White Rabbit is planned to provide a common timestamps base for the whole facility - except for the lasers, which need a femtosecond level synchronization. Moreover, triggers from the lasers are to be acquired and redistributed over White Rabbit towards the experimental areas. The currently installed WR network at ELI-ALPS consists of 3 WR Switches in XX layers and YYY WR Nodes. The final WR Network, to be operational in 20YY, is to consists of VV WR Switches in XX layers and YYY WR Nodes.
% \subsection{ELI Beamline Facility (ELI-BEAMS) }
% \label{sec:}
% In the ELI-BEAMS, White Rabbit is considered as the Electronic Timing System (ETS) [3] that is part of facility level control system. The EST provides synchronization and trigger signals to all subsystems and devices during operation, while each laser experiment is driven by its local timing system. ETS is required to provide picosecond level accuracy for typical laser experiments when driving streak cameras and low level of accuracy when pulse duration and pulse rise time are greater than 100ps (typically CCD cameras, Osciloscopes etc.). ELI-BEAMS evaluates White Rabbit as a potential EST with a test setup consisting of 2 WR Switches and two WR Nodes. If positively evaluated, the WR Network at ELI-BEAMS will consist of XXX WR Switches in YYY Layers and ZZZ WR nodes.
% \subsection{The German Aerospace Center (DLR)}
% \label{sec:}
% The German Aerospace Center (DLR) [1] is the national aeronautics and space research center of the Federal Republic of Germany. Its extensive research and development work in aeronautics, space, energy, transport, digitalization and security is integrated into national and international cooperative ventures. Within this scope, DLR performs optical reconnaissance [2] using Satellite Laser Ranging (LSR) [3]. Laser ranging to objects in space is of great importance in many different fields. By measuring the distance to dedicated satellites various scientific phenomena such as tectonic plate drifts, crustal deformation, the Earth’s gravity field or ocean tides can be investigated. Furthermore, precise orbit determination of cooperative as well as non-cooperative objects can support evasive manoeuvres to avoid collisions with space debris. Besides, reentry predictions and mission preparation can benefit from satellite laser ranging. Together with Very Long Baseline Interferometry (VLBI), LSR is one of the core technologies for the Global Geodetic Observing System (GGOS).
% The Satellite Laser Ranging is an established technology for geodesy, fundamental science and precise orbit determination. While many SLR stations are in operation, the SLR station that has been successfully operated by DLR since 206 is unique [3]. Apart from the fact that it uses optical fibre rather than a coude path, it has been assembled completely from commercial off-the-shelf components, which increases flexibility and significantly reduces hardware costs. To synchronize the measurement with UTC, a small White Rabbit network is used [4]. A WR switch coupled to a dedicated timing GPS module serves as grandmaster clock. W WR Node with FMC-DEL is connected to this WR switch. The FMC-DEL pulse generator card triggers the laser at precisely defined times and synchronizes the PicoHarp event timer to UTC. Thus, all time tags are synchronised to UTC to better than 100 ns, while time differences are measured with an accuracy of better than 1 ns.
% \section{Power Industry and Smart Grid}
% \label{sec:power}
% The Distributed Electrical Systems Laboratory (DESL) [1] at the Swiss Federal Institute of Technology Lausanne (EPFL), one of world’s best technical universities, operates its own experimental Smart Grid [2][5]. Within the framework of technology improvements for the Smart Grid, one of the research studies [3] evaluated White Rabbit for distributed measurement of synchrophasors via Phasor Measurement Units (PMUs). This study investigated application of WR as a way to improve the steady state PMU accuracy and mitigate the well-known disadvantages of the currently-used GPS-base synchronization of PMUs: accessibility of clear-sky view and vulnerability of GPS signals. The results of the research show that WR-PMU can provide slightly better performance than the GPS-PMU, and is an appropriate alternative for GPS-based PMUs. The performance of the WR-PMU was limited by PMU’s hardware and not WR performance[4]. In terms of security, neither GPS nor WR are secure. In the case of WR, an attack can be counteracted by using redundant and disjoint communication paths or using the GPS as a redundant time sources.
\begin{table*}[!t]
\caption{White Rabbit Applications}
\centering
\scriptsize
\tiny
\begin{tabular}
{| p{1.15cm} | c | c | c | p{1.3cm} | p{1.3cm} | p{1.2cm} | c | c | c | p{1.2cm} |} \hline
\textbf{System} &\textbf{Facility}&\textbf{Location}&\textbf{Type}&\textbf{Reqirements}&\textbf{Status} &\textbf{Performance} &\textbf{Link lenght} & \textbf{ $<$ Sept 2018} & ~2020 &\textbf{Remarks} \\
\textbf{Name} & & & & & & &(Total distance)& \textbf{WRN/WRS/WRL} &\textbf{WRN/WRS/WRL} & \\
& & & & & & & & & & \\ \hline
CNGS & CERN & Switzerland & TS & ~1ns & Operated successfully & A:0.517ns, P:0.119ns & $<$10km (16km) & 10 / 4 / 2 & & Dismantled \\ \hline
WR-BTrain & CERN & Switzerland & FL & latency $10\mu s\pm 8ns$ & Operational (PS,ELENA) & & $<$1km & 6 / 2 / 1 & ~20 / 8 / 1 & 6 WR networks \\ \hline
Trigger distribution & CERN & Switzerland & TD & & & & & & & \\ \hline
RF & CERN & Switzerland & RF & & & & & & & \\ \hline
GMT & CERN & Switzerland & TC & & & & & & & \\ \hline
& & & & & & & & & & \\ \hline
GMT & GSI & Germany & TC & A: 1-5ns P:$<$10ps & First Beam in June 2018 & & $<$1km & & & \\ \hline
& & & & & & & & & & \\ \hline
& & & & & & & & & & \\ \hline
{| p{0.9cm} | p{1cm} | p{0.4cm} | p{1.7cm} | p{1.8cm} | p{1.1cm} | c | c | p{1.5cm} |} \hline
& & & & & & \multicolumn{2}{c |}{\textbf{ Network Size}} & \\
\textbf{Facility}&\textbf{Location}&\textbf{Type}&\textbf{Reqirements} &\textbf{Status} &\textbf{Link len.} & \textbf{May 2018} & \textbf{2020} &\textbf{Remarks} \\
& & & & &(Tot. distance) & N / S / L & N / S / L & \\
& & & & & & & & \\ \hline
% \multicolumn{9}{|l|}{} \\
\multicolumn{9}{|c|}{\textbf{Accelerators, synchrotrons and spallation sources}} \\
% \multicolumn{9}{|l|}{} \\ \hline
CERN & Switz. & TF & A: 100ns & Operational (AD) & $<$10km & 0 / 2 / 1 & 0 / 2 / 1 & \\ \hline
CERN & Switz. & FL & lat: $10\mu s\pm 8ns$ & Operational (PS,ELENA) & $<$1km & 6 / 2 / 1 & ~20 / 8 / 1 & 6 WR networks \\ \hline
CERN & Switz. & TD & & & & & & \\ \hline
CERN & Switz. & RF & & & & & & \\ \hline
CERN & Switz. & TC & & & & & & \\ \hline
GSI & Germany & TC & A: 1-5ns P:$<$10ps & First Beam in June 2018 & $<$1km & & & \\ \hline
JINR & Russia & TS & P:20ps~(rms) & Operational & $<$1km & 50 / 5 / 3 & & \\ \hline
JINR & Russia & TS,TD & P:50ps~(rms) lat:$<$5us & Under contraction & $<$1km & & 200 / 15 / - & \\ \hline
ESRF & France & RF,TS & P:$<$50ps jitter & Testing & $<$1km & 7 / 1 / 1 & 40 / 5 / 2 & Partial operation in 2018 \\\hline
CSNS & Chine & TF,TS & A:10ns & Operational & $<$1km & 50 / 4 / 2 & & \\ \hline
% \multicolumn{9}{|l|}{} \\
\multicolumn{9}{|c|}{\textbf{Neutrino Detectors}} \\ \hline
% \multicolumn{9}{|l|}{} \\ \hline
CERN & Switz. & TS & ~1ns & Operated successfully & $<$10km (16km) & 10 / 4 / 2 & & Dismantled \\ \hline
KM3Net & France & TF,TS & A:1ns, P:100ps & valided, construction & 40km & 18 / 1 / 1 & 4140 / ? / ? & \\ \hline
KM3Net & Spain & TF,TS & A:1ns, P:100ps & valided, construction & 100km & 18 / 1 / 1 & 2070 / ? / ? & \\ \hline
CHIPS & USA & & A:1ns, P:100ps & valided, construction & $<$1km & & 200 / 16 / ? & \\ \hline
DUNE & Switz/USA & TS,TD & sub-us \& sub-ns & Prototype in 08.2018 & $<$1km & 14 / 5 / 2 & 36 / 5 / 2 & \\ \hline
SBN & USA & TS,TD & $approx$ns & Testing & $<$1km & 6 / 1 / 1 & 6 / 1 / 1 & Operational in 2019 \\ \hline
% \multicolumn{9}{|l|}{} \\
\multicolumn{9}{|c|}{\textbf{Cosmic Ray Detectors}} \\ \hline
% \multicolumn{9}{|l|}{} \\ \hline
LHAASO & China & TF,TS & A:500ps (rms) & prototype operational & $<$1km & 40 / 4 / 4 & 6734 / 564 / 4 & 1/4 operational in 2018 \\ \hline
HiSCORE & Russia & TS & & & & & & \\ \hline
CTA & Spain/Chile & TF,TS & A:$<$2ns P:$<$1ns (rms) & valided, construction & few km & 32 / 3 / 2 & 220 / 10 / 2 & Two WR networks \\ \hline
SKA & Australia/Africa& TF & A: 2ns (1 sigma) & valided, construction & 80km (173km) & 2 / 1 / 1 & 233 / 15 / 3 & \\ \hline
% \multicolumn{9}{|l|}{} \\
\multicolumn{9}{|c|}{\textbf{National Time Laboratories}} \\ \hline
% \multicolumn{9}{|l|}{} \\ \hline
MIKES & Finland & TF & & Operational, expanding & up to 950km & 10 / few /2 & & \\ \hline
NE-SYRTE & France & TF & & Evaluation, tests & up to 125km (500km)& 4 / 2 / 4 & & \\ \hline
VLS & Nederland & TF & & Evaluation, tests & 137km & & & \\ \hline
NIST & USA & TF & & Operational & $<$10km & & & \\ \hline
NLP & UK & TF & & ? & & & & \\ \hline
INRIM & Italy & TF, TS & & Operational, expanding & up to 400km & & & \\ \hline
% \multicolumn{9}{|l|}{} \\
\multicolumn{9}{|c|}{\textbf{Other Applications}} \\ \hline
% \multicolumn{9}{|l|}{} \\ \hline
DLR & Germany & TS & & & $<$1km & & & \\ \hline
ELI-ALPS & Hungry & TS & ns to ps & &$<$1km & & & \\ \hline
ELI-BEAMS & Czech & TF,TS, TD,TC& & & $<$1km & 70 / 16 / 2 & & \\ \hline
EPFL & Switzerland & TS & & & $<$1km & 2 / 1 / 1 & & \\ \hline
\multicolumn{9}{|l|}{} \\
% \multicolumn{11}{|c|}{\textbf{ALL}} \\
% \multicolumn{11}{|l|}{} \\ \hline
\multicolumn{6}{|r|}{\textbf{Total number: }} & & & \\ \hline
% \multicolumn{9}{|l|}{} \\
\multicolumn{9}{|l|}{\textbf{Abbreviations used}} \\
\multicolumn{9}{|l|}{TF = time and frequency transfer, TC = time-triggered control, TS = timestamping, TD = trigger distribution, FL = Fixed-latency data transfer, RF = Radio=-frequency transfer} \\
\multicolumn{9}{|l|}{A: accuracy, P: precision, Link len. - it is the lenght of link between devices in the network, Tot. distance = the total distance that the networks spans } \\
\multicolumn{9}{|l|}{} \\ \hline
\end{tabular}
\label{tab:rawData}
\end{table*}
% \begin{table*}[!t]
% \caption{White Rabbit Applications}
% \centering
% \scriptsize
% \begin{tabular}
% {| p{1.5cm} | p{0.9cm} | p{1.3cm} | p{0.4cm} | p{1.5cm} | p{1.3cm} | p{1.2cm} | p{1.5cm} | c | c | p{1.5cm} |} \hline
% & & & & & & & & \multicolumn{2}{c |}{\textbf{ Network Size}} & \\
% \textbf{System} &\textbf{Facility}&\textbf{Location}&\textbf{Type}&\textbf{Reqirements} &\textbf{Status} &\textbf{Performance} &\textbf{Link len.} & \textbf{May 2018} & \textbf{2020} &\textbf{Remarks} \\
% \textbf{Name} & & & & & & &(Tot. distance) & N / S / L & N / S / L & \\
% & & & & & & & & & & \\ \hline
% \multicolumn{11}{|l|}{} \\
% \multicolumn{11}{|c|}{\textbf{Accelerators, synchrotrons and spallation sources}} \\
% \multicolumn{11}{|l|}{} \\ \hline
% Time \& Freq Dist. & CERN & Switz. & TF & A: 100ns & Operational (AD) & & $<$10km & 0 / 2 / 1 & 0 / 2 / 1 & \\ \hline
% WR-BTrain & CERN & Switz. & FL & lat: $10\mu s\pm 8ns$ & Operational (PS,ELENA) & & $<$1km & 6 / 2 / 1 & ~20 / 8 / 1 & 6 WR networks \\ \hline
% Trigger distribution & CERN & Switz. & TD & & & & & & & \\ \hline
% RF & CERN & Switz. & RF & & & & & & & \\ \hline
% GMT & CERN & Switz. & TC & & & & & & & \\ \hline
% GMT & GSI & Germany & TC & A: 1-5ns P:$<$10ps & First Beam in June 2018 & & $<$1km & & & \\ \hline
% Clock\&time in BM@N & JINR & Russia & TS & P:20ps~(rms) & Operational & & $<$1km & 50 / 5 / 3 & & \\ \hline
% Clock\&time in MDP & JINR & Russia & TS, TD & P:50ps~(rms) lat:$<$5us & Under contraction & & $<$1km & & 200 / 15 / - & \\ \hline
% % WR-BTrain & JINR & Russia & FL & & Considering & & $<$1km & & & \\ \hline
% Synchronous Timing & ESRF & France & RF, TS & P:$<$50ps jitter & Testing & & $<$1km & 7 / 1 / 1 & 40 / 5 / 2 & Partial operation in 2018 \\\hline
% Instrumentation Ctrl & CSNS & Chine & TF, TS & A:10ns & Operational & & $<$1km & 50 / 4 / 2 & & \\ \hline
%
% \multicolumn{11}{|l|}{} \\
% \multicolumn{11}{|c|}{\textbf{Neutrino Detectors}} \\
% \multicolumn{11}{|l|}{} \\ \hline
%
% CNGS & CERN & Switz. & TS & ~1ns & Operated successfully & A:0.517ns, P:0.119ns & $<$10km (16km) & 10 / 4 / 2 & & Dismantled \\ \hline
% Timing Sys. at ARCA & KM3Net & France & TF, TS & A:1ns, P:100ps & valided, construction & & 40km & 18 / 1 / 1 & 4140 / ? / ? & \\ \hline
% Timing Sys. at ORCA & KM3Net & Spain & TF, TS & A:1ns, P:100ps & valided, construction & & 100km & 18 / 1 / 1 & 2070 / ? / ? & \\ \hline
% Timing System & CHIPS & USA & & A:1ns, P:100ps & valided, construction & & $<$1km & & 200 / 16 / ? & \\ \hline
% Timing Systtem & (proto) DUNE & Switz/USA & TS, TD & sub-us \& sub-ns & Prototype in 08.2018 & timestamps at 900ps & $<$1km & 14 / 5 / 2 & 36 / 5 / 2 & \\ \hline
% Timing Systtem & SBN & USA & TS, TD & $approx$ns & Testing & timestamps at 900ps & $<$1km & 6 / 1 / 1 & 6 / 1 / 1 & Operational in 2019 \\ \hline
%
% \multicolumn{11}{|l|}{} \\
% \multicolumn{11}{|c|}{\textbf{Cosmic Ray Detectors}} \\
% \multicolumn{11}{|l|}{} \\ \hline
%
% Timing System & LHAASO & China & TF, TS & A:500ps (rms) & prototype operational & & $<$1km & 40 / 4 / 4 & 6734 / 564 / 4 & 1/4 operational in 2018 \\ \hline
% Timing System & HiSCORE & Russia & TS & & & & & & & \\ \hline
% Timing System & CTA & Spain/Chile & TF, TS & A:$<$2ns P:$<$1ns (rms) & valided, construction & 175ps$\pm12$ps & few km & 32 / 3 / 2 & 220 / 10 / 2 & Two WR networks \\ \hline
% Timing System & SKA & Australia/
% South Africa & TF & A: 2ns (1 sigma) & valided, construction & $\approx$1ns & 80km (173km) & 2 / 1 / 1 & 233 / 15 / 3 & \\ \hline
%
% \multicolumn{11}{|l|}{} \\
% \multicolumn{11}{|c|}{\textbf{National Time Laboratories}} \\
% \multicolumn{11}{|l|}{} \\ \hline
%
% Time and Freq. Dissimination & MIKES & Finland & TF & & Operational, expanding & & up to 950km & 10 / few /2 & & \\ \hline
% Time and Freq. Dissimination &LNE-SYRTE& France & TF & & Evaluation, tests & &up to 125km (500km)& 4 / 2 / 4 & & \\ \hline
% Time and Freq. Dissimination & VLS & Nederland & TF & & Evaluation, tests & & 137km & & & \\ \hline
% Time and Freq. Dissimination & NIST & USA & TF & & Operational & & $<$10km & & & \\ \hline
% Time and Freq. Dissimination & NLP & UK & TF & & ? & & & & & \\ \hline
% Time and Freq. Dissimination & INRIM & Italy & TF, TS & & Operational, expanding & & up to 400km & & & \\ \hline
%
% \multicolumn{11}{|l|}{} \\
% \multicolumn{11}{|c|}{\textbf{Other Applications}} \\
% \multicolumn{11}{|l|}{} \\ \hline
% Laser Ranging & DLR & Germany & TS & & & & $<$1km & & & \\ \hline
% Extreme Laser Experiment & ELI-ALPS & Hungry & TS & ns to ps & & & $<$1km & & & \\ \hline
% Extreme Laser Experiment & ELI-BEAMS & Czech & TF,TS, TD,TC & & & & $<$1km & 70 / 16 / 2 & & \\ \hline
% Diagnostics in Power Industry & EPFL & Switzerland & TS & & & & $<$1km & 2 / 1 / 1 & & \\ \hline