Commit 03880d99 authored by Grzegorz Daniluk's avatar Grzegorz Daniluk

papers/ISPCS2011: use common figures

parent ace5ce05
all : WhiteRabbit.pdf
.PHONY : all clean
WhiteRabbit.pdf : WhiteRabbit.tex
latex $^
bibtex WhiteRabbit
latex $^
latex $^
dvips WhiteRabbit
ps2pdf -dPDFX -dEmbedAllFonts=true -dSubsetFonts=true -dEPSCrop=true WhiteRabbit.ps
clean :
rm -f *.eps *.pdf *.dat *.log *.out *.aux *.dvi *.ps *~ *.bbl *.blg
\section{Synchronous Ethernet}
Synchronous Ethernet (SyncE, \cite{biblio:SynchE}) plays a very
important role in timing distribution over the WRN -- it is
responsible for clock syntonization. The reference clock is used to
encode the outgoing data stream. The same clock is retrieved on the
other side of the physical link (by a so-called slave) using a
PLL. SyncE allows to use phase detector technologies to achieve
sub-nanosecond synchronization by increasing the precision of
timestamps far beyond the that allowed by the 125MHz clock resolution
(8ns).
The WR switch implementation of PLLs used to retrieve frequency from the
encoded data is designed to accommodate multiple sources of frequency (physical
links) with a single source being used (active) at a given time. A seamless
switching of the active source is one of the goals of the design. This enables
network topology redundancy. Additionally, the PLL uses a precise PTP link delay
measurement to correct the clock phase.
The WR clock recovery system uses Digital Dual Mixer Time Difference
(DDMTD) phase detection \cite{biblio:WRproject}. It consists of two
units: the Helper PLL (HPLL) and the Main PLL (MPLL), as depicted in
\figurename~\ref{}.
The MPLL uses offset frequency (\ref{equation:offsetFreq}, DMTD clock)
produced by the HPLL.
\begin{equation}
\label{equation:offsetFreq}
f_{offset}[ns] = 125[MHz] * \frac{2^N}{2^N+ \Delta}
\end{equation}
The DMTD clock produced by the VCXO is locked to the output of the
multiplexer of all the possible reference frequencies (uplinks or
external clock). The holdover unit incorporated into the HPLL
automatically switches the reference source in case of failure of the
currently used source.
The MPLL uses a DDMTD for each input channel (reference and feedback
clocks) to produce phase tags. The phase tags, corrected with the
phase shift obtained by WRPTP, are sent to the control
algorithm. Thus, the output reference clock produced by the MPLL is an
in-phase copy of the grandmaster clock. At any time, all the reference
channels are compared with the feedback channel and their frequency
and phase errors are ready for the dual PI controller. Such a solution
enables fast switching between different frequency sources. Although
the detection of reference channel failure is fast (3 symbols, X
time), to avoid instability of the produced reference clock, a delay
buffer can be implemented to accommodate the failure detection time.
%% bare_conf.tex
%% V1.3
%% 2007/01/11
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.7 or later) with an IEEE conference paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/
%% and
%% http://www.ieee.org/
%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE!
%% User assumes all risk.
%% In no event shall IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including **
%% ** renaming them and changing author support contact information. **
%%
%% File list of work: IEEEtran.cls, IEEEtran_HOWTO.pdf, bare_adv.tex,
%% bare_conf.tex, bare_jrnl.tex, bare_jrnl_compsoc.tex
%%*************************************************************************
%
\documentclass[conference]{IEEEtran}
\usepackage{graphicx}
\usepackage{color}
\usepackage{multirow}
%\graphicspath{{fig/}}
\newcommand \todo[1]{\textcolor{red}{\textsl{TODO: }}{\textcolor{black}{#1}}}
\newcommand \modified[1]{{\textcolor{black}{#1}}}
\hyphenation{op-tical net-works semi-conduc-tor}
\begin{document}
\title{White Rabbit: a PTP Application for Robust Sub-nanosecond Synchronization}
\input{authors}
\maketitle
\input{abstract}
% \begin{IEEEkeywords}
% IEEE 1588 Precision Time Protocol (PTP); Synchronous Ethernet; frequency syntonization; time
% synchronization; network redundancy;
% IEEE 1588 Precision Time Protocol (PTP)
% Synchronous Ethernet
% --------------------
% Ethernet networks
% Next generation networking
% Optical fiber networks
% Real time systems
% Time measurement
% Phase frequency detector
% Networked control systems
% Redundancy
% Determinism
% -----------------------
% \end{IEEEkeywords}
%\input{peerreview}
\input{introduction}
\input{wrptp}
\input{hwSupport}
%\input{clockResilience}
\input{testResults}
\input{conclusion}
%\input{biblio}
\bibliographystyle{IEEEtran}
\bibliography{IEEEabrv,./biblio}
\end{document}
\begin{abstract}
%\boldmath
This article describes time distribution in a White Rabbit Network.
We start by presenting a short overview of the White~Rabbit
project explaining its requirements to highlight the importance of
the timing aspects of the system. We then introduce the
technologies used to achieve high clock accuracy, stability and
resilience in all the components of the network. In particular,
the choice of the IEEE~1588-2008 (PTP) and Synchronous Ethernet
standards are explained. In order to accommodate
hardware-supported mechanisms to increase PTP synchronization accuracy,
we introduce the White Rabbit extension to PTP (WRPTP).
The hardware used to support WRPTP is presented.
Measured results of WRPTP performance demonstrate sub-nanosecond accuracy over a 5km
fiber optic link with a precision below 10ps and
a reduced PTP-message exchange rate. Tests of the implementation show
full compatibility with existing PTP gear.
\end{abstract}
% IEEEtran.cls defaults to using nonbold math in the Abstract.
% This preserves the distinction between vectors and scalars. However,
% if the conference you are submitting to favors bold math in the abstract,
% then you can use LaTeX's standard command \boldmath at the very start
% of the abstract to achieve this. Many IEEE journals/conferences frown on
% math in the abstract anyway.
% no keywords
\section*{Acknowledgment}
The authors would like to thank...
\ No newline at end of file
% author names and affiliations
% use a multiple column layout for up to three different
% affiliations
\author{
%\IEEEauthorblockN{Bunch of Freaks}
%\IEEEauthorblockA{CERN\\
%Geneva\\
%Email: white.rabbit@cern.ch}
\IEEEauthorblockN{Maciej Lipi\'{n}ski, Tomasz W\l{}ostowski, Javier Serrano, Pablo Alvarez}
\IEEEauthorblockA{CERN, Geneva\\
Email: \{maciej.lipinski, tomasz.wlostowski, javier.serrano, pablo.alvarez.sanchez\}@cern.ch}
}
% conference papers do not typically use \thanks and this command
% is locked out in conference mode. If really needed, such as for
% the acknowledgment of grants, issue a \IEEEoverridecommandlockouts
% after \documentclass
% for over three affiliations, or if they all won't fit within the width
% of the page, use this alternative format:
%
%\author{\IEEEauthorblockN{Michael Shell\IEEEauthorrefmark{1},
%Homer Simpson\IEEEauthorrefmark{2},
%James Kirk\IEEEauthorrefmark{3},
%Montgomery Scott\IEEEauthorrefmark{3} and
%Eldon Tyrell\IEEEauthorrefmark{4}}
%\IEEEauthorblockA{\IEEEauthorrefmark{1}School of Electrical and Computer
%Engineering\\
%Georgia Institute of Technology,
%Atlanta, Georgia 30332--0250\\ Email: see
%http://www.michaelshell.org/contact.html}
%\IEEEauthorblockA{\IEEEauthorrefmark{2}Twentieth Century Fox, Springfield,
%USA\\
%Email: homer@thesimpsons.com}
%\IEEEauthorblockA{\IEEEauthorrefmark{3}Starfleet Academy, San Francisco,
%California 96678-2391\\
%Telephone: (800) 555--1212, Fax: (888) 555--1212}
%\IEEEauthorblockA{\IEEEauthorrefmark{4}Tyrell Inc., 123 Replicant Street, Los
%Angeles, California 90210--4321}}
@standard{biblio:IEEE1588,
title = "IEEE Standard for a Precision
Clock Synchronization Protocol for Networked Measurement and Control Systems",
organization = "IEEE",
address = "New York",
number = "1588-2008",
year = "2008",
}
@standard{biblio:IEEE8023,
title = "IEEE Standard for
Information Technology--Telecommunications and Information Exchange Between
Systems--Local and Metropolitan Area Networks--Specific Requirements Part 3:
Carrier Sense Multiple Access With Collision Detection (CSMA/CD) Access Method
and Physical Layer Specifications - Section Three",
year = "2008",
organization = "IEEE",
address = "New York",
number = "802.3-2008",
}
@standard{biblio:SynchE,
title = "Timing characteristics of a synchronous Ethernet equipment slave clock {(EEC)}",
year = "2007",
number = "G.8262",
organization = "ITU-T",
}
@inproceedings{biblio:GMT,
author = "J.Serrano and P.Alvarez and D.Dominguez, J.Lewis",
title = "Nanosecond Level {UTC} Timng Generation and Stamping in {CERN}'s {LHC}",
booktitle = "Proceedings of ICALEPSC2003",
address = "Gyeongju, Korea",
year = "2003",
}
@techreport{biblio:FAIRtimingSystem,
author = "T. Fleck and C. Prados and S. Rauch and M. Kreider",
title = "{FAIR} Timing System",
institution = "GSI",
address = "Darmstadt, Germany",
year = "2009",
note = "v1.2",
}
@inproceedings{biblio:distOscilloscope,
author = "S. Deghaye and D. Jacquet and I. Kozsar and J. Serrano",
title = "{OASIS}: A NEW SYSTEM TO ACQUIRE AND DISPLAY THE ANALOG SIGNALS FOR {LHC}",
booktitle = "Proceedings of ICALEPCS2003",
address = "Gyeongju, Korea",
year = "2003",
}
@Inproceedings{biblio:WRproject,
author = "J. Serrano and P. Alvarez and M. Cattin and E. G. Cota and J. H. Lewis, P.
Moreira and T. W\l{}ostowski and others",
title = "{The White Rabbit Project}",
booktitle = "Proceedings of ICALEPCS TUC004",
address = "Kobe, Japan",
year = "2009",
}
@Misc{biblio:WRPTP,
author = "E.G. Cota and M. Lipi\'{n}ski and T. W\l{}ostowski and E.V.D. Bij and J. Serrano",
title = "{White Rabbit Specification: Draft for Comments}",
note = "v2.0",
month = "july",
year = "2011",
howpublished = {\url{http://www.ohwr.org/documents/21}}
}
@mastersthesis{biblio:TomekMSc,
author = "T. W\l{}ostowski",
title = "Precise time and frequency transfer in a {White} {Rabbit} network",
month = "may",
year = "2011",
school = "Warsaw University of Technology",
howpublished = {\url{http://www.ohwr.org/documents/80}}
}
@Inproceedings{biblio:Takahide,
author = "Takahide Murakami and Yukio Horiuchi",
title = "{A Master Redundancy Technique in IEEE 1588 Synchronization with a Link Congestion
Estimation}",
booktitle = "Proceedings of ISPCS",
year = "2010",
}
@electronic{biblio:whiteRabbit,
title = "{White Rabbit}",
howpublished = {\url{http://www.ohwr.org/projects/white-rabbit}}
}
@article{biblio:ohl,
author = "M. Giampietro",
title = "Hardware joins the open movement",
journal = "CERN Courier",
address = "CERN, Geneva",
year = "2011",
howpublished = {\url{http://cerncourier.com/cws/article/cern/46054}},
}
\ No newline at end of file
\section{Clock Resilience}
While the previous chapter described WRPTP focusing on a single
Master-to-Slave link, clock resilience needs to be considered in terms
of the entire WR Network.
\subsection{Clock Path redundancy}
In a WRN, the clock is distributed along so-called \textit{clock
paths}. A path is understood as the cables and switches by which
information is sent from the transmitter (node/switch) to the receiver
(switch/node). The continuity of clock distribution -- existence of
clock paths to all WRN components -- is essential in ensuring clock
resilience. Therefore, clock path redundancy is introduced. This
allows to prevent a \textit{single point of failure} \footnote{Failure
of a single network component.} from affecting clock distribution
and inevitably translates into network topology redundancy, which is
supported by the \textcolor{red}{WR Switch}. The \textcolor{red}{WR
Switch V2 (WRSv2)} has two uplinks which can be connected to
separate sources of timing (downlinks of other \textcolor{red}{WR
Switches} or a node being \textit{grandmaster}). Redundancy of the
WRN is limited by a number of factors:
\begin{itemize}
\item Latency of data delivery which limits the number of network
layers.
\item SyncE which enforces a tree-like network structure and ensures
high-quality frequency distribution only through a limited number of
switches.
\item Data delivery reliability which enforces a tree-like topology
with the roles of ports defined \textit{a priori} by the Rapid
Spaning Tree Protocol (RSTP) algorithm (\cite{biblio:IEEE8021D},
\cite{biblio:Robustness}).
\end{itemize}
Studies (\cite{biblio:Robustness}) suggest that, given the limitations
in topology, the two uplinks of WRSv2 might be not sufficient to
achieve high network reliability. The next version of the
\textcolor{red}{WR Switch (WRSv3)} will eliminate this limitation.
The choice of an active clock path -- the uplink which is used for
syntonization and synchronization -- is made based on the RSTP
algorithm.
\subsection{Switch-over}
The redundancy of clock path ensures continuity of clock distribution
but introduces a possible instability of the recovered clock during
the process of switching between sources (active uplink), further
called \textit{switch-over}. Syntonization and synchronization are
governed by SyncE and WRPTP respectively. Therefore, both need to
take account of stability during \textit{switch-over}.
\subsubsection{SyncE-wise}
The White Rabbit clock recovery unit (described in
Sec.~\ref{sec:hwSupport}), by design, enables multiple inputs (RX
clocks). The phase and frequency errors of all the input clocks are
continuously tracked and fed into the VCTCXO control algorithm and a
\textcolor{red}{delay can be introduced to wait for freqency/phase
error validation}, if tests show such a need. Therefore, SyncE-wise
switch-over is considered seamless for syntonization.
\textcolor{blue}{I would need some numbers and tests here}
\subsubsection{WRPTP-wise}
Since a WRN is a set of independent M-to-S link connections, WRPTP is
unaware of whether a given link is active or not. Delay and offset
measurements are performed on all the links all the time and the
information is provided to the clock recovery unit (see
\figurename~\ref{fig:PLL}). Therefore, the switch-over is unnoticible
for WRPTP and shall be seamless for synchronization.
\textcolor{blue}{ Measurement of "backup" offset and delay with
reference to the primary one - idea by Tomek to further decrease
switch-over instability }
\subsection{External conditions variation}
Apart from the switch-over process, another potential source of clock
instability is a variation of external conditions,
e.g. temperature. It affects the characteristics of the physical
connections, consequently changing the delay introduced by the medium
--the variable delay ($\delta_{ms,sm}$). It is important to note that
frequency distributed over SyncE is not affected by this
phenomenon. Therefore, only synchronization over WRPTP, i.e. delay
change, needs to be compensated. This is done by periodically
measuring the delay through a standard exchange of PTP messages. The
frequency of measurements needs to be greater then the speed of
temperature changes, which is reasonably slow.
\textcolor{red}{Therefore, a much lower rate of message exchange than
in standard PTP is sufficient.}
\subsection{Loss of WRPTP-messages}
PTP employs timeouts to address PTP-specific message loss, provoking
repetition of operations and re-sending of messages. WRPTP uses the
same idea during the \textit{WR Link Setup} (see
Sec.~\ref{sec:wrLinkSetup}) repeating operations and re-sending WR
management messages in case of message loss. The measurement of the
offset and delay in WRPTP is much more tolerant to multiple message
loss. Unlike standard PTP, WRPTP is responsible only for
synchronization (syntonization is done through SyncE). After achieving
synchronization with the master at the beginning of the connection,
the offset changes only due to temperature-related delay
variation. The rate of delay measurements through the PTP-message
exchange is supposed to be much greater than the rate of change of
physical medium parameters. Therefore, multiple PTP-message loss is
tolerated with no effects on clock stability. \textcolor{blue}{we
should probably have a sanity check in the PTP daemon for ruling out
impossible corrections, i.e. very fast changes probably due to a
measurement or transmission error}
\subsection{Cascading Boundary Clocks}
A switch can be seen as a boundary clock. In standard PTP, a cascade
of boundary clocks faces nonlinear decreasing synchronization accuracy
problems due to error accumulation. \textcolor{blue}{
\begin{itemize}
\item if it is possible to make measurements (we need $\geq$ 3
switches), measurements here
\item the deterioration should be due to SyncE...
\item measurements would answer the question: how many layers of
switches we can have.
\end{itemize}
}
\ No newline at end of file
\section{Conclusions}
White Rabbit is an open hardware \modified{\cite{biblio:ohl}} and open software project pushing the
frontiers of technology but also introducing new trends in
cooperation between public institutions and companies.
The WRN is purposely based on well-established
standards to ensure its long lifetime, wide support and commercial
feasibility.
By blending existing technologies, hardware-supporting and extending
them, \modified{still} %Erik's remarks
staying compatible, exceptional results are achieved. This
includes sub-ns accuracy of \modified{high precision and robust} synchronization, which
proved to be the
most accurate known PTP implementation (ISPCS, September 2010, USA) and frequency
transfer performance of below 2ps integrated jitter using a FPGA-based
PLL design.
The WRPTP extension allows not only for increased accuracy,
but also offers much lower PTP-message traffic and higher reliability
by supporting network redundancy without any loss of performance,
\modified{thus offering robust synchronization}.
Compatibility with existing standards enables hybrid networks, where
time- and data-critical end nodes are connected directly to a White
Rabbit Network, while less critical devices use standard switches to
connect to it.
\ No newline at end of file
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Hardware support
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{WR Hardware Support}
\label{sec:hwSupport}
SyncE is responsible for clock syntonization in the WRN. In the SyncE scheme, the
reference clock (125MHz) is used to encode the outgoing data
stream. The same clock is retrieved on the other side of the physical
link using the \textit{Clock Recovery System} (CRS, section~\ref{sec:wrCRS}).
Having the same frequency allows us to use phase detector technologies as a means of evaluating
delays. WR implements Digital Dual Mixer Time Difference (DDMTD) \cite{biblio:WRproject}
phase detection. The measurement of phase is used for
increasing the precision of timestamps beyond the resolution allowed by the 125MHz clock.
This process \modified{is} described in the next section.
The DDMTD phase detection is also used to obtain
\textit{the transmit/receive (Tx/Rx) latencies} (section~\ref{sec:TxRxLatencies})
\modified{and compare frequencies in the CRS}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% timestamps and fine delay
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Fine Delay Measurement}
\label{sec:fineDelay}
We call \textit{Fine Delay Measurement} the process during which the DDMTD-detected round-trip phase
shift ($phase_{MM}$ in \figurename~\ref{fig:wrLink}) is used to enhance
timestamp precision and to calculate the precise round-trip delay
($delay_{MM}$).
During the fine delay process only the reception timestamp ($t_2$, $t_4$)
measurements need to be improved as they are transmitted and timestamped in
different clock domains.
A basic timestamp (to be enhanced) is obtained by the detection of
the Start-of-Frame Delimiter (SFD) in the Physical Coding Sublayer (PCS).
In order to acquire a precision-improved timestamp, we need to eliminate the possible
$\pm$~1~LSB error \modified{(8ns)} %Pablo's remark to add unit
due to jitter of clock signals and clock-domain crossing.
Therefore, the Time-Stamping Unit (TSU) produces timestamps on both the rising
and falling edges of the clock, and later in the process one of them is chosen.
The process involves three steps (see \cite{biblio:TomekMSc} for details):
% \begin{figure}[!t]
% \centering
% \includegraphics[width=2.5in]{fig/preciseTimestamp.ps}
% \caption{Timestamp enhancing \cite{biblio:TomekMSc}.}
% \label{fig:preciseTimestamp}
% \end{figure}
% % Only the reception timestamps ($t_2$, $t_4$)
% need to be enhanced as they are transmitted and timestamped in
% different clock domains.
%(see \figurename~\ref{fig:preciseTimestamp} for $t_{4}$ to $t_{p4}$
%enhancement explanation)
\begin{enumerate}
\item rising/falling edge timestamp choice ($t_f$ or $t_r$),
\item calculating the picosecond part, checking
its sign and adding a clock period if necessary,
\item extending timestamps with the picosecond part ($t_{2p}$, $t_{4p}$).
\end{enumerate}
The modified timestamps are used to calculate the precise round-trip delay
(\figurename~\ref{fig:wrLink}):
\begin{equation}
\label{eq:delaymm}
delay_{MM} = (t_{4p}-t_{1}) - (t_{3}-t_{2p})
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% WR Clock Recovery System
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\modified{Robust} Clock Recovery System}
\label{sec:wrCRS}
\begin{figure}[!t]
\centering
%\includegraphics[width=2.95in]{fig/wrCRS.eps}
\includegraphics[height=0.95in]{../../figures/robustness/wrCRS.eps}
\caption{\modified{White Rabbit Clock Recovery System.}}
\label{fig:PLL}
\end{figure}
The White Rabbit implementation of the CRS (WR CRS) is explained in
detail in \cite{biblio:TomekMSc}, \figurename~\ref{fig:PLL} presents
a very simplified overview of the WR CRS. It is designed to accommodate
multiple sources of frequency (physical links) with a single source
being used (active) at a given time. A seamless switching of the
active source is one of the goals of the design to enable network
topology redundancy \modified{and, as a consequence, offer robust and stable synchronization
(see section~\ref{sec:wrBMC})}.
All input reference clocks (\textit{ref clk}) and a feedback clock (\textit{feedback clk}) are
fed into DDMTD and degliching units (\textit{DDMTD}). The input clocks are mixed by the DDMTDs with
the offset frequency (\ref{equation:offsetFreq}) which is derived from the
active ref~clock. The frequency-mixing results in a low frequency clock signal which maintains the
original
phase shift.
\begin{equation}
\label{equation:offsetFreq}
f_{offset}[ns] = 125[MHz] * \frac{2^N}{2^N+ \Delta}% \mbox{ , }N=14 \mbox{ \& } \Delta=1
\end{equation}
A prototype implementation using $N=14$ and $\Delta=1$ has demonstrated satisfactory performance
\cite{biblio:TomekMSc}. Each output of the DDMTD reference channel is compared with the output
of the feedback channel in the \textit{phase and frequency error detection} units. The additional
input to the units is the \textit{WRPTP phase shift} obtained in the \textit{Fine Delay
Measurement}. The phase and frequency error (\textit{phase and freq error}) between the feedback
clock and each reference clock, corrected by WRPTP phase shift, is calculated and fed into the
multiplexer (\textit{MUX}).
During normal operation, the active reference clock (from the Primary Slave port,
section~\ref{sec:wrBMC}) is used to feed the PI controller and produce the output reference signal.
However, if the malfunction of the active ref clock is detected, the switch-over process
takes place: the MUX is switched to feed the PI with the error data from another reference channel
(from the Secondary Slave port).
\modified{An estimate of the average phase and frequency errors can be provided to the PI controller
to improve hold-over performance if all the reference channels fail.}
The active clock malfunction takes 3 consecutive invalid symbols
($\sim24ns$) to detect while the phase and frequency error detection is performed
at a low frequency (kHz). Therefore, the continuity of the synchronization is
guaranteed during the switch-over.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Tx/Rx calibration
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{The transmit/receive (Tx/Rx) latencies}
\label{sec:TxRxLatencies}
\begin{figure}[!t]
\centering
%\includegraphics[width=1.74in]{fig/calibration.eps}
\includegraphics[height=0.95in]{../../figures/misc/calibration.eps}
\caption{\modified{Rx/Tx latency measurement \cite{biblio:WRPTP}. }}
\label{fig:calibration}
\end{figure}
The transmit/receive (Tx/Rx) latencies in most PHYs vary for
each \modified{Phase-Locked Loop/Clock Data Recovery (PLL/CDR)}
%\modified{PLL/CDR\footnote{\modified{Phase-Locked Loop/Clock Data Recovery}}}
lock cycle, but stay constant once the PHY is
locked. This is the case of the PHY used in the WR Switch prototype
(TCK1221), therefore an Tx/Rx latency measurement is required. Tx
latency is measured by feeding the transmit path with a sequence of
RD+ K28.7 code-group (Appendix~36A.2 of \cite{biblio:IEEE8023}).
Such signal creates a 125MHz clock on
the SerDes I/O. Since the Tx clock frequency is also 125MHz, the DDMTD
can be used to measure the phase shift between the SerDes I/O and the
Tx clock, effectively measuring Tx latency. By receiving the K28.7
signal from the link partner, the Rx latency can be measured using the
same method. This process, depicted in \figurename~\ref{fig:calibration},
is performed during the WR Link Setup (section~\ref{sec:wrLinkSetup}).
\section{Introduction}
White Rabbit (WR, \cite{biblio:whiteRabbit}) is a project which aims at creating an
Ethernet-based network with low-latency, deterministic packet delivery
and network-wide, transparent, high-accuracy timing distribution. The
White Rabbit Network (WRN) is based on existing standards, namely
Ethernet (IEEE 802.3 \cite{biblio:IEEE8023}), Synchronous Ethernet
(SyncE \cite{biblio:SynchE}) and PTP \cite{biblio:IEEE1588}.
It is fully compatible with these standards.
A WRN consists of White Rabbit Nodes (nodes) and White Rabbit Switches
(switches) interconnected by fiber or copper links.
\modified{The focus of this article is on 1000Base-LX \cite{biblio:IEEE1588}
single-mode fiber connections only. WR }%It also
supports integration of nodes and/or switches that are not White Rabbit.
A simple WRN is presented in \figurename~\ref{fig:WRnetwork}.
A node is considered the source and destination of information sent
over the WRN. The information distributed over a WRN includes:
\begin{itemize}
\item Timing - frequency and International Atomic Time (TAI).
\item Data - Ethernet traffic between nodes.
\end{itemize}
In order to understand the goals of the WR project -- namely
determinism, high reliability and accurate synchronization -- these
terms are explained below.
\textbf{Determinism} is guaranteed by having a worst-case upper bound
in frame delivery latency.
\textbf{Accuracy} is a measure of the deviation between the clock of
the \textit{grandmaster} node\modified{/switch} of a WRN and that of any other node.
Assuming only fiber interconnections, a WRN is meant to achieve
sub-nanosecond accuracy.
\textbf{Reliability} of a WRN refers to robust delivery of data and
timing to all the nodes.
% data being delivered in a deterministic manner.
%A WRN is considered reliable only if all the nodes, receive
%data and timing.
The timing must allow all the nodes to be
synchronized with the required accuracy and the data must always be
delivered on time.
\begin{figure}[!t]
\centering
\includegraphics[height=2.10in]{../../figures/network/hierarchy.eps}
\caption{A White Rabbit Network \cite{biblio:TomekMSc}.}
\label{fig:WRnetwork}
\end{figure}
All these features are required to create a timing and control system which may
replace the General Machine Timing (GMT) \cite{biblio:GMT} at CERN and
fulfill a similar role at the Facility for Antiproton and Ion
Research (FAIR) in GSI \cite{biblio:FAIRtimingSystem}. Such system requires
synchronization of up to 2000 nodes with sub-nanosecond accuracy,
an upper bound in frame delivery and a very low data loss rate.
However, many other applications of White Rabbit are possible. This includes industry,
telecommunications and other large distributed systems (e.g. distributed
oscilloscopes \cite{biblio:distOscilloscope}).
This article focuses solely on the PTP-based timing distribution in a WRN. PTP is a packet-based
protocol designed to synchronize devices in distributed systems. The accuracy of the PTP
synchronization is implementation-dependent. The standard is foreseen for
% this is how I understand the fact that corretionField can carry fractional
% nanoseconds and the syncEventEgressTimestamp includes this figures (see e.g.:
% clause 9.5.9.3, page 103 of PTP
sub-nanosecond accuracies. However, such performance is not achieved in typical PTP
implementations for two reasons:
\begin{itemize}
\item Limited precision and resolution of PTP timestamps.