White Rabbit Newsletter - December 2015
JINR
The Data Acquisition Group in the Laboratory of High Energy Physics (LHEP) of the Joint Institute for Nuclear Research (Dubna, Russia) develops and produces readout electronics and software for data taking. This year WR-enabled devices took part in a real data taking run. The BMN [1] technical run was carried out at the extracted beam of the Nuclotron in February - March 2015.
Readout electronics included waveform digitizers [2] for Zero-Degree and Electromagnetic Calorimeters readout and time-stamping TDC for Time-of-Flight and Drift Chamber detectors. Multiple TDC boards [3] [4], Trigger Master [5] and Readout Controller were built in the VME standard. Trigger and clock are distributed within the crate over a private bus driven by the Trigger Master. Both the waveform digitizer and the trigger master run the White Rabbit Node Core for time synchronization. They have a hardware IP stack for communication.
In the waveform digitizers we had to synchronize a 62.5 MHz clock across devices. It is derived directly from the 125 MHz WR clock and its phase is aligned with timestamp. The TDC system uses a 41.667 MHz clock that is exactly 1/3 of the WR clock. A hardware divider-by-3 is reset every time WR acquires sync and timestamp is a multiple of 3. This method is not ideal and requires a complete system reset after each divider reset.
White Rabbit technology provided a stable and precise time base. The Gigabit data path has been used for readout and board control. The use of an 18-port WR switch for bulk data transfer is limited by the 1 Gb/s port bandwidth. This may be improved by either link aggregation (etherchannel) or implementing 10G interfaces.
WR technology is planned to be used for the control and monitoring systems of NICA, Nuclotron-based Ion Collider Facility in JINR. WR is considered as a timing system in the Multi-Purpose Detector (MPD) of NICA.
[1] http://nica.jinr.ru/
[2] http://afi.jinr.ru/ADC64s2
[3] http://afi.jinr.ru/TDC72VHL
[4] http://afi.jinr.ru/TDC64V
[5] http://afi.jinr.ru/FVME2TM
Anders Wallin
Anders performed a stability and phase-noise measurement of the WRS when
free-running and
GM-locked:
http://www.anderswallin.net/2015/09/white-rabbit-switch-noise-measurement/
Activities in the Netherlands
Nikhef)
Xilinx Artix-7 GTP (Work is in progress to create and validate VHDL sources for a
deterministic PHY for the Xilinx Artix-7 family (using a GTP). This will
supplement the already validated 7-series GTX (used in Kintex-7
and
Virtex-7).
Nikhef, Vrije Universiteit Amsterdam):
Calibration (Some WR users want the freedom to use different SFPs (or other converters), for example for long haul applications. One would like to be able to field-exchange SFPs without the need to do re-calibration. To be able to do so, each network component should be individually calibrated and contain as set of calibration parameters. System level aspects in WR implementations call for remote (and possibly dynamic) update procedures of calibration parameters from network management entities.
Currently there is a single set of calibration parameters for a WR port (combining FPGA, PCB, FMC and SFP propagation delays). We propose to split these parameters by defining an electrical phase plane for each WR port and calibrate the SFPs and FMCs individually. Having the electrical phase plane as a reference implies calibration of the electrical/optical and optical/electrical converters. This domain crossing calibration is under development and first results are promising.
Work is in progress to enable absolute delay calibrations of WR gear. This would enable independent developers or vendors to exchange their WR gear. The latter is an important feature for standardisation. Absolute calibrated devices can be used as "golden standards" for the existing relative calibration procedure.
Uniformly defined measurement definitions, procedures and tooling are under under development. Plans for system level calibration aspects still need to be detailed.
Nikhef, SURFnet, VSL, Vrije Universiteit Amsterdam):
Long haul WR link Delft - Amsterdam (A long-haul test link of 2 x 137km between the Dutch National Metrology
institute (VSL) in Delft and Nikhef in Amsterdam is used to disseminate
UTC (VSL). The link includes two quasi-bidirectional
semiconductor optical amplifiers. After a standard WR-calibration, first
time transfer tests result in 2(8) ns time transfer accuracy on the
round trip over the fibre. The accuracy and uncertainty on this link are
almost completely determined by the unknown chromatic dispersion
encountered. Calibration efforts aiming at sub-ns time transfer accuracy
are in progress. Schemes to determine chromatic
dispersion without modifying the fibre infrastructure between the sites
will be tested. If successful, this will lift a barrier for
implementation of WR on existing fibre infrastructure.
CTA (UVA, Nikhef):
The Cherenkov Telescope Array (CTA:http://cta-observatory.org) is the next generation very high energy gamma-ray experiment surpassing current instruments by at least an order of magnitude in sensitivity. The University of Amsterdam with the help of Nikhef in Amsterdam, the APC (http://www.apc.univ-paris7.fr/APC_CS) in Paris and DESY in Zeuthen (http://www.desy.de) are working on a prototype of the timing system for CTA using White Rabbit as the baseline technology. For Amsterdam and DESY, these developments are also integrated into the ASTERICS infrastructure cluster which is a part of the european Horizon 2020 program (http://asterics2020.eu/).
Within ASTERICS, as part of the CLEOPATRA work package, there is a joint timing effort to enhance White Rabbit to deliver higher precision over very large distances for radio telescopes for VLBI, to facilitate the precision calibration for large scale projects, to operate up to thousands of timing nodes in harsh environments, and to port White Rabbit to 10 Gb/s Ethernet. This ASTERICS timing project is a cooperation of the University of Granada in Spain, DESY in Zeuthen in Germany, JIVE, ASTRON, Vrije Universiteit Amsterdam and the University of Amsterdam, Nikhef, and SURFnet in the Netherlands.
The CTA focus is on using White Rabbit in harsh environments and improvements in White Rabbit calibration, and synergies between KM3NeT and CTA are being explored. The VLBI subtask aims for a proof-of-principle demonstration of VLBI between telescope sites at Dwingeloo and Westerbork, synchronised through a 200 km DWDM fiber link with live data traffic.