White Rabbit Newsletter - December 2015
JINR
The Data Acquisition Group in the Laboratory of High Energy Physics (LHEP) of the Joint Institute for Nuclear Research (Dubna, Russia) develops and produces readout electronics and software for data taking. This year WR-enabled devices took part in a real data taking run. The BMN [1] technical run was carried out at the extracted beam of the Nuclotron in February - March 2015.
Readout electronics included waveform digitizers [2] for Zero-Degree and Electromagnetic Calorimeters readout and time-stamping TDC for Time-of-Flight and Drift Chamber detectors. Multiple TDC boards [3] [4], Trigger Master [5] and Readout Controller were built in the VME standard. Trigger and clock are distributed within the crate over a private bus driven by the Trigger Master. Both the waveform digitizer and the trigger master run the White Rabbit Node Core for time synchronization. They have a hardware IP stack for communication.
In the waveform digitizers we had to synchronize a 62.5 MHz clock across devices. It is derived directly from the 125 MHz WR clock and its phase is aligned with timestamp. The TDC system uses a 41.667 MHz clock that is exactly 1/3 of the WR clock. A hardware divider-by-3 is reset every time WR acquires sync and timestamp is a multiple of 3. This method is not ideal and requires a complete system reset after each divider reset.
White Rabbit technology provided a stable and precise time base. The Gigabit data path has been used for readout and board control. The use of an 18-port WR switch for bulk data transfer is limited by the 1 Gb/s port bandwidth. This may be improved by either link aggregation (etherchannel) or implementing 10G interfaces.
WR technology is planned to be used for the control and monitoring systems of NICA, Nuclotron-based Ion Collider Facility in JINR. WR is considered as a timing system in the Multi-Purpose Detector (MPD) of NICA.
[1] http://nica.jinr.ru/
[2] http://afi.jinr.ru/ADC64s2
[3] http://afi.jinr.ru/TDC72VHL
[4] http://afi.jinr.ru/TDC64V
[5] http://afi.jinr.ru/FVME2TM
Anders Wallin
Anders performed a stability and phase-noise measurement of the WRS when
free-running and
GM-locked:
http://www.anderswallin.net/2015/09/white-rabbit-switch-noise-measurement/
Activities in the Netherlands
Nikhef)
Xilinx Artix-7 GTP (Work is in progress to create and validate VHDL sources for a
deterministic PHY for the Xilinx Artix-7 family (using a GTP). This will
supplement the already validated 7-series GTX (used in Kintex-7
and
Virtex-7).
Nikhef, Vrije Universiteit Amsterdam):
Calibration (Some WR users want the freedom to use different SFPs (or other converters), for example for long haul applications. One would like to be able to field-exchange SFPs without the need to do re-calibration. To be able to do so, each network component should be individually calibrated and contain as set of calibration parameters. System level aspects in WR implementations call for remote (and possibly dynamic) update procedures of calibration parameters from network management entities.
Currently there is a single set of calibration parameters for a WR port (combining FPGA, PCB, FMC and SFP propagation delays). We propose to split these parameters by defining an electrical phase plane for each WR port and calibrate the SFPs and FMCs individually. Having the electrical phase plane as a reference implies calibration of the electrical/optical and optical/electrical converters. This domain crossing calibration is under development and first results are promising.
Work is in progress to enable absolute delay calibrations of WR gear. This would enable independent developers or vendors to exchange their WR gear. The latter is an important feature for standardisation. Absolute calibrated devices can be used as "golden standards" for the existing relative calibration procedure.
Uniformly defined measurement definitions, procedures and tooling are under under development. Plans for system level calibration aspects still need to be detailed.
Nikhef, SURFnet, VSL, Vrije Universiteit Amsterdam):
Long haul WR link Delft - Amsterdam (A long-haul test link of 2 x 137km between the Dutch National Metrology
institute (VSL) in Delft and Nikhef in Amsterdam is used to disseminate
UTC (VSL). The link includes two quasi-bidirectional
semiconductor optical amplifiers. After a standard WR-calibration, first
time transfer tests result in 2(8) ns time transfer accuracy on the
round trip over the fibre. The accuracy and uncertainty on this link are
almost completely determined by the unknown chromatic dispersion
encountered. Calibration efforts aiming at sub-ns time transfer accuracy
are in progress. Schemes to determine chromatic
dispersion without modifying the fibre infrastructure between the sites
will be tested. If successful, this will lift a barrier for
implementation of WR on existing fibre infrastructure.
CTA (UVA, Nikhef):
The Cherenkov Telescope Array (CTA:http://cta-observatory.org) is the next generation very high energy gamma-ray experiment surpassing current instruments by at least an order of magnitude in sensitivity. The University of Amsterdam with the help of Nikhef in Amsterdam, the APC (http://www.apc.univ-paris7.fr/APC_CS) in Paris and DESY in Zeuthen (http://www.desy.de) are working on a prototype of the timing system for CTA using White Rabbit as the baseline technology. For Amsterdam and DESY, these developments are also integrated into the ASTERICS infrastructure cluster which is a part of the european Horizon 2020 program (http://asterics2020.eu/).
Within ASTERICS, as part of the CLEOPATRA work package, there is a joint timing effort to enhance White Rabbit to deliver higher precision over very large distances for radio telescopes for VLBI, to facilitate the precision calibration for large scale projects, to operate up to thousands of timing nodes in harsh environments, and to port White Rabbit to 10 Gb/s Ethernet. This ASTERICS timing project is a cooperation of the University of Granada in Spain, DESY in Zeuthen in Germany, JIVE, ASTRON, Vrije Universiteit Amsterdam and the University of Amsterdam, Nikhef, and SURFnet in the Netherlands.
The CTA focus is on using White Rabbit in harsh environments and improvements in White Rabbit calibration, and synergies between KM3NeT and CTA are being explored. The VLBI subtask aims for a proof-of-principle demonstration of VLBI between telescope sites at Dwingeloo and Westerbork, synchronised through a 200 km DWDM fiber link with live data traffic.
University of Granada (UGR) and Seven Solutions
UGR is contributing to the White-Rabbit community with different R&D projects involving the WR Switch, WR nodes and, together with Seven Solutions, the development of new stand-alone versions of WR nodes based on the Xilinx Zynq architecture [1]. At the same time, UGR is collaborating with other WR partners in the development of a reliable WR node with redundancy capabilities to avoid single point of failure issues.
h3 WR-ZEN developments
The WR-ZEN [2] is an open hardware board designed by Seven Solutions. It uses the Xilinx Zynq technology that contains an ARM processor and a FPGA device in the same System on a Chip (SoC) taking the advantages of a better HW/SW partitioning and interoperability between the FPGA and the ARM processor.
UGR with the collaboration of Seven Solutions have implemented a WR node for the WR-ZEN board with two fully functional SFP ports implementing Fine Delay FMC features.
The WR-ZEN gateware has been designed by UGR and Seven Solutions including some additional IP cores such as a dual-port WRPC, an AXI-WB Bridge core and an I2C Arbiter. In addition, LM32 firmware has been updated to work properly with the new gateware and board architecture taking into account its new features and peculiarities.
In terms of software, the board runs a SMP Linux kernel [3] where several drivers have been adapted from the SPEC project to control the NIC and the Fine Delay cores. On the userspace, some tools have been modified to work with this new architecture. Some examples of these tools are the zen-vuart, zenmem and those related with the Fine Delay driver such as fmc-fdelay-input and fmc-fdelay-pulse.
White Rabbit High-availability Seamless Redundancy (WR-HSR)
UGR is working on increasing availability and robustness of the White
Rabbit technology to extend its utilization to industrial networks such
as Smart Grid. Applications related to this field demand high
availability and very short switch over time. In order to meet these
requirements, the electrical substation automation standard IEC 61850
[4] suggests the High-availability Seamless Redundancy (HSR, IEC
62439-3) [5] as one of the redundancy protocols. HSR guarantees
zero-time recovery in case of failure in ring and mesh network
topologies for both time and data frames and it is based on the
duplication of frames in the network.
UGR is currently implementing the HSR protocol for WR devices, in
particular the WR-Switch to increase availability and avoid single point
of failure for time and data distribution. This work involves the
development of the Peer-2-Peer mechanism [6] for clock propagation,
PeerDelay [6] for the delay measurement, the utilization of HSR tags
[5] in frames, and the SwitchOver mechanism developed by CERN to
recover from a node failure. Furthermore, additional gateware must be
developed to duplicate and forward frames. Current progress and more
information can be found at the OHWR WR-HSR webpage:
https://www.ohwr.org/project/wr-hsr/wiki
References
[1] Xilinx Zynq technology and devices, Available online at
http://www.xilinx.com/products/silicon-devices/soc/zynq-7000.html
[2] WR-ZEN board, Available online at
http://www.sevensols.com/es/productos/wr-zen.html
[3] Xilinx Linux OS, Available online at
https://github.com/Xilinx/linux-xlnx
[4] IEC 61850: Power Utility Automation Standard
[5] IEC 62439-3:2012. Industrial communication networks - High
availability automation networks
[6] IEEE 1588-2002. Precision Clock Synchronization Protocol for
Networked Measurement and Control Systems
DESY (CTA, HiSCORE)
DESY has continued its WR-based timing support for HiSCORE, the Gamma and Cosmic Ray detector in the Tunka-valley/Siberia.
The success of this pioneering WR-timing application yielded critical
input to propose for the Cherenkov Telescope Array (CTA; see
http://cta-observatory.org) - the next generation very high energy
gamma-ray experiment - a centralized, WR-based clock-distribution and
trigger-time stamping system with sub-nsec precision and sufficient
redundancy to operate in harsh conditions.
The system is now in its prototyping phase (activity in collaboration
with APC, Paris and UvA, Amsterdam).
HiSCORE
The 9-station HiSCORE setup has been successfully operating since the 2013/2014 observation period. The Array has been enlarged to 28 stations covering 0.3 km^2 in 2014.
The 9-station WR-setup is based on a SPEC with DIO5Ch, which is operated as a 1-nsec time-stamping unit, with the analog PMT signals directly fed into the DIO5Ch, thus acting as a discriminator with adjustable threshold and digital pulse selection capability. Additional DAQ functionality for waveform sampling readout (DRS4-EB; trigger and busy) turned the SPEC/DIO5Ch into a complete front-end board. See contributions to ECRS2014 and ICRC2015 conferences.
The new stations (2014) have a custom-made front-end board (DRS4-based,
MSU/Moscow) connected as Mezzanine to a WR-SPEC. Detailed long-term test
of WR performance with a
custom made time-distribution system shows excellent WR-perfomance.
DESY has worked in close collaboration with Humboldt-University Berlin (Martin Brueckner, now PSI).