This section tries to identify all the possible ways the White Rabbit Switch can
fail. The structure of each error description is the following:
\begin{itemize}[leftmargin=0pt]
	\item [] \underline{Status}: describes the implementation status of the WRS
		diagnostics detecting the fault. Can be one of the following:
		\begin{packed_items}
			\item DONE: all the SNMP objects are implemented and the problem is
				reported by a switch
			\item TODO: not all of the SNMP objects are already implemented, the
				problem is either reported only in some situations or not reported at
				all
			\item \emph{for later}: the problem concerns functionality that is not yet
				present in the stable release of the WR switch firmware i.e. it will
				never happen with the current stable firmware release.
		\end{packed_items}
	\item [] \underline{Severity}: describes how critical is the fault. Currently
		we distinguish two severity levels:
		\begin{packed_items}
			\item WARNING - means that despite the fault the synchronization and
				Ethernet switching functionality were not affected so the switch behaves
				correctly in the WR network.
			\item ERROR - means that the fault is critical and most probably a WR
				switch misbehaves in a WR network, possibly causing also problems to
				other WR devices connected to this switch.
		\end{packed_items}
	\item [] \underline{Mode}: for timing failures, it describes which modes are
		affected. Possible values are:
		\begin{packed_items}
			\item \emph{Boundary Clock} - the WR Switch has at least one Slave port
				synchronized to another WR device higher in the timing hierarchy (though
				it may be also Master to other WR/PTP devices lower in the timing
				hierarchy).
			\item \emph{Grand Master} - the WR Switch at the top of the
				synchronization hierarchy. It is synchronized to an external clock (e.g.
				GPS, Cesium) and provides timing to other WR/PTP devices.
			\item \emph{Free-Running Master} - the WR Switch at the top of the
				synchronization hierarchy. It provides timing to other WR/PTP devices
				but runs from a local oscillator (not synchronized to an external
				clock).
			\item \emph{all} - any WR switch can be affected regardless the timing
				mode.
		\end{packed_items}

	\item [] \underline{Description}: What the problem is about, how important it
		is and what are the effects if it occurs.
	\item [] \underline{SNMP objects}: Which SNMP objects should be monitored to
		detect the failure. These may be objects from \texttt{WR-SWITCH-MIB} or one
		of the standard MIBs used by the \emph{net-snmp}.
	\item [] \underline{Notes}: Optional comment for the SNMP implementation. It
		may describe the current implementation of ideas or how to implement it in
		the future.
\end{itemize}

\subsection{Timing error}
\label{sec:timing_fail}
As a timing error we define the WR Switch not being able to provide its slave
nodes/switches with correct timing information consistent with the rest of the
WR network.

\noindent This section contains the list of faults leading to a timing error.

\subsubsection{\bf \emph{PTP/PPSi} went out of \texttt{TRACK\_PHASE}}
		\label{fail:timing:ppsi_track_phase}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Boundary Clock}
			\item [] \underline{Description}:\\
				If the \emph{PTP/PPSi} WR servo goes out of the \texttt{TRACK\_PHASE}
				state, this means something bad has happened and the switch lost the
				synchronization to its Master.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPtpServoState.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpServoStateN.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpServoStateErrCnt.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPTPStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }

			\item [] \underline{Note}: PTP servo state is exported as a string and a number.
		\end{pck_descr}

\subsubsection{\bf Offset jump not compensated by Slave}
		\label{fail:timing:offset_jump}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Boundary Clock}
			\item [] \underline{Description}:\\
				This may happen if the Master resets its WR time counters (e.g. because
				it lost the link to its Master higher in the hierarchy or to external
				clock), but the WR Slave does not follow the jump.
				\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPtpClockOffsetPs.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpClockOffsetPsHR.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpClockOffsetErrCnt.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPTPStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Detected jump in the RTT value calculated by \emph{PTP/PPSi}}
		\label{fail:timing:rtt_jump}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Boundary Clock}
			\item [] \underline{Description}:\\
				Once a WR link is established the round-trip delay (RTT) can change
				smoothly due to the temperature variations. However, if a sudden jump is
				detected, that means that an erroneous timestamp was generated either on
				the Master or the Slave side.
				One cause of that could be the wrong value of the t24p transition point.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPtpRTT.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpRTTErrCnt.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPTPStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Wrong $\Delta_{TXM}$, $\Delta_{RXM}$, $\Delta_{TXS}$,
		$\Delta_{RXS}$ values are reported to the \emph{PTP/PPSi} daemon}
		\label{fail:timing:deltas_report}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If \emph{PTP/PPSi} doesn't get the correct values of fixed hardware delays,
				it won't be able to calculate a proper Master-to-Slave delay. Although
				the estimated offset in \emph{PTP/PPSi} is close to 0, the WRS won't be
				synchronized to the Master with the sub-nanosecond accuracy.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPtpDeltaTxM.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpDeltaRxM.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpDeltaTxS.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPtpDeltaRxS.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPTPStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf \emph{SoftPLL} became unlocked}
		\label{fail:timing:spll_unlock}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE (to be improved with holdover)
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If the \emph{SoftPLL} loses lock, for any reason, Boundary Clock or
				Grand Master switch can no longer be syntonized and phase aligned with
				its time source. WRS in Free-running mode without properly locked Helper
				PLL is not able to perform reliable phase measurements for enhancing Rx
				timestamps resolution. For a Grand Master the reason of \emph{SoftPLL}
				going out of lock might be disconnected 1-PPS/10MHz signals or that the
				external clock is down. In that case, the switch goes into Free-running
				mode and resets the WR time. Later we will have a holdover to keep the
				Grand Master switch disciplined in case it loses external reference.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsSpllMode}\\
				\snmpadd{WR-SWITCH-MIB::wrsSpllSeqState}\\
				\snmpadd{WR-SWITCH-MIB::wrsSpllAlignState}\\
				\snmpadd{WR-SWITCH-MIB::wrsSpllHlock}\\
				\snmpadd{WR-SWITCH-MIB::wrsSpllMlock}\\
				\snmpadd{WR-SWITCH-MIB::wrsSpllDelCnt}\\
				\snmpadd{WR-SWITCH-MIB::wrsSoftPLLStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf \emph{SoftPLL} has crashed/restarted}
		\label{fail:timing:spll_crash}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO \emph{(depends on SoftPLL mem read), (requires changes in lm32 software)}
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If the LM32 software crashes or restarts for some reason, its state may
				be either reset or random (if for some reason variables were overwritten
				with junk values). In such case, PLL becomes unlocked and switch is not
				able to provide synchronization to other devices.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsSpllIrqCnt}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntSPLL} }
			\item [] \underline{Note}: We have a similar mechanism as in the
				\emph{wrpc-sw} to detect if the LM32 program has restarted because of
				the CPU following a NULL pointer. However, LM32 program hangs on
				re-initialization phase. 
				In addition to that, we can detect if
				\emph{SoftPLL} is hanging (but not restarted) based on irq counter.
		\end{pck_descr}

\subsubsection{\bf Link to WR Master is down for slave}
		\label{fail:timing:master_down}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR (will become WARNING with the
				switch-over)
			\item [] \underline{Mode}: \emph{Boundary Clock}
			\item [] \underline{Description}:\\
				If a Boundary Clock switch loses the link on its Slave port, the timing
				reference is lost. The switch resets counters responsible for keeping
				the WR time, and starts operating in a Free-Running Master mode.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusLink.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsSlaveLinksStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Link to WR Master is up for master}
		\label{fail:timing:master_up}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Grand Master}, \emph{Free-Running Master}
			\item [] \underline{Description}:\\
				In that case there is probably a wrong configuration. Neither the
				\emph{Grand Master} nor the \emph{Free-Running Master} should be
				connected to another WR Master.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusLink.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsSlaveLinksStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf PTP frames don't reach ARM}
		\label{fail:timing:no_frames}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				In this case, \emph{PTP/PPSi} will fail to stay synchronized and provide
				synchronization. Even if the WR servo is in the \texttt{TRACK\_PHASE}
				state, it calculates a new phase shift based on the Master-to-Slave delay
				variations. To calculate these variations, it still needs timestamped
				PTP frames flowing. There could be several causes of such fault:
				\begin{itemize}
					\item HDL problem (e.g. SwCore or Endpoint hanging)
					\item \emph{wr\_nic.ko} driver crash
					\item wrong VLANs configuration
				\end{itemize}
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusPtpTxFrames.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusPtpRxFrames.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusLink.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPTPFramesFlowing}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsVlansSetStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatusGroup}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
			\item [] \underline{Note}: If the kernel driver crashes, there is not much
				we can do. We end up with either our system frozen or a reboot. For
				wrong VLAN configuration and HDL problems we can monitor if PTP frames
				are flowing on Slave port(s) of WRS and raise an alarm (change status
				word) if they don't flow anymore. We should combine this with the link
				status (up/down). If VLANs are mis configured, we don't receive PTP
				frames, but the link is still up. This could let us distinguish from a
				lack of frames due to the link down (which is a separate issue).
		\end{pck_descr}

\subsubsection{\bf Detected SFP not supported for WR timing}
		\label{fail:timing:wrong_sfp}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				By not supported SFP for WR timing we mean a transceiver that doesn't
				have the \emph{alpha} parameter and fixed hardware delays defined in the
				SFP database (\texttt{CONFIG\_SFPXX\_PARAMS} parameters in dot-config). The consequence is
				\emph{PTP/PPSi} not having the right values to estimate link asymmetry.
				Despite \emph{PTP/PPSi} offset being close to 0 \emph{ps}, the device won't
				be properly synchronized.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpVN.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpPN.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpVS.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpInDB.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpGbE.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpError.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusMonitor.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsSFPsStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsNetworkingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
			\item [] \underline{Note}: WRS configuration allow to disable this check on some ports.
				That is because ports may be used for regular (non-WR) PTP
				synchronization or for data transfer only (no timing). In that case any
				Gigabit SFP can be used (also copper). Detecting if a non-Gigabit
				Ethernet SFP is plugged into the cage is covered in issue
				\ref{fail:other:sfp}.
		\end{pck_descr}

\subsubsection{\bf \emph{PTP/PPSi} process has crashed/restarted}
		\label{fail:timing:ppsi_crash}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If the \emph{PTP/PPSi} daemon crashes we lose any synchronization
				capabilities. Then \texttt{Monit} restarts the missing process.
				The number of process starts is stored in a corresponding object.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsStartCntPTP}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\snmpadd{HOST-RESOURCES-MIB::hrSWRunName.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf \emph{HAL} process has crashed/restarted}
		\label{fail:timing:hal_crash}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If \emph{HAL} crashes, \emph{PTP/PPSi} is not able to communicate with
				the hardware i.e. read phase shift, get timestamps, phase shift the
				clock etc. When \emph{HAL} crashes, \texttt{Monit} will restart it.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsStartCntHAL}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\snmpadd{HOST-RESOURCES-MIB::hrSWRunName.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf System/WR time has diverged from the reference NTP time}
		\label{fail:timing:systime}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				System clock monitoring periodically checks
				whether the system (and thus White Rabbit time)
				have not diverged above the configured
				thresholde from the reference NTP time.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
    				\snmpadd{WR-SWITCH-MIB::wrsSystemClockStatus}\\
    				\snmpadd{WR-SWITCH-MIB::wrsSystemClockStatusDetails}\\
    				\snmpadd{WR-SWITCH-MIB::wrsSystemClockDrift}\\
    				\snmpadd{WR-SWITCH-MIB::wrsSystemClockDriftThreshold}\\
    				\snmpadd{WR-SWITCH-MIB::wrsSystemClockCheckInterval}\\
				\snmpadd{WR-SWITCH-MIB::wrsSystemClockCheckIntervalUnit} }
		\end{pck_descr}

\subsubsection{\bf Leap seconds file is incorrect or expired}
		\label{fail:timing:systime}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING/ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				This problem does not affect applications
				relying on TAI time. However if an incorrect
				leap seconds file is used, applications relying
				on UTC will receive incorrect/outdated
				information.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
    				\snmpadd{WR-SWITCH-MIB::wrsSystemClockStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsLeapSecStatusDetails}\\
				\snmpadd{WR-SWITCH-MIB::wrsLeapSecSource}\\
				\snmpadd{WR-SWITCH-MIB::wrsLeapSecSourceURL} }
		\end{pck_descr}

\subsubsection{\bf Wrong configuration applied}
		\label{fail:timing:wrong_config}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If there is a wrong configuration applied to the \emph{PTP/PPSi} or HAL
				(i.e.  wrong fixed delays, mode of operation etc.) there is not much we
				can do. The responsibility of WR experts (or person deploying the
				system) is to make sure that all the devices have a correct
				configuration. Later we can only generate warnings, if the key
				configuration options are changed remotely (e.g. Grand Master mode to
				Free-running Master or updated fixed hardware delays values).\\
				For misconfigured VLANs, we can monitor if PTP frames are flowing on
				Slave port(s) of the switch.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: When a new configuration file is fetched on
				boot time, compare it with a previously used config (the whole file,
				but especially timing-critical fields like PTP/WR mode, fixed hardware
				delays). Report using the Syslog (\emph{info}/\emph{warning}) if the
				configuration has changed.
		\end{pck_descr}

\subsubsection{\bf Switchover failed}
		\begin{pck_descr}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Boundary Clock}, \emph{Grand Master}
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				In case the primary timing link breaks, switchover is responsible for
				seamless switching to the backup one to keep the device in sync. If WRS
				operates in a \emph{Boundary Clock} mode, switchover is about switching
				between two (or more) WR links to one or multiple WR Masters. If it
				operates in a \emph{Grand Master} mode, it is about broken/lost
				connection to an external reference and switching to a backup WR link
				(another WR Master). Regardless of the configuration, if we fail to
				switch-over to a backup link (e.g. because it is down), WRS resets
				the time counters and continue the operation as a Free-Running Master.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: we should probably use parameters reported by
				the backup channel(s) of the SoftPLL and the backup PTP servo to be able
				to detect and report that something went wrong.
		\end{pck_descr}

\subsubsection{\bf Holdover for too long}
		\begin{pck_descr}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Mode}: \emph{Grand Master}
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				Signaling active holdover is one thing, but if a Grand Master switch is
				kept in holdover for too long, it may drift away from the ideal external
				reference too much. All devices in a WR network will be still
				synchronized, but no longer in sync with the external reference.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
		\end{pck_descr}

\subsubsection{\bf T2/T4 phase transition point was not provided in the configuration}
		\label{fail:timing:t24p}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				Starting from release v6.0 WR switch needs to have T2/T4 phase transition\
				point (T24P) value provided in the configuration file. In case the switch
				is booted with configuration file of the previous release, this parameter
				is not present and can result in wrong correction of reception timestamps.
				This can cause on some switches occasional synchronization problems.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusT24p.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusT24pValid.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPTPStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\newpage
\subsection{Data error}
When the WR switch is not able to forward Ethernet traffic between devices
connected to the ports, we consider this a data error. This section contains the
list of faults leading to a data error.

\subsubsection{\bf Link down}
		\label{fail:data:link_down}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE  \emph{(to be changed later for switchover)}
			\item [] \underline{Severity}: ERROR (will be WARNING with the
				switch-over)
			\item [] \underline{Description}:\\
				This obviously stops the flow of frames on an Ethernet port and there is
				not much we can do besides reporting an error. Topology redundancy is a
				cure for that (if a backup link is fine, and reconfiguration does not
				fail). There might be several causes of a link down:
				\begin{itemize}
					\item unplugged fiber
					\item broken fiber
					\item broken SFP
					\item wrong (non-complementary) pair of WDM SPFs used
				\end{itemize}
				However, we are not able to distinguish between them inside the switch.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{IF-MIB::ifOperStatus.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusLink.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsSlaveLinksStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsTimingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Fault in the Endpoint's transmission/reception path}
		\label{fail:data:ep_txrx}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				This fault covers various errors reported by the Endpoint, e.g. FIFO
				underrun in the Tx PCS or FIFO overrun in the Rx PCS, receiving invalid
				\emph{8b10b} code, CRC error etc.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCTXUnderrun.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXOverrun.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXInvalidCode.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXSyncLost.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPfilterDropped.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPCSErrors.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXCRCErrors.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsEndpointStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsNetworkingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Problem with the SwCore or Endpoint HDL module}
		\label{fail:data:swcore_hang}
		\begin{pck_descr}
		\item [] \underline{Status}: TODO (add monitoring of the Endpoint hangs, depend on
			HDL)
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If the SwCore is hanging, then the Ethernet forwarding is not
				performed on one or multiple ports. We have a HDL watchdog module which
				constantly monitors if the SwCore is not stuck. If such a situation is
				detected the whole SwCore is reset, all the frames queued in the
				Endpoints are acknowledged and lost. After this the switch can continue
				its operation and the watchdog triggers counter is incremented.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsGwWatchdogTimeouts}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCTXFrames.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCForwarded.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsSwcoreStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsNetworkingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
			\item [] \underline{Note}: For Endpoint monitoring we could compare
				per-port \emph{RTUfwd} counter with the \emph{Tx} Endpoint counter for
				each port. \emph{RTUfwd} counts all forwarding decisions from RTU to the
				port $<$n$>$ (excluding PTP frames from NIC). If the sum of this number
				and RTU decisions generated from NIC is equal to the number of frames
				actually transmitted by the Endpoint, then everything works fine.
		\end{pck_descr}

\subsubsection{\bf RTU is full and cannot accept more requests}
		\label{fail:data:rtu_full}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If RTU is full for a given port, it's not able to accept more requests
				and generate new responses. In such case frames are dropped in the
				Rx path of the Endpoint.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXDropRTUFull.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsRTUStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsNetworkingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Too much HP traffic / Per-priority queue full}
		\label{fail:data:too_much_HP}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO \emph{(depends on HDL)}
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If we get too much High Priority traffic, then SwCore will be busy all
				the time forwarding HP frames. This way regular/best effort traffic
				won't be flowing through the switch. In the extreme case, HP traffic
				queue may become full and we start losing HP frames, which is
				unacceptable.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCFastMatchPriority.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXFrames.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio0.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio1.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio2.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio3.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio4.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio5.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio6.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsPstatsHCRXPrio7.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsSwcoreStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsNetworkingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
			\item [] \underline{Note}: we need to get from SwCore the information
				about per-priority queue utilization, or at least an event when it's
				full.
		\end{pck_descr}

\subsubsection{\bf \emph{RTUd} has crashed}
		\label{fail:data:rtu_crash}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				If \emph{RTUd} crashed, traffic would be still routed between the WRS ports, but
				only based on the already existing static and dynamic rules. There would be
				no learning or aging functionality. This means, MAC addresses wouldn't be
				removed from the RTU table if a device is disconnected from a port.
				Without learning, each frame with yet unknown destination MAC would be
				broadcast to all ports (within a VLAN). When \emph{RTUd} crashes,
				\texttt{Monit} will restart it.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsStartCntRTUd}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\snmpadd{HOST-RESOURCES-MIB::hrSWRunName.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Network loop - two or more identical MACs on two or more ports}
		\label{fail:data:net_loop}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				In such case we have a ping-pong situation. If two ports receive frames
				with the same source MAC, it is learned on one of these ports. Then if
				it comes on a second port, it is learned on a second port, and removed
				from the first one. Later, MAC is learned again on the first port, and
				removed from the MAC table for the second port, and so on. This
				situation is a network configuration problem or eRSTP failure.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: we need to monitor the \emph{rtu\_stat} to
				detect ping-pong in the RTU table.
		\end{pck_descr}

\subsubsection{\bf Wrong configuration applied (e.g. wrong VLAN config)}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				The same problem as described in the timing fault
				\ref{fail:timing:no_frames}
		\end{pck_descr}

\subsubsection{\bf Topology Redundancy failure}
		\begin{pck_descr}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				Topology redundancy lets us prevent from losing data when the primary
				uplink is down for some reason. However, if a backup link is also down
				or if the reconfiguration to backup link fails, we start losing data and
				an alarm should be raised.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: One thing we need to report is a backup link(s)
				going down, but we should also think about how to determine if there is
				some problem with eRSTP and if it may fail/has failed if the primary
				link is down.
		\end{pck_descr}

\newpage
\subsection{Other errors}
\label{sec:other_fail}

\subsubsection{\bf WR Switch did not boot correctly}
		\label{fail:other:boot}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO (add rebooting system when boot is
				 not successful, add stop restarting system after defined number of restarts)
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				Every time the switch boots, we verify that all the services have
				started and are running correctly. If any of them fails, an alarm is
				raised.

				The SNMP object \texttt{wrsBootSuccessful} says if a WRS has booted
				correctly, FPGA is programmed, all kernel drivers are loaded and all
				daemons are up and running. If it's not the case, we report what went
				wrong:
				\begin{itemize}
					\item status of reading HW information from dataflash
					\item status of programming FPGA and LM32
					\item status of loading kernel modules
					\item status of starting userspace daemons
					\item status of execution of a custom boot script
					\item status of setting up auxclk on connector clk2
					\item status of setting up a limit on the Rx bandwidth of the traffic that
					      goes from WR ports to Linux (throttling)
					\item status of setting up VLANs
				\end{itemize}
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsRestartReason}\\
				\snmpadd{WR-SWITCH-MIB::wrsRestartReasonMonit}\\
				\snmpadd{WR-SWITCH-MIB::wrsConfigSource}\\
				\snmpadd{WR-SWITCH-MIB::wrsConfigSourceUrl}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootHwinfoReadout}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootLoadFPGA}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootLoadLM32}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootKernelModulesMissing}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\snmpadd{WR-SWITCH-MIB::wrsFwUpdateStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsCustomBootScriptSource}\\
				\snmpadd{WR-SWITCH-MIB::wrsCustomBootScriptSourceUrl}\\
				\snmpadd{WR-SWITCH-MIB::wrsCustomBootScriptStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsAuxClkSetStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsThrottlingSetStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsVlansSetStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful} \\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
			\item [] \underline{Note}: 
				The idea is to reboot the system if it was not able to boot correctly.
				Then we use the scratchpad registers of the processor to keep
				the boot count. If the value of this counter is more than X we stop
				rebooting and try to have a system running with at least \emph{sshd}
				for SSH and \emph{net-snmp} to allow remote diagnostics. If on the other
				hand the switch has booted correctly, we set the boot count to 0.
		\end{pck_descr}

\subsubsection{\bf Dot-config error}
		\label{fail:other:dot-config}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				A dot-config file used to configure the switch can be stored locally or
				retrieved from a central server. Additionally a URL to the remote
				dot-config can be retrieved via DHCP request. When the dot-config is
				fetched from the server it has to be verified before being applied. If
				downloading or verification has failed, an alarm is raised.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsConfigSource} \\
				\snmpadd{WR-SWITCH-MIB::wrsConfigSourceUrl} \\
				\snmpadd{WR-SWITCH-MIB::wrsBootConfigStatus} \\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful} \\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Any userspace daemon has crashed/restarted}
		\label{fail:other:daemon_crash}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO \emph{(depends on monit)}
			\item [] \underline{Severity}: ERROR / WARNING (depending on the process)
			\item [] \underline{Description}:\\
				Running processes are monitored by \texttt{Monit}. When any of them
				crashes, \texttt{Monit} restarts a missing process and increments a
				corresponding start counter. If a process is restarted 5 times within
				100 seconds, then the entire switch is restarted.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{HOST-RESOURCES-MIB::hrSWRunName.<n>} \\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntHAL}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntPTP}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntRTUd}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntSshd}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntHttpd}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntSnmpd}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntSyslogd}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntWrsWatchdog}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntLldpd}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntSPLL}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntLdap}\\
				\snmpadd{WR-SWITCH-MIB::wrsStartCntRvlan}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful} \\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
			\item [] \underline{Note}: We shall distinguish between crucial
				processes - error should be reported if one of them crashes; and less
				important processes (warning should be reported if they crash). If any
				of the processes has crashed, we need to restart it and increment a
				per-process counter reported through the SNMP. Dot-config should also
				let us define which processes are not that important and the switch
				should not restart even if such a process fails to start (e.g.
				\emph{lighttpd}).

				Crucial processes (Error report if any of them crashes):
				\begin{itemize}
					\item \emph{PTP/PPSi}
					\item \emph{wrsw\_rtud} -- after adding configuration preserving code
						on restart, RTUd could be crossed out from this list
					\item \emph{wrsw\_hal}
				\end{itemize}
				Less critical processes (Restarting them and Warning generation is
				enough):
				\begin{itemize}
					\item \emph{sshd}
					\item \emph{udhcpc}
					\item \emph{rsyslogd}
					\item \emph{snmpd}
					\item \emph{lighttpd}
					\item \emph{lldpd}
					\item \emph{nslcd} (LDAP)
					\item \emph{TRUd/eRSTPd} -- not yet implemented
				\end{itemize}

				\emph{wrsw\_rtud} -- we need to set the flag informing the process has
				crashed so that when it runs again it knows that HDL is already
				configured. It should not erase static entries in the RTU table (e.g.
				multicasts for PTP), the static entries set by-hand as well as VLANs.
				Dynamic entries are not a problem. RTUd can learn all MACs after
				restarting. The only consequence will be increased network traffic due
				to frames broadcast until all the MACs are learned. In general,
				the source code has to be checked to make sure what is cleared on the
				startup and modified to preserve the configuration.\\

				\emph{TRUd/eRSTPd} -- topology reconfiguration is done in hardware if
				needed, the daemon is used only to configure the TRU/RTU HDL module.
				However, the story is similar as with the RTUd. If eRSTPd crashes, we
				need to store this information so that when it runs again, it does not
				erase the whole configuration. Also if a topology reconfiguration
				happens while eRSTPd is down, HDL should keep the flag for the eRSTPd so
				that it's aware the backup link is active.
		\end{pck_descr}

\subsubsection{\bf Kernel crash}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO (preserving stats of IP/LR registers)
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If the Linux kernel has crashed, the system reboots. Until the next boot
				we have no synchronization, no SNMP to report the status, and the FPGA
				may be still forwarding Ethernet traffic, but based on dynamic and
				static routing rules from before the crash. Based on the SNMP objects
				below it is possible to figure out that reboot took place and what was
				the reason of the last reboot.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsBootCnt}\\
				\snmpadd{WR-SWITCH-MIB::wrsRebootCnt}\\
				\snmpadd{WR-SWITCH-MIB::wrsRestartReason}\\
				\snmpadd{WR-SWITCH-MIB::wrsFaultIP}\\
				\snmpadd{WR-SWITCH-MIB::wrsFaultLR}\\
				\snmpadd{WR-SWITCH-MIB::wrsBootSuccessful}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
			\item [] \underline{Note}:
				Unfortunately, right now it is not possible to distinguish whether the
				reboot was caused by the kernel panic function or the \texttt{reboot}
				command. Preserving the state of IP and LR registers has to be
				implemented.
		\end{pck_descr}
\subsubsection{\bf System nearly out of memory}
		\label{fail:other:no_mem}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				We need to monitor the amount of free memory, report it through SNMP and
				raise an alarm if it's extremely low (but still enough to keep the
				system running).
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsMemoryTotal}\\
				\snmpadd{WR-SWITCH-MIB::wrsMemoryUsed}\\
				\snmpadd{WR-SWITCH-MIB::wrsMemoryUsedPerc}\\
				\snmpadd{WR-SWITCH-MIB::wrsMemoryFree}\\
				\snmpadd{WR-SWITCH-MIB::wrsMemoryFreeLow}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}
\subsubsection{\bf Disk space low}
		\label{fail:other:no_disk}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				We need to monitor the amount of free disk space, report it through SNMP
				and raise an alarm if it's extremely low (but still enough to keep the
				system running).
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsDiskMountPath.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsDiskSize.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsDiskUsed.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsDiskFree.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsDiskUseRate.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsDiskFilesystem.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsDiskSpaceLow}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus}\\
				\snmpadd{HOST-RESOURCES-MIB::hrStorageDescr.<n>}\\
				\snmpadd{HOST-RESOURCES-MIB::hrStorageSize.<n>}\\
				\snmpadd{HOST-RESOURCES-MIB::hrStorageUsed.<n>} }
			\item [] \underline{Note}:
				Objects like \texttt{HOST-RESOURCES-MIB::hrStorage*.<n>} are available
				via standard MIB. The same functionality is implemented in
				\texttt{WR-SWITCH-MIB} objects \texttt{wrsDisk*.<n>} (to ease the
				implementation of \texttt{wrsDiskSpaceLow}).
		\end{pck_descr}

\subsubsection{\bf CPU load too high}
		\label{fail:other:cpu}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				On a healthy switch the average CPU load should be below \emph{0.1} (10\%).
				Some actions like SNMP queries or web interface activity may increase
				the average system load. The system load averages for the past 1, 5 and
				15 minutes are exported via SNMP objects. Additionally
				\texttt{wrsCpuLoadHigh} alerts when the load is too high.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsCPULoadAvg1min}\\
				\snmpadd{WR-SWITCH-MIB::wrsCPULoadAvg5min}\\
				\snmpadd{WR-SWITCH-MIB::wrsCPULoadAvg15min}\\
				\snmpadd{WR-SWITCH-MIB::wrsCpuLoadHigh}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Temperature inside the box too high}
		\label{fail:other:temp}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				If the temperature raises too high we might break our electronics inside
				the box. It also means that most probably one or both of the fans inside
				the box are broken and should be replaced. There are 4 temperature
				sensors monitored:
				\begin{itemize}
					\item \emph{IC19} -- temperature below the FPGA
					\item \emph{IC20}, \emph{IC17} -- temperature near the SCB power supply
						circuit
					\item \emph{IC18} -- temperature near the VCXO and PLLs (AD9516,
						CDCM6100)
				\end{itemize}
				\texttt{wrsTemperatureWarning} is raised when the temperature read from
				any of these sensors exceeds a threshold configured in the
				\emph{dot-config} (80 degrees by default). When at least one threshold
				temperature is not set \texttt{wrsTemperatureWarning} is set to
				\emph{Threshold-not-set}.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsTempFPGA}\\
				\snmpadd{WR-SWITCH-MIB::wrsTempPLL}\\
				\snmpadd{WR-SWITCH-MIB::wrsTempPSL}\\
				\snmpadd{WR-SWITCH-MIB::wrsTempPSR}\\
				\snmpadd{WR-SWITCH-MIB::wrsTempThresholdFPGA}\\
				\snmpadd{WR-SWITCH-MIB::wrsTempThresholdPLL}\\
				\snmpadd{WR-SWITCH-MIB::wrsTempThresholdPSL}\\
				\snmpadd{WR-SWITCH-MIB::wrsTempThresholdPSR}\\
				\snmpadd{WR-SWITCH-MIB::wrsTemperatureWarning}\\
				\snmpadd{WR-SWITCH-MIB::wrsOSStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf Not supported SFP plugged into the cage (especially non 1-Gb SFP)}
		\label{fail:other:sfp}
		\begin{pck_descr}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				If a not supported Gigabit optical SFP (or an SFP that couldn't have
				been matched with the \texttt{CONFIG\_SFP<XX>\_PARAMS} entries in the
				configuration file) is plugged into the cage, then it's a timing issue
				\ref{fail:timing:wrong_sfp}. However, if a non 1-Gb
				SFP is used, then no Ethernet traffic would be flowing on that port.
				It's due to the fact, that we don't have 10/100Mbit Ethernet implemented
				inside the WRS.
			\item [] \underline{SNMP objects}:\\
				{\footnotesize
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpVN.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpPN.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpVS.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpGbE.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsPortStatusSfpError.<n>}\\
				\snmpadd{WR-SWITCH-MIB::wrsSFPsStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsNetworkingStatus}\\
				\snmpadd{WR-SWITCH-MIB::wrsMainSystemStatus} }
		\end{pck_descr}

\subsubsection{\bf IP address on the management port has changed}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				The change of an IP address on the management port might be a normal
				situation or a result of an accidental modification of a DHCP server or
				the WR Switch configuration. Notifying about such a situation is not
				done through SNMP, since the IP address of a switch has to be known to
				the SNMP manager prior querying the switch. Therefore, the switch only
				generates a Syslog warning message if setting a new IP address is
				detected.
			\item [] \underline{SNMP objects}: \emph{(none)}, Syslog message is
				generated
		\end{pck_descr}

\subsubsection{\bf Multiple unauthorized access attempts}
		\begin{pck_descr}
			\item [] \underline{Status}: TODO
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				Many attempts to gain a root access through the ssh (or the web
				interface), might mean that somebody tries to do something nasty. Every
				unsuccessful attempt to login is reported as a Syslog warning message.
			\item [] \underline{SNMP objects}: \emph{(none)}, Syslog message is
				generated
		\end{pck_descr}

\subsubsection{\bf Network reconfiguration (RSTP)}
		\label{fail:other:rstp}
		\begin{pck_descr}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				If topology reconfiguration occurs because of the primary link failure,
				this fact should be reported through SNMP as a warning. It's not
				critical situation, WR network still works. However, further
				investigation should be performed to repair the broken link.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
		\end{pck_descr}

\subsubsection{\bf Backup link down}
		\begin{pck_descr}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				This is related to the issue \ref{fail:other:rstp}. If the WRS uses
				primary uplink, but the backup one fails, it's not a critical fault. WR
				Network still works, but the link should be diagnosed and repaired to
				have the backup link operational in case the primary one fails.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
		\end{pck_descr}

\newpage
\subsection{Undetectable errors}

Beside the various errors already listed in previous sections, there are some
situations when reporting a problem to the SNMP manager or Syslog server is not
possible. This section lists some of them and proposes alternative ways of
diagnostics.

\subsubsection{\bf File system / Memory corruption}
		\label{fail:other:memory}
		\begin{pck_descr}
			\item [] \underline{Description}:\\
				Memory or file system corruption can produce unpredictable results. It
				may cause a failure of any of the processes running on the switch.
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{pck_descr}

\subsubsection{\bf Kernel freeze}
		\begin{pck_descr}
			\item [] \underline{Description}:\\
				If the Linux kernel freezes there is nothing that can be done. It can
				freeze e.g. due to some infinite loop in the irq handler. It is similar
				to the power failure, somebody has to go to the place where the WRS is
				installed and investigate/restart the device.
			\item [] \underline{SNMP objects}: \emph{(none)}
			\item [] \underline{Note}:
				If we have watchdog in our CPU it should be used.
		\end{pck_descr}

\subsubsection{\bf Power failure}
		\begin{pck_descr}
			\item [] \underline{Description}:\\
				Power failure may be either a WRS problem (i.e. broken power supply
				inside the switch) or an external voltage problem. It's up to the
				Network Management Station to raise an alarm if the SNMP Agent does
				not respond to the SNMP requests.
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{pck_descr}

\subsubsection{\bf Hardware problem}
		\begin{pck_descr}
			\item [] \underline{Description}:\\
				If any crucial hardware part breaks, it will be most probably noticed
				as one (or multiple) timing / data errors described in the previous
				sections. Besides that, there is no self-diagnostics built-in on the
				switch hardware boards. A few examples of hardware failures and problems
				it may cause:
				\begin{itemize}
					\item DAC / VCO -- problems with synchronization (failures in
						\ref{sec:timing_fail})
					\item cooling fans -- rise of the temperature inside the WRS box
						(failure \ref{fail:other:temp})
					\item power supply, ARM, FPGA -- booting problem (failure
						\ref{fail:other:boot})
					\item memory chip -- data corruption (failure \ref{fail:other:memory})
				\end{itemize}
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{pck_descr}

\subsubsection{\bf Management link down}
		\label{fail:other:management_link}
		\begin{pck_descr}
			\item [] \underline{Description}:\\
				For obvious reasons we are not able to report through SNMP that the
				management link is down. This should be detected and reported by the NMS
				if it does not receive SNMP and ICMP responses from the WRS.
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{pck_descr}

\subsubsection{\bf No static IP on the management port \& failed to DHCP}
		\begin{pck_descr}
			\item [] \underline{Description}:\\
				From the operator's point of view it is similar to the issue
				\ref{fail:other:management_link}. WRS is not accessible through the
				management port, so its status cannot be reported. This should be
				detected and reported by the NMS if it does not receive SNMP and ICMP
				responses from the WRS. In such case the configuration of the switch and
				management network should be verified.
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{pck_descr}

%\subsection{Switch out of sync to Master}
%
%\subsection{Switch made a big offset jump to follow Master}
%
%\subsection{Unsupported SFP plugged to one of the cages}
%
%\subsection{Lost lock to external 1-PPS \& 10 MHz}
%
%\subsection{Switch wasn't able to fetch initial time from NTP}
%
%\subsection{Suspicious value of any PTP parameter}
%e.g. bitslide > 16000;  dTx/dRx = 0, etc.
%
%\subsection{PPSi/HAL/SNMP/any other userspace daemon has crashed}
%
%\subsection{LM32 software has crashed/restarted}
%
%\subsection{Cooling fan broken}
%
%\subsection{Power supply broken}
%
%\subsection{Switch not reachable after power cut}
%
%\subsection{Switch not reachable through SNMP}
%
%\subsection{One of the links went down}
%
%\subsection{Ethernet frames being dumped}
%
%\subsection{Linux is out of memory}
%
%\subsection{Filesystem error/corruption}
%
%\subsection{HW version not recognized, FPGA bitstream not loaded}
%
%\subsection{Frames storm coming from one or multiple ports to CPU}
