\input texinfo    @c -*-texinfo-*-
@c 
@c   - main file for the documentation
@c 
@c %%%

@c ------------------------------------------------------------------------------
@c 
@c                          NOTE FOR THE UNAWARE USER
@c                          =========================
@c 
@c     This file is a texinfo source. It isn't the binary file of some strange
@c     editor of mine. If you want ASCII, you should "make cortex-m3.txt".
@c 
@c ------------------------------------------------------------------------------

@c 
@c  This is not a conventional info file...
@c  I use three extra features:
@c    - The '%' as a comment marker, if at beginning of line ("\%" -> "%")
@c    - leading blanks are allowed (this is something I can't live without)
@c    - braces are automatically escaped when they appear in example blocks
@c 


@comment %**start of header
@documentlanguage en
@documentencoding ISO-8859-1
@setfilename cortex-m3.info
@settitle wrs-test
@iftex
@afourpaper
@end iftex
@paragraphindent none
@comment %**end of header

@setchapternewpage off

@set update-month July 2011

@finalout

@titlepage

@center @titlefont{Benchmarking and Compliance Methodology}
@title for White Rabbit Switches
@subtitle @value{update-month}

@author Cesar Prados (@code{c.pradosi@@gsi.de})
@author Maciej Lipinski (@code{maciej.lipinski@@cern.ch})
@end titlepage
@headings single

@contents

@c ##########################################################################
@node Top, Benchmarking Tests, (dir), (dir)
@top Introduction

This document describes the methodology for testing White Rabbit Switches. Part
of the tests follows the RFC 2544 and 2889, others are specific for Whiter
Rabbit or specific protocols. In turn the methodology is divided in Benchmarking and
Compliance Test.


The goal of this test methodology is threefold:

@itemize
@item  Benchmark the White Rabbit switch
@item  Ascertain the compliance of the protocols implemented in the  WR Switch
@item  Ascertain the WR protocol compliance of  others manufacturers 
@end itemize 


@c ##########################################################################

@node Benchmarking Tests, Top
@chapter Benchmarking Tests

To asses the performance of the WR Switch regarding:

@c----------------------------------------------------------------------------
@section  Throughput
@code  RFC 2544 Section 26.1

To determine the throughput.

@strong{Test Parameters }

@multitable @columnfractions  .20 .60
@item Duration:  @tab > 60 secs
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Traffic Direction:  @tab  bi, uni-directional, input-output and vice versa.
@item Frame Rate:  @tab  100 % of the theoretical maximum.
@item Trials:  @tab  10 for every frame length and CoS
@item Burst size:  @tab  ?
@end multitable


@strong{Test Output}

@multitable @columnfractions  .20 .30 .20 
@item Frames  Length   @tab   Theoretical Max Rate  @tab Throughput
@item 64   @tab   ....  @tab ... 
@item ...   @tab   ....  @tab .... 
@item 1518   @tab   ....  @tab ...  
@end multitable

  @c----------------------------------------------------------------------------
@section    Frame Latency 
@code  RFC 2544 Section 26.2
 
   To determine the latency of a frame traversing the DUT.

@strong{Test Parameters}

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Traffic Direction:  @tab  bi, uni-directional, input-output and vice versa.
@item Trials:  @tab  10 for every frame length and CoS
@item Burst size:  @tab  ?
@end multitable

@strong{Test Output}

@multitable @columnfractions  .20 .20 .20 .20 
@item Frames  Length   @tab   Frame Rate  @tab Latency  @tab CoS
@item A   @tab   ....  @tab N  @tab 7
@item ...   @tab   ....  @tab ....  @tab ...
@item A   @tab   ....  @tab N  @tab 1
@end multitable

  @c----------------------------------------------------------------------------
@section  Broadcast and 7th CoS  Frame Forwarding and Latency  
To determine the throughput and latency of the DUT when forwarding 7th CoS  and 
broadcast traffic a.k.a  "High Priority Traffic".

@strong{Test Parameters}

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Traffic Direction:  @tab  bi, uni-directional, input-output and vice versa.
@item Trials:  @tab  10 for every frame length and CoS
@item  Load per port: @tab ? (%) 
@item Burst size:  @tab  ?
@end multitable

@strong{Test Output}

@multitable @columnfractions  .20 .20 .20 .20 
@item Frames  Length   @tab   Load  @tab Throughput  @tab Latency
@item 64     @tab   ....  @tab ...  @tab ...
@item ...     @tab   ....  @tab ...  @tab ...
@item 1518 @tab   ....  @tab ...  @tab ...
@end multitable
 
* Throughput,  maximum load with no frame loss
* Average latency.

@c------------------------------------------------------------------------------
@section  Frame Loss Rate 
@code  RFC 2544 Section 26.3

To determine the frame loss rate of the DUT over various loads, frame lengths
and CoS.

@strong{Test Parameters }
    
@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Frame Rate:  @tab 100 % of the theoretical maximum.
@item Traffic Direction:  @tab  bi, uni-directional, input-output and vice versa.
@item Trials:  @tab  10 for every frame length and CoS
@item Burst size:  @tab  ?
@end multitable

@strong{Test Output}

@multitable @columnfractions  .20 .20 .20
@item Frame Rate   @tab   Frame Loss Rate @tab Frame Length
@item ...   @tab   ....  @tab   64 
@item ...   @tab   ....  @tab ....  
@item ..   @tab   ....   @tab 1518
@end multitable

@c----------------------------------------------------------------------------

@section Back-to-Back frames
@code  RFC 2544 Section 26.3

    To determine the largest  burst of frames with minimum  interframe gap
    (back-to-back frames) the DUT can handle with zero loss.


@strong{Test Parameters }

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item  Back-to-back value:  @tab from min - max ?
@item Traffic Direction:  @tab  bi, uni-directional, input-output and vice versa.
@item Trials:  @tab  10 for every frame length and CoS
@item Burst size:  @tab  ?
@end multitable


@strong{Test Output}

@multitable @columnfractions  .20 .20 .30
@item Frame Length   @tab  Frame Rate @tab Recovery Time
@item 64   @tab   ....  @tab   ....  
@item ...   @tab   ....  @tab   ....  
@item 1518   @tab   .... @tab   ....    
@end multitable

@c----------------------------------------------------------------------------
 
@section  System recovery
@code  RFC 2544 Section 26.4

To characterize the speed at which a DUT recovers from an overload condition.

@strong{Test Parameters }

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Traffic Direction:  @tab  bi, uni-directional, input-output and vice versa.
@item Trials:  @tab  10 for every frame length and CoS
@end multitable


@strong{Test Output}

@multitable @columnfractions  .20 .20 .30
@item Frame Length   @tab Frame Rate @tab  Recovery Time
@item 64   @tab   ....  @tab   ....  
@item ...   @tab   ....  @tab   ....  
@item 1518   @tab   .... @tab   ....    
@end multitable


@c----------------------------------------------------------------------------
@c----------------------------------------------------------------------------

@section  Fully Meshed 
@code  RFC 2889 Section 5.1

    To determine the throughput, frame loss and forwarding rates of the DUT offered fully meshed traffic.
    
@strong{Test Parameters }

@multitable @columnfractions  .20 .60
@item Duration:  @tab 30 secs 
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Load per port: @tab ? (%)
@item Trials:  @tab  5 for every frame length and CoS
@item Burst size:  @tab  from 1 to 1000 frames
@end multitable 

@strong{Output}
@itemize
@item Forwarding rate for each frame legnth, load and  CoS
@item Throughput for each frame size with no frame loss.
@item  Flood count
@end itemize
@c----------------------------------------------------------------------------
@section Partially Meshed
@subsection One to Many/Many to One
@code  RFC 2889 Section 5.2

To determine the throughput when transmitting from one-to-many ports or from 
many-to-one port. To measure the capability to switch frames without frame loss 
and determine the ability to utilize a port when switching traffic from multiple 
ports.

@strong{Test Parameters }

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Duration:  @tab 30 secs 
@item Traffic Direction:  @tab one port to many ports, many ports to one port.
@item Load per port: @tab ? (%)
@item Trials:  @tab  5 for every frame length and CoS
@item Burst size:  @tab  from 1 to 1000 frames
@end multitable 

@strong{Test Output}

@itemize
@item  Forwarding rate for each frame size, for each load and CoS.
@item  Flood count
@end itemize
@c----------------------------------------------------------------------------

@subsection Multiple Devices
@code  RFC 2889 Section 5.3

To determine the throughput, frame loss and forwarding rates of two WR Switches 
connected in Back-Bone Configuration.

@strong{Test Parameters }

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Duration:  @tab 30 secs 
@item Traffic Direction:  @tab one port to many ports, many ports to one port.
@item Load per port: @tab ? (%)
@item Trials:  @tab  5 for every frame length and CoS
@item Burst size:  @tab  from 1 to 1000 frames
@end multitable 

@strong{Test Output}

@itemize
@item  Forwarding rate for each frame size, for each load and CoS.
@item  Flood count
@end itemize

@c----------------------------------------------------------------------------

@subsection  Unidirectional Traffic
@code  RFC 2889 Section 5.3

To determine the throughput of the DUT when multiple streams of one-way traffic 
using half of the ports on the DUT are sending frames to the other half of the ports.

@strong{Test Parameters }

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Duration:  @tab 30 secs 
@item Trials:  @tab  5 for every frame length and CoS
@end multitable 

@strong{Test Output}

@itemize
@item  Forwarding rate for each frame size, for each load and CoS.
@item  Flood count
@end itemize


@c----------------------------------------------------------------------------
@subsection Maximum Forward Rate
@code  RFC 2889 Section 5.4

To determine the throughput of the DUT when multiple streams of one-way traffic 
using half of the ports on the DUT  are sending frames to the other half of the 
ports. 

@multitable @columnfractions  .20 .60
@item Length of Frames:  @tab  Recommended  Ethernet frame sizes
@item Duration:  @tab 30 secs 
@item Load per port: @tab ? (%)
@item Trials:  @tab  5 for every frame length and CoS
@item Burst size:  @tab  from 1 to 1000 frames
@end multitable 



@c----------------------------------------------------------------------------

@subsection  RTU Capacity
@code  RFC 2889 Section 5.5

To determine the address caching capacity of the WR Switch.

@strong{Test Parameters}

@multitable @columnfractions  .20 .60
@item RTU Age time: @tab  ?
@item Address learning rate :  @tab ?
@item Trials:  @tab  5 for every frame length and CoS
@end multitable 

@strong{Test Output}

@itemize
@item  RTU address caching capacity 
@item  Flood count
@end itemize

@c----------------------------------------------------------------------------

@subsection  Learning Rate
@code  RFC 2889 Section 5.6

To determine the rate of address learning of the White Rabbit Switch.

@strong{Test Parameters}

@multitable @columnfractions  .20 .60
@item RTU Age time: @tab  ?
@item Address learning rate :  @tab ?
@item Trials:  @tab  5 for every frame length and CoS
@end multitable 

@strong{Test Output}

@itemize
@item  RTU address learning rate using search algorithm.
@item  Flood count
@end itemize

@c----------------------------------------------------------------------------
@section (Multiple)  Rapid Spanning Tree Protocol
@strong{NOTE:}
Part of the tests can be done using a WR Test Switch connected to DUT, who 
receives from the Test Switch BPDUs and at same point gives up simulating a 
failure. By doing this we can simulate the failure of port with a designate role.

@subsection Time of coverage 
To determine the time of coverage of RSTP in the network when:

@itemize 
@item new switch added to the network
@item change of topology
@item failure of a root, designated, alternate and buckup  port.
@end itemize

@strong{Test Parameters}

@multitable @columnfractions  .20 .60
@item BPDU frames rate: @tab  ?
@item BPDU missing time :  @tab ?
@item Port Role: @tab ?
@item Type of traffic: @tab "High Priority" or not
@end multitable 

@strong{Test Output}

@itemize 
@item Time of convergence for the different roles of a designate port. 
@item Number of lost frames
@end itemize

@subsection Switch-over for "High Priority" Traffic 

to determine the time of Switch-over in case of port failure. Ref "High Priority
Traffic in RSTP" in the document "White Rabbit and Robustness".

@c----------------------------------------------------------------------------
@page

@section White Rabbit PTP (WRPTP)


@strong{NOTE:} Measurements are done by comparing the PPS output signal of a switch which is
chosen a Timing Master (reference source of time) and PPS outputs of the other switches  
in the network.
 
@subsection Synchronization Quality of a Static Network 

To determine the quality of the synchronization (accuracy, precision and stability) in a static 
network (no changes to the network configuration and no switching between redundant devices) under 
varying traffic load and varying external conditions.

@strong{Test Parameters}

@multitable @columnfractions  .40 .60
@item Duration:                   @tab 24h
@item Temperature range:          @tab 0-30 Celsius degrees
@item Temperature change rate:    @tab 1 Celsius degrees per minute
@item Number of Timing Masters:   @tab 1
@item Traffic load                @tab variable (0-100%)
@item Number of cascaded switches @tab 0-4
@end multitable 


@strong{Test Output}
@itemize 
@item Accuracy
@item Precision
@item Stability
@end itemize

@subsection Reliability of Network Synchronization

To determine the quality of the synchronization (accuracy,precision and stability) in a 
network which is subject to changes due to:
@itemize 
@item network re-configuration : adding/removing redundant switches and connections 
      (fiber/copper cables),
@item failure of redundant elements (connections, master ports, switches): switch-overload,
@item failure of non-redundant elements: hold-over.
@end itemize

@strong{Test Parameters}

@multitable @columnfractions  .40 .60
@item Duration:                   @tab  24h
@item Temperature range:          @tab 0-30 Celsius degrees
@item Temperature change rate:    @tab 1 Celsius degrees per minute
@item Number of Timing Masters:   @tab 1 and more
@item Traffic load                @tab variable (0-100%)
@item Number of cascaded switches @tab 0-4
@end multitable 


@strong{Test Output}
@itemize 
@item Accuracy 
@item Precision
@item Stability
@end itemize

@page

@c ##########################################################################

@node Compliance Tests, Top
@chapter Compliance Tests 

@section Standards 


@subsection  RSTP / MSTP

TBD

@subsection  VLAN

TBD

@subsection PTP/WRPTP

TBD

@subsection Sync Ethernet ??

TBD

@section RFC 2889

@subsubsection Congestion Control Flow Control

TBD

@subsubsection  Frame Filtering

TBD

@c ##########################################################################

@setchapternewpage on

@node Terminology, Top
@unnumbered Terminology

@strong{Recommended Ethernet Frames Sizes:} The RFC recommends for the tests
frame sizes of: 64, 128, 256, 512, 1024,1280 and 1518 bytes


@strong{DUT:} Device Under Test


@strong{Back-to-Back:} Fixed length frames presented at a rate where the
minimum legal separation for a given medium between frames over a short to
medium period of time, starting from an idle state. (RFC 1242)

@strong{Latency:}For store and forward devices: The time interval
starting when the last bit of the input frame reaches the input port and 
ending when the first bit of the output frame is seen on the output port. 
For bit forwarding devices: The time interval starting when the end of the 
first bit of the input frame  reaches the input port and ending when the start 
of the first bit of the output frame is seen on the output port. (RFC 1242)

@strong{Throughput:}The maximum rate at which none of the offered frames are 
dropped by the device. (RFC 1242)


@strong{Accuracy:} Mean offset between the source of time (Timing Master) and the DUT.

@strong{Precision:} Standard deviation of the offset between the source of time 
(Timing Master) and the DUT.

@strong{Stability:} 

@strong{PPS:} Pulse per Second.


 





@c ##########################################################################



@node Bibliography, Top
@unnumbered Bibliography

@itemize  
@item RFC1242
@item RFC2285
@item RFC2544
@item RFC2889
@item Test Methodologies Spirent
@item Calnex Solutions, Application Note Testing 1588v2
@item Agilent, Testing Spanning Tree Protocol 
@item White Rabbit and Robustness 
@item White Rabbit PTP 
@end itemize 


@c ##########################################################################
@bye

