\input texinfo    @c -*-texinfo-*-
%
% wrs-developer-manual.in - main file for the documentation
%
%%%%

%------------------------------------------------------------------------------
%
%                         NOTE FOR THE UNAWARE USER
%                         =========================
%
%    This file is a texinfo source. It isn't the binary file of some strange
%    editor of mine. If you want ASCII, you should "make wrs-developer-manual.txt".
%
%------------------------------------------------------------------------------

%
% This is not a conventional info file...
% I use three extra features:
%   - The '%' as a comment marker, if at beginning of line ("\%" -> "%")
%   - leading blanks are allowed (this is something I cannot live without)
%   - braces are automatically escaped when they appear in example blocks
%

@comment %**start of header
@documentlanguage en
@documentencoding ISO-8859-1
@setfilename wrs-developer-manual.info
@settitle wrs-developer-manual
@iftex
@afourpaper
@end iftex
@paragraphindent none
@comment %**end of header

@setchapternewpage off

@set update-month August 2017
@c the release name below is substituted at build time
@set release __RELEASE_GIT_ID__

@finalout

@titlepage
@title White Rabbit Switch: Developer's Manual
@subtitle Information about software in the White Rabbit switch, for developers and advanced users
@subtitle @value{update-month} (@value{release})
@author Alessandro Rubini, Adam Wujek, Benoit Rat, Federico Vaga, ...
@end titlepage
@headings single

@c ##########################################################################
@iftex
@contents
@end iftex

@c ##########################################################################
@c in texinfo we are mandated to have a Top node
@node Top
@top Introduction

The White Rabbit switch (or @sc{wrs}) is a major component of the
White Rabbit (@sc{wr}) network.  Like any modern managed switch, the
@sc{wrs} includes a CPU with its own operating system.

This manual is for people rebuilding or modifying @sc{wrs} software.
It explains how to rebuild the whole software stack from source
and how is the switch itself designed.

@c ##########################################################################
@node WRS Documentation
@chapter WRS Documentation


Up to and including release 4.0 of @sc{wrs} software this manual
was the only official documentation item; now that the device is mature
and the deployed base increases, we reorganized documentation.

@c ==========================================================================
@node The Official Manuals
@section The Official Manuals

This is the current set of manuals that accompany the @sc{wrs}:

@itemize @bullet

@item @i{White Rabbit Switch: Startup Guide}: hardware installation
   instructions. This manual is provided by the manufacturer: it describes
   handling measures, the external connectors, hardware features and the
   initial bring-up of the device.

@item @i{White Rabbit Switch: User's Manual}: documentation about
   configuring the @sc{wrs}, at software level.  This guide is maintained
   by software developers.  The manual describes
   configuration in a deployed network, either as a standalone device or
   as network-booted equipment.  The guide also describes how to upgrade
   the switch, because we'll release new official firmware images over
   time, as new features are implemented.

@item @i{White Rabbit Switch: Developer's Manual}: it describes the
   build procedure and how internals work; use of scripts and
   @sc{wrs}-specific executables and so on.  The manual is by developers
   and for developers.  This is the
   document to check if you need to customize your @i{wrs} rebuild
   software from new repository commits that are not an official release
   point, or just install your @i{wrs} with custom configuration values.

@item @i{White Rabbit Switch: Failures and Diagnostics}: describe various
   failure scenarios of a switch and ways how to recognize them.
   Additionally describe SNMP exports of a switch (@t{WR-SWITCH-MIB}).

@end itemize

The official @sc{pdf} copy of these manuals at each release
is published in the @i{files} tab of the software project in @t{ohwr.org}:
(@url{http://www.ohwr.org/projects/wr-switch-sw/files}).
This doesn't apply to release 4.0 and earlier.

The source form of all four manuals is maintained in @t{wr-switch-sw/doc}.
Within the repository, three of them the @i{User's Manual},
the @i{Developer's Manual} and the @i{Failures and Diagnostics}
are always tracking the software commits, while the @i{Startup Guide} may not
be authoritative because it is bound to device shipping rather than software
development.

@c ==========================================================================
@node Supported Hardware Versions
@section Supported Hardware Versions

This document applies to versions 3.3 and 3.4 of the @sc{wrs}
device.

Very few specimens of @t{wrs} 3.0 though 3.2 were manufactured; if you
are the owner of one of them, please refer to version 3.3 of the
@i{wrs-build} document, that includes appendixes about using older
versions. As usual, it is in the @i{files} tab of @t{ohwr.org}.

V1 and V2 were development items, never shipped.

@c ##########################################################################
@node Building WRS Software
@chapter Building WRS Software

@c ==========================================================================
@node Overview
@section Overview

To build your switch firmware you have to first clone the main git repository,
which contains all the sources and building scripts:
@example
   git clone git://ohwr.org/white-rabbit/wr-switch-sw.git
@end example

Original build scripts were developed by Tomasz Wlostowski who first
made the whole thing work and stick together.

The purpose of the build-script rewrite is achieving the following targets:

@itemize @bullet
@item One-command build.
	The non-technical user should be able to rebuild the whole software
        package with a single command. This includes the IPL and boot-loader
        even though they are expected to be pre-installed in the switch with
        no real need for upgrading.
@item Sub-package separation.
	Users and developers should be able to rebuild each
        sub-package by itself. Sub-packages are the kernel,
	@i{buildroot}, libraries and so on. If you have a problem (or a
        customization you need on one sub-package), you should be able to work
        on the specific part ignoring the whole as much as possible.
@item Documentation.
	The steps are documented as much as possible, because mishaps
        do happen, and you should easily understand where the problem is.
@item Avoid redundant downloads.
	People with non-mainstream network connections would rather
	avoid downloading the same package over and over. Thus, a centralized
	download directory is defined where all external packages are
        retrieved. Even if you ``make distclean'' in the build scripts you
        will not need re-get everything from the network.  In a similar
        mood, people who already have a local copy of the big packages
        (kernel, barebox, white-rabbit) will not need to re-download not
        even the first time they build the WRS software.
@end itemize

After release 3.3, we decided to add @i{Kconfig} support. This means
that the first build step is expected to be ``@t{make menuconfig}'' (from v5.0
@t{make nconfig} and @t{make xconfig} are supported alternatives),
like it happens for the kernel.  The default configuration is selected
by default when one of the build scripts is run, so the procedure for
the final user is the same as for v3.3 and earlier.

After release 4.1 we support dynamic reconfiguration. As opposed
to build-time configuration, run-time configuration is expected to
be frequent, and it's thus documented in the @i{@sc{wrs} Users' Manual}.

The build system is set up as a mix of scripts and makefiles. Every
sub-package is built by its own script and/or Makefile, and configuration is
passed over through environment variables. The top-level build script
sets all environment variables, while keeping defaults from your
preexisting environment -- so you can override anything even when
rebuilding it all from scratch.

@c --------------------------------------------------------------------------
@node Other Repositories
@subsection Other Repositories

Not everything that runs in the switch comes from this package:
both the @i{gateware} image and the @i{lm32} firmware binary
are built from sources external to this package.

The @i{gateware} is being downloaded as a pre-build tar archive
(currently called @code{wrs-gw-v5.0-20161214.tar.gz}).  This
is built from the @code{wr-switch-sw-v5.0} tag of the @code{wr-switch-hdl}
repository.  Please note that the repository uses @i{git} submodules,
so it depends on other @code{ohwr} repositories too, which in turn
have not been tagged because the submodule mechanism ensures you'll
get the exact version you need. All relevant commit identifiers
are shown by command @t{wrs_version -t} or in the SNMP version objects
(for more information please refer to switch's MIB file @t{WR-SWITCH-MIB.txt}).

The LM32 program is provided as a pre-compiled binary in
@code{binaries/rt_cpu.bin}. The respective source code is the
@i{wrpc-sw} package, because all WR installations run the same
@sc{pll} software code and we chose to avoid duplication. Moreover,
@i{wr-switch-sw} builds do not require an LM32 development environment.

If you need to rebuild the @t{rt_cpu.bin} file from source, to make
your own modifications, you can run @t{make wr_switch_defconfig}
in @i{wrpc-sw} and then @t{make}. Please checkout the
@code{wr-switch-sw-v5.0}
tag to get the exact commit.

@c --------------------------------------------------------------------------
@node Portability
@subsection Portability

The scripts in their current status are not expected to be very
portable. A number of @i{bashisms} exist,
internally the name @i{bash} is used instead of @i{sh}, so things
work in systems where the default shell is @i{dash}, provided @i{bash}
is installed.

Similarly, the scripts are likely to fail if spaces are used in directory
names; that is because not all uses of shell variables are properly quoted.

The expected build environment is a native GNU/Linux system; success reports
about other environments (e.g. cygwin) are welcome, possibly with associated
patches.

@c --------------------------------------------------------------------------
@node Environment Variables
@subsection Environment Variables

The scripts use a number of environment variables; you can pre-set
them as you wish. If they are not pre-set, defaults apply as
described below.

When building running the @i{build/wrs_build-all} script (whether you
build everything or rebuild individual steps) the defaults are applied
for each unset variable.  Developers working under the hood will need
to set the variables.  Each sub-package complains if it needs
variables that are not set in their environment.

The following variables are used in one or more parts of the scripts;
let me restate, though, that sensible default values apply by default,
so this list is mainly for your curiosity unless you are a developer.

They are listed in an order that seems logical to me, but may sound
random to a different person, please forgive this.  Most of the
variables are prefixed with @code{WRS_} to make them easily identified
in the overall mess of variables and command names (all scripts used here
have a similar prefix for the same reason).

@table @code
@item WRS_BASE_DIR
	The absolute pathname of the build directory (i.e., the @code{build/}
        subdirectory of @code{wr-switch-sw}).
        The variable is internally set to the directory
        name of the main script. Note that the script cannot
        be run from the same directory or from the wr-switch-sw project
        directory (i.e.: @code{./wrs_build-all} @code{./build/wrs_build-all}
        are not allowed), you must call it from your output directory
        using a pathname to  invoke it.  This variable cannot be
        overridden in the main script, but must be pre-set if you run
        a sub-script to rebuild only part of the software suite.

@item WRS_OUTPUT_DIR
	The absolute pathname of the directory where output is placed. It 
	defaults
        to the current directory whence you invoke the script (i.e., you
        can invoke @code{/path/to/wrs_build-all} to have all output
        in the current directory).  Compilation
        happens in a @i{build} subdirectory of @code{WRS_OUTPUT_DIR},
        done-markers are placed in a @i{_done} subdirectory
        and final images are placed in a @i{images} subdirectory.

@item WRS_DOWNLOAD_DIR
	The absolute pathname of the directory where downloaded files 
	are placed. If unset
        it defaults to @code{$WRS_OUTPUT_DIR/downloads}, which is
        created if needed.  By pre-setting this variable you can
        simply recursively delete
        the output directory to force a full rebuild, without
        the overhead of re-downloading everything. I personally pre-set
        this so it always points to the same place, even when I remove
        the whole output directory.

@item WRS_HW_DIR
	The absolute pathname of the directory where you build HDL, if any.
        If this variable is set, FPGA binaries will be copied from there
        instead of being extracted by the official archive on @code{owhr.org}.
        This is only used by HDL developers.

@item CROSS_COMPILE
	The variable is the usual cross-compilation prefix. For example,
        @code{arm-linux-} if you have @i{arm-linux-gcc} in your path,
        or a full pathname without the trailing @code{gcc}. If unset,
        it defaults to the compiler that @i{buildroot} self-builds.
        See @ref{The Compiler} for some more details.

@end table

Other variables are used internally in the script; since they are only
useful to people working on the script itself, they are documented in
place.


@c --------------------------------------------------------------------------
@node Downloading Files
@subsection Downloading Files

Every downloaded file is saved to the @code{downloads} directory 
(@code{$WRS_DOWNLOAD_DIR} if set, or the default place
@code{$WRS_OUTPUT_DIR/downloads}). You should
arrange not to remove that directory when you recompile over and over
during development. Download process is divided into two parts. Firstly,
our buildsystem downloads only base packages (at91bootstrap, barebox,
linux kernel, switch's gateware and the builroot). The rest of packages are
downloaded by the buildroot.
In the first step the script downloads mentioned packages, before starting
any build, to help telling download errors from other issues.

For each upstream archive needed, the following steps are performed:

@itemize @bullet
@item If the file exists in the download directory, the @i{md5sum} is
      checked; on success, nothing else is done.
@item If the previous step fails, the file is retrieved from upstream.
@item If the previous step fails, the file is downloaded from the
      buildroot web site.
@end itemize

The policy just described is implemented in @i{wrs_download}, in the file
@code{scripts/wrs_functions}, based on @code{download-info} in the
main build directory.

The messages of a download run are like the following ones:

@smallexample
    2016-12-14 17:10:46: --- Downloading base packages
    2016-12-14 17:10:50: Retrieved at91bootstrap-3-3.0.tar.gz from upstream
    2016-12-14 17:10:51: Retrieved barebox-2014.04.0.tar.bz2 from upstream
    2016-12-14 17:11:21: Retrieved linux-3.16.38.tar.xz from upstream
    2016-12-14 17:11:22: Retrieved wrs-gw-v5.0-20161214.tar.gz from upstream
    2016-12-14 17:11:27: Retrieved buildroot-2016.02.tar.bz2 from upstream
@end smallexample

After buildroot is downloaded, it is unpacked and then configured. Buildroot
uses simillar mechanism to the one described above to download packages that
it needs. Buildroot prints the progress of downloading of each package.
After downloading is over you can work even without a network connection.

@c ==========================================================================
@node Building Procedure
@section Building Procedure
If you just want to build stuff, with no concern about network
downloads and without even knowing what is happening, just create a
directory where you want the output to be generated and start
compilation. Note that it takes around 4GB of storage in excess
of the 400MB downloaded.

Then run this (but please read more for a better command):

@example
   /path/to/wr-switch-sw/build/wrs_build-all
@end example

Note that progress messages are sent to @i{stderr}, so you may want to
save @i{stdout} to a file, like this (again, it is recommended you read
further for a better command):

@example
   /path/to/wr-switch-sw/build/wrs_build-all > logfile
@end example

Please note that there are also a number of warning messages being
printed to @i{stderr}. It is a few hundred lines over the many
minutes it takes to build @i{buildroot}, but you can safely ignore them,
trusting the build process will complete successfully.

The progress messages look like what is shown here below.  The log
file will be rather big (~18MB), as all the compilation steps are
quite verbose.

The following example shows a run on a quad core, dual hyperthreaded system
(8*6800 bogoMips in total). If files had already been downloaded, the first few
step takes only a few seconds, as shown, to verify the checksums:

@smallexample
2016-12-14 17:26:39: --- Downloading base packages
2016-12-14 17:26:39: --- Buildroot: unpack and configure
2016-12-14 17:26:39: Uncompressing buildroot
2016-12-14 17:26:40: Configuring with "[...]/../configs/buildroot/wrs_release_br2_config"
2016-12-14 17:26:40: Patching buildroot
2016-12-14 17:26:40: Reconfiguring buildroot
2016-12-14 17:26:41: --- Buildroot: download packages
2016-12-14 17:26:48: --- Buildroot: compiler and filesystem
2016-12-14 17:26:48: Compiling buildroot
2016-12-14 17:47:54: --- AT91Boot
2016-12-14 17:47:54: Patching AT91Boot
2016-12-14 17:47:54: Building AT91Boot
2016-12-14 17:47:55: --- Barebox
2016-12-14 17:47:55: Patching Barebox
2016-12-14 17:47:55: Building Barebox
2016-12-14 17:48:03: --- Linux kernel for switch
2016-12-14 17:48:52: --- Kernel modules from this package
2016-12-14 17:48:56: --- PTP daemon (ppsi repository as a submodule)
2016-12-14 17:49:05: --- User space tools
2016-12-14 17:49:15: --- Deploying FPGA firmware
2016-12-14 17:49:15: Using pre-built binaries from wrs-gw-v5.0-20161214.tar.gz
2016-12-14 17:49:16: --- Wrapping filesystem
2016-12-14 17:49:21: --- Packing into wr-switch-sw-v5.0-20161214_binaries.tar
2016-12-14 17:49:21: Complete build succeeded, apparently
@end smallexample

You may prefer to save @i{stderr} with @i{stdout} to the log file
but still see the time-stamped messages from the scripts. In this
case you can issue the following command -- which is what I used
to generate the terse output shown above:

@example
   /path/to/wr-switch-sw/build/wrs_build-all 2>&1 | tee logfile \
        | grep "^20..-..-.. ..:"
@end example

If you are lucky, everything completes by itself. The time taken
depends on you CPU, disk and network speed.
At the end you will find your final files in the @code{images}
subdirectory, 

If you are not too lucky, the build stops because
you have found a bug in the build scripts; most likely because your
setup differs from the ones we have been testing on.

In order to re-run the build from the beginning, please remove (or
rename) the output directory and reissue the command.  To only
redo some steps, please see @ref{Rebuilding Parts}.

@c ==========================================================================
@node Build Script Description
@section Build Script Description

The @code{wrs_build-all} can be used to quickly build the White Rabbit 
Software as seen above. However it admits other functionalities detailed
in this chapter. You might also want to check its embedded documentation using:

@example
   /path/to/wr-switch-sw/build/wrs_build-all --help
@end example

@c --------------------------------------------------------------------------
@node Release Package
@subsection Release Package

By default, a complete compilation creates a ``release'' package,
i.e. a @i{tar} archive of all files needed to flash a brand new WR
Switch.  The example above shows that the name is something like:

@example
	wr-switch-sw-v5.0.1-20170825_binaries.tar
@end example

In other words, we include both the tag name (from @t{git describe})
and a date.  If the repository is not checked-out at a release (a ``tag''),
this will be apparent in the filename used for the output.

The ``official'' release package is available from @t{ohwr.org}, in
the @i{Files} section of the @t{wr-switch-sw} project.

In any case, the file must be renamed to @t{wrs-firmware.tar}
to be used at installation time. See @ref{Flashing Procedure} for details.

@c --------------------------------------------------------------------------
@node Build Time Configuration
@subsection Build Time Configuration


Some details of the complete firmware archive depend on the values of
active @t{Kconfig} variables. If no manual configuration is performed,
what applies is @t{configs/wrs_release_defconfig}. Please note that
we now support dynamic reconfiguration at run-time. See the @i{@sc{wrs}
Users' Manual}.

@c --------------------------------------------------------------------------
@node Rebuilding Parts
@subsection Rebuilding Parts

When the main script succeeds in building one part (sub-package),
it creates a file in the @code{build/_done} directory.

When you rebuild everything, steps for which the marker file exists
are not rebuilt. To force rebuilding of one specific part, just remove
the marker.  Markers are numbered, reflecting the order of compilation
steps, but they also have a name: names like @code{06-kernel} should be
self-explicative. 

To ease the rebuilding of a specific module a shortcut has been created 
in the @code{wrs_build-all} script. For example if you want to recompile
the kernel alone you should execute.

@example
   /path/to/wr-switch-sw/build/wrs_build-all --step=06
@end example

You can list all compiled modules by calling

@example
   /path/to/wr-switch-sw/build/wrs_build-all --list
@end example
 
If you want to rebuild various modules at the same time, you should run
something similar as:

@example
   /path/to/wr-switch-sw/build/wrs_build-all --step="5 7"
@end example

Please note that the final step (``@i{wrap root filesystem''}) is
always performed, to ensure any changes are applied in the generated
firmware file.

An alternative way to build parts, though a more difficult one, is running the
individual script from within @i{build/scripts/}, after setting the proper
environment variables.


@c --------------------------------------------------------------------------
@node Rebuilding From Scratch
@subsection Rebuilding From Scratch

If you have updated the repository with new modifications, you might want 
to check that you can rebuild from scratch. To clean your output 
directory by deleting all compiled modules (except downloaded files), just call:

@example
   /path/to/wr-switch-sw/build/wrs_build-all --clean
@end example


@c ==========================================================================
@node The Individual Build Steps
@section The Individual Build Steps

This chapter details the individual build steps, for the users that want
to customize one or more of them to build a different switch firmware
image (or kernel, or whatever).

@c --------------------------------------------------------------------------
@node The Compiler
@subsection The Compiler

The predefined compiler used here is the one built by @i{buildroot}.
The default configuration selects this choice.  If you pre-set a
different @code{CROSS_COMPILE} prefix in your environment, your own
choice will be used by modifying the @i{buildroot} configuration file.
Note, however, that not all cross-compilers will work (@i{buildroot}
wants one that has been configured with @code{--sysroot} and it is
quite unlikely yours has been).

In practice, you may want to set @code{CROSS_COMPILE} when you compile
the boot loader and kernel by themselves, and avoid it when compiling
the complete package.

@c --------------------------------------------------------------------------
@node Buildroot
@subsection Buildroot

The distribution being used here is @i{buildroot}. It is the first
step being built, because it creates the cross-compiler it will use. This
compiler is later used to compile all other software for the White
Rabbit Switch.

The configuration for @i{buildroot} comes from
@code{configs/buildroot/wrs_release_br2_config}. The configuration
is then changed only if you pre-set your own @code{CROSS_COMPILE}
variable.  A different configuration can be chosen in the Kconfig
interface, by running ``@t{make menuconfig}'' or equivalent, in the
top-level source directory.

@c The following can be shown in a set of commands instead of this hard
@c to read explanation.

If you want to change the configuration, you can do so after the first
build iteration: change directory to @code{build/buildroot-2016.02}
and run @code{make menuconfig} (this the Buildroot configuration,
not the one of wr-switch-sw). After making your choices, copy back
the file @code{.config} to @t{configs/buildroot} in this package,
so you can select it by running @t{make menuconfig} in wr-switch-sw.

Then, please run @t{configs/buildroot/RUNME} (without arguments)
in order to remove your local pathnames in the configuration file;
the file without local pathnames can be committed and published for
other people to use.

You can also set @code{WRS_BUILDROOT_CONFIG} to the full pathname of
your configuration file of choice (this used to be the only way
to pass a custom configuration file). The file must be a copy of the
@code{.config} after the @code{make menuconfig} step described above,
within buildroot.
Note that if the variable is not pointing to a regular file it is
ignored with a simple warning -- rather than stopping the build procedure.

An example alternate configuration file is @t{configs/buildroot/wrs_with_musl_config}, that builds a version of the White Rabbit Switch with a different
system library (@i{musl} instead of @i{uclibc}).

@c --------------------------------------------------------------------------
@node The IPL
@subsection The IPL

The version of @i{at91bootstrap} being used in the switch as
@i{Initial Program Loader} is version 3.3, download from  @code{timesys.com/}
(the full URL is in @i{build/download-info}.
The patches we applied are in the directory @i{patches/at91boot/v3.3},
and we are piggy-backing on the Atmel
evaluation board without even changing the board name):

@example
   0001-printf-added-files-from-pptp-unchanged.patch
   0002-printf-fixes-and-addition-to-makefile.patch
   0003-build-Add-gitversion-to-binary-and-a-script-to-compi.patch
   0004-board-9g45ek-fix-ddr-config-for-WRS-V3.patch
   0005-boot-disable-watchdog-asap-added-flip_leds-count-run.patch
   0006-boot-Correct-crash-due-to-an-Atmel-bug-during-boot-w.patch
   0007-gpios-Correct-FPGA-LED-problems-and-add-CPU-LEDs-FAN.patch
   0008-fix-refresh-value.patch
   0009-one-more-fix-to-RAM-timing-according-to-datasheet.patch
@end example

The script @i{wrs_build_at91boot} uncompresses, patches and builds, leaving
@code{images/@-at91bootstrap.bin} after it is over.  This file is
the one to be loaded in the hardware.

If you build using a local @i{git} repository, we suggest to use
@code{git am --whitespace=nowarn} because we have a number of
white space errors, and we apologize for that.

@b{Warning}: with most distributions, this compilation step will print
a scary message about memory corruption. The message is reporting a bug in
the configuration program which has no actual effects and can be ignored.
Maybe we will switch to another version in the future that doesn't show
the bug, or to the newer @i{barebox} that obsoletes @i{at91boot}.

@c --------------------------------------------------------------------------
@node The Boot Loader
@subsection The Boot Loader

The switch uses @i{barebox} as a boot loader. We are running version
2014-04, with a few local patches and the chosen configuration
file. Note that we are piggy-backing on the Ronetix PM9G45 board, out
of laziness.

The patches are part of this package in @i{patches/barebox/v2014.04/} and
the set is made up of the following ones:

@smallexample
   0001-sam945-include-mtd-nand.h-in-device-file.patch
   0002-arm-change-prompt-for-pm9263-wrs-piggy-backs-on-that.patch
   0003-nand-wrs-our-nand-is-16-bit-connected-fix-accordingl.patch
   0004-gpio-add-function-to-check-them.patch
   0005-wrs-on-pm9g45-change-nand-setup.patch
   0006-wrs-on-pm9g45-add-dataflash-initialization.patch
   0007-barebox-add-MAC-addresses-to-environment.patch
   0008-wrs-on-pm9g45-force-memory-to-64MB.patch
   0009-pm9g45-init-for-wrs-move-environment-for-the-UBI-mov.patch
   0010-pm9g45-init-for-wrs-more-relaxed-nand-timings.patch
   0011-lib-sdb-and-sdb.h-from-fpga-config-space-sdbfs-commi.patch
   0012-libsdb-integrate-in-build-system.patch
   0013-commands-add-sdb-commands-to-list-read-setvar.patch
   0014-Read-EDI-bytes-in-JEDEC-to-support-AT45DB641E.patch
@end smallexample

If you build using a local @i{git} repository, we suggest to use
@code{git am --whitespace=nowarn} because we have a number of
white space errors, and we apologize for that.

The @i{barebox} boot loader is organized as a small Unix-like
environment, and its own configuration and scripts live in a small
filesystem.  To ease modification of such configuration and boot steps
the build script copies over the configuration instead of patching it
in the sources.  You can thus edit the files you find in
@file{patches/barebox/v2014.04/env} and rebuild your customized bootloader.
The script that is executed at boot time is @file{env/bin/init} and as
you see it calls the other ones.  The menus included in the shipped
configuration are described in the the @i{@sc{wrs} User's Manual}.

Building @i{barebox} relies on a @i{Kconfig} setup, and the
configuration file we use is
@file{patches/barebox/v2014.04/@-wrs3_defconfig}. Again, this is copied over and
not patched in (see the simple @file{build/scripts/@-wrs_build_barebox}
for details).

@c FIXME: describe how to change the barebox configuration file

After patching and copying over the files, the following commands
build the boot loader using the
cross-compiler built by @i{buildroot}. If you run these
by hand you can use a different compiler (as shown):

@example
   export CROSS_COMPILE=/opt/arm-2010q1/bin/arm-none-eabi-
   export ARCH=arm
   make wrs3_defconfig
   make
   cp barebox.bin images/
@end example

To use the same compiler the scripts use, you need this setting (which is split
in two lines with a local variable to fit the page with in documentation):

@smallexample
   BR=${WRS_OUTPUT_DIR}/build/buildroot-2016.02
   export CROSS_COMPILE=${BR}/output/host/usr/bin/arm-linux-
@end smallexample

@c --------------------------------------------------------------------------
@node The Linux Kernel
@subsection The Linux Kernel

The kernel is currently version 3.16.38, compiled from an uncompressed
tar file (so not within a @i{git} repository).  The upstream
vanilla kernel is downloaded, then
local patches are applied (they come from a @i{git}
repository, but they are currently applied with a simple @i{patch}
command).

The relevant patches are available in @i{patches/kernel/v3.16.38},
and are currently the following ones:

@example
  0001-initramfs-stop-after-one-cpio-archive.patch
  0002-mtd_dataflash-Read-EDI-bytes-in-JEDEC-to-support-AT4.patch
  0003-wr-switch-sam9m10g45ek-change-USB-vbus_pin-from-PB19.patch
  0004-wr-switch-sam9m10g45ek-enable-FPGA-access-from-EBI1-.patch
  0005-wr-switch-sam9m10g45ek-store-device-partitioning.patch
  0006-wr-switch-sam9m10g45ek-more-relaxed-nand-timings.patch
  0007-wr-switch-sam9m10g45ek-provide-bootcount-using-scrat.patch
  0008-wr-switch-at91-udc-force-full-speed.patch
  0009-hack-architecture-to-boot-on-our-boot-loader.patch
  0010-disable-BBT-for-the-nand-flash.patch
@end example

The configuration we use to build the kernel is not a patch but a plain
@code{.config} file (@t{configs/wrs_linux_defconfig}), so you
can change it easily, if needed. As an alternative,
you can also set @code{WRS_KERNEL_CONFIG} to the full pathname of
your configuration file of choice. The file must be a copy of the
@code{.config} found in the main kernel directory,
(for example the one left after the @code{make menuconfig} step).
Note that if the @code{WRS_KERNEL_CONFIG}
variable is not pointing to a regular file it is
ignored with a simple warning, without stopping the build procedure.

The build scripts copy both @i{zImage} and all compiled kernel
modules to the @i{images/} directory of the build place. This currently
includes modules.

@c --------------------------------------------------------------------------
@node Kernel Modules
@subsection Kernel Modules

In the next step the scripts compile modules that are part of this
package.  The step depends on the kernel being available in the
build directory. The modules are then copied into the
@file{images/wr/lib/modules/} subdirectory of the main build directory.

Please note that modules (and later user-space) are compiled in-place;
i.e. not in the output directory.  The disadvantage is that your repository
becomes dirty with output and intermediate files. The advantage is that
any modification you make to the code is already in the repository
for you to commit.

Currently, the package includes the following modules:

@itemize @bullet

@item @i{htvic.ko}: the interrupt controller for in-FPGA devices.
@item @i{wr-nic.ko}: the network ``card'' driver for WR ports.
@item @i{wr_rtu.ko}: the routing-table interface between the
      switching core and the associated user-space daemon.
@item @i{wr_pstats.ko}: exports per-port statistics to /proc/sys.

@item @i{wr_clocksource.ko}: uses WR time as a source for system time.
This driver uses the @sc{wr} counters to make host time flow at the
right speed as soon as @i{ppsi} synchronizes. This ensures no drift
will accumulate in system time, keeping the same
offset (well lower than a second, after the initial ntp-driven setting event).
This is considered acceptable, because system time is only
used for logging.


@item @i{wrs_devices.ko}: a dummy module that register our platform devices.

@end itemize

@c --------------------------------------------------------------------------
@node PTPd
@subsection PTPd

Configuration used to support two different PTP engines, but now
we only support PPSi.

The code is hosted in its own
repository; it is a @i{git} submodule in this package.
The repository is hosted on @code{ohwr.org}, like others.

PPSi is @i{Kconfig} based: you may build it in its own directory by
using its @i{wrs_defconfig} and the proper @t{CROSS_COMPILE} variable.

@c --------------------------------------------------------------------------
@node User Space Applications
@subsection User Space Applications

The filesystem of the switch includes some user-space applications
and tools.  Some of the @i{tools} are actually used by the init
scripts and some are just utilities for the developer.

The subdirectories in @file{userspace} include the various applications
needed for the operation of the switch itself, as well as support libraries
used by the applications themselves.

The main components are:

@table @i

@item mini-rpc
	A remote procedure call library used by most other programs
        to exchange information among themselves or query the LM32
        that is running on the FPGA.

@item libwr
	A series of utility functions to access the switch itself.

@item wrsw_hal
	The main application program for the White Rabbit Switch
        operation. The script installs the executable in @code{images/wr/bin}.

@item wrsw_rtud
	The daemon for the routing table unit, used for routing around
        data frames. It is installed in
        @code{images/wr/bin}.

@end table

@sc{wrs} user space includes also some tools and scripts.
They are all described in the User's Manual, even though some of
them are only useful to software developers.

Please note that to compile the applications and tools outside of the build
scripts you need to specify both the kernel
directory (@code{LINUX=}) and the cross-compiler to use
(@code{CROSS_COMPILE=}).

@c --------------------------------------------------------------------------
@node VHDL Binaries
@subsection VHDL and LM32 Binaries

The gateware binaries that are needed to run the FPGA are added to the
target filesystem by the @file{wrs_build_gateware} script. If the
variable @code{WRS_HW_DIR} is set, the script uses it to retrieve the
binaries you just compiled (but the script is not compiling gateware).

If the variable is not set, the script extracts a tar file downloaded
from @code{ohwr.org} as part of the initial download step.

The LM32 program is provided as a pre-compiled binary in
@code{binaries/rt_cpu.bin}.

More details about the binaries are in @ref{Other Repositories}.

@c --------------------------------------------------------------------------
@node The Complete Filesystem
@subsection The Complete Filesystem

The final step in building the switch software is wrapping together
the filesystem for the switch, also making the archives.

The step of setting up the complete filesystem is performed by
@file{build/scripts/@-wrs_build_wraprootfs}.  The script
does not leave a directory tree on the build system because that would require
administrator privileges. We think it is best not to call @i{sudo} from
within build scripts, to respect our users' security concerns.

The script creates an archive for the whole filesystem,
called @t{wrs-image.tar.gz}.  It is used by the installation
procedure and it is ready to be unpacked for NFS-Root. It is currently 
slightly less than 25MB of data.

To make your NFS-root place, you can run
the following command in a newly-created empty directory:

@example
   tar xzf $WRS_OUTPUT_DIR/images/wrs-image.tar.gz
@end example

To boot with NFS-root you should use a custom boot script,
as described in the section @i{Using wrboot}, in the @i{@sc{wrs} User's
Manual}.  Please note that the kernel now needs @t{root=/dev/nfs},
as the old convention @t{root=nfs} is not supported any more.

The archives include a number of device special files in
@i{dev}. The pre-created devices come from
@i{userspace/devices.tar.gz}. Note that the buildroot output
directory, @i{build/buildroot-2016.02/output/target} does not
include any device (and no white-rabbit specific files), so it
cannot be used as a root filesystem by itself.

The content of the final filesystem comes from several sources:

@itemize @bullet

@item The @i{buildroot} output (from its own @file{output/target/}).
@item The switch-specific overlay (@file{userspace/roofs_override}).
@item The @file{images/wr} and @file{images/lib} trees,
      filled by the build scripts.
@item The file @file{userspace/devices.tar.gz}
@item The file @file{$WRS_BASE_DIR/authorized_keys} if it exists.

@end itemize

The final step allows a predefined set of users to enter as system
administrator without the need to type a password (which, anyways is
empty by default).  It is useful if you @i{scp} files in and out of
the switch. In the shipped binaries no user is authorized, but the
root password is still the empty string.

@c ##########################################################################
@node Flashing the Switch
@chapter Flashing the Switch

This chapter describes the steps to install the @sc{wrs} with the 
current firmware images. As far as hardware is concerned, this
procedure describes the installation of the switch with a @sc{scb}
version 3.3 or 3.4, and a @sc{Mini-Backplane} version 3.3.
Older versions are not documented here
any more (please get an older manual, if needed).

@b{Note:} Most likely you won't need to reflash your @sc{wrs}.
Even if you rebuilt software from scratch, the ``update'' procedure
that is available since August 2014 (release 4.0 of @t{wr-switch-sw})
allows complete replacement of the firmware image without physical
access to the device.

@c ==========================================================================
@node USB connections
@section USB connections

In order to perform the flashing operation easily, you should connect
two @i{mini-USB} cables to the switch ports (Actually, one is enough, but the
second one is useful to get more diagnostics while flashing).

The two back panel @i{mini-USB} sockets correspond to the serial 
port of the FPGA and the ARM. They are labeled @b{FPGA test} and 
@b{ARM test}.  You should connect to ``ARM test'' to get diagnostics.

You can connect to it using minicom 
@footnote{You can use other programs for accessing serial ports, for 
example @uref{http://brokestream.com/tinyserial.html, tinyserial}} 
like this:

@example
	minicom -D /dev/ttyUSB0 -b 115200
@end example

The port, however, will only appear on the PC after the switch is turned
on, so you may want to delay this command.

@sp 1

The front panel USB connection, labeled as @b{management} USB port, communicates 
with the internal ROM of the CPU. It is the one used to perform the 
flashing procedure.  No special program is needed, as the flashing tool
will communicate with this port by itself.

You first need to set up the switch in @emph{"Flashing mode"} to 
continue with the flashing procedure. To do so, you should turn on 
the power while pressing the @b{flash button} on the rear panel.

If the operation succeed you should see the message @code{bootROM} 
appears on the ARM UART. (You will likely not see it, because your
@i{minicom} or equivalent can't run before the switch is turned on).

You can also see the enumerated device in your own host:

@smallexample
   $ lsusb | grep Atmel
   Bus 001 Device 025: ID 03eb:6124 Atmel Corp. at91sam SAMBA bootloader
@end smallexample

Finally, the kernel should also load the proper device driver, and you
are expected to see @code{/dev/ttyACM0} or equivalent in your system.
This is the device used for flashing.

If it is not the case, this means that the button used to disable the dataflash
and enter @emph{"Flashing mode"} is not working. You should contact support.

@c =============================================================================
@node Flashing Procedure
@section Flashing Procedure

If the update procedure, described in the @i{User's Manual}
is not suitable for you (because, for example, you are the manufacturer),
you need physical access to your @sc{wrs} to flash it.

Unlike what happened with previous releases (up to the end of 2013),
the filesystem of the switch can't fit in RAM memory during installation
any more: the image is now downloaded through the network.  Thus
you need to following items to flash a switch:

@itemize @bullet
@item The USB cable connected to the front ``management'' USB port
@item A Linux host connected as a master to this cable
@item An Ethernet cable connected to the front ``management'' Ethernet port
@item A DHCP server on your network, offering an IP address to the switch
@item A TFTP server, exporting the file @t{wrs-firmware.tar}
@end itemize

The flashing procedure will use the @i{server address} reported by
DHCP as IP address for the TFTP transfer.

Also, since release v4.1, you should not provide MAC addresses
while flashing any more. The two MAC addresses are expected to be stored
in @i{dataflash} by the manufacturer and not changed any more. If you
upgrade your switch from a previous software version, please refer
to the @i{User's Manual} for details.

The tool used to flash the firmware into the switch is the @i{USB-loader}
we inherited from Atmel. The @file{flash-wrs} script is what
you'll use to run the loader with appropriate parameters.

The script can be invoked in the following way to flash a
@i{package} into the switch. The package is the @code{wrs-firmware.tar}
file created by ``@code{wrs_build-all}'' (see @ref{Release Package}).

@b{Note:} the release file must be renamed to @code{wrs-firmware.tar},
because the pathname is hardwired in the installation procedure.

The command to flash is as follows:

@c FIXME: check this ``-e''
@example
    /path/to/wr-switch-sw/build/flash-wrs -e wrs-firmware-<revision>.tar.gz 
@end example

You can also flash the image you have built using the procedure
described in @ref{Building WRS Software} by
adding the tag @code{-b|--build}. To use this option you must call
the script from the build directory, or define the @t{WRS_OUTPUT_DIR}
environment variable.

@example
    /path/to/wr-switch-sw/build/flash-wrs -e -b
@end example

Please note that the ``@code{-e}'', which requires erasing the
whole data flash, is almost mandatory because otherwise bits of your previous
installation may leak into the newly-programmed one. Only on factory-new
devices you can avoid this ``@code{-e}'' argument. 

@c FIXME: -e for dataflash

@b{Note:} White Rabbit switches are shipped with their pre-allocated
MAC addresses, reported in a sticker on the back side of the switch;
if re-flashing, you may want to use the same values.

Please remember that bits 0 and 1 of the first byte are special: if
the first byte is odd, the MAC address is reserved for multicast
transmission (the script doesn't check, and the kernel will refuse to
enact such address).  Bit 1 is set for ``locally assigned'' numbers:
while official MAC addresses have bit 1 clear, if you choose your
unofficial addresses you should set the bit.

If you don't configure a MAC address, a warning will be displayed and
you can abort the procedure. If you don't abort the flashing
procedure, the script will use default MAC addresses. Default MAC addresses
are: 02:34:56:78:9A:BC for MAC1 (the Ethernet port of the ARM CPU)
and 02:34:56:78:9A:00 for MAC2 (the base address for the 18 SFP ports).


@smallexample
   tornado% ~/wip/wr-switch-sw/build/flash-wrs -e -b
   flash-wrs: Working in /tmp/flash-wrs-1vV9z6
   Warning: you did not set the MAC1 value; using "02:34:56:78:9A:BC"
   Warning: you did not set the MAC2 value; using "02:34:56:78:9A:00"

   flash-wrs: Waiting for at91sam SAMBA bootloader on usb.
       Please check the Management USB cable is connected
       and keep pressed the Flash button while
       resetting/powering the switch.
   ............. Ok
   flash-wrs: I'm talking with the switch;
      please release the flash button and press Enter to start flashing:
@end smallexample

If the script cannot find the Atmel programming interface on your USB
bus, it prints a message and waits for the switch to be turned on
in the proper way (with the button pressed or, for older hardware
versions, the jumper plugged).

The process calls the flasher program twice (so you'll see the
initialization strings two times) and takes slightly less than 5
minutes.  the longest step is erasure of @i{DataFlash}: if run
without @t{-e} the script takes 2 minutes.

This is the summary of the output you are expected to see,
trimmed to save pages:

@smallexample
Initializing SAM-BA: CPU ID: 0x819b05a2

[...]

Initializing DDR...
loading applet isp-extram-at91sam9g45 at 0x00300000
Initializing DDR > Done

Initializing DataFlash...
loading applet isp-dataflash-at91sam9g45 at 0x00300000
Initializing DataFlash > Done! 

Erasing DataFlash [... there is a long delay here ...] > DONE

Programming DataFlash...
	 @ 0x70000000 : at91bootstrap.bin ; size 0xf7c (3Kb)
DataFlash: Writing 3964 bytes at offset 0x0 buffer 70000000....ABCDEF OK
	 @ 0x70008400 : barebox.Fb09jx ; size 0x2f1bc (188Kb)
DataFlash: Writing 192956 bytes at offset 0x8400 buffer 70000000....ABCDEF OK
Programming DataFlash Done!!!
[...]

Initializing NandFlash...
loading applet isp-nandflash-at91sam9g45 at 0x00300000
Initializing NandFlash > Done! 

Erasing NandFlash > DONE

[...]

Initializing DDR...
loading applet isp-extram-at91sam9g45 at 0x00300000
Initializing DDR > Done

Loading DDR...
         @ 0x70000000 : /tmp/flash-wrs-tAqUAs/bb.new ; size 0x637b0 (397Kb)
         @ 0x71000000 : /data/morgana/build-v4/images/zImage ; size 0x1afb44 (1726Kb)
         @ 0x717ffff8 : /tmp/flash-wrs-tAqUAs/magicstr ; size 0x8 (0Kb)
         @ 0x71800000 : /data/morgana/build-v4/images/wrs-initramfs.gz ; size 0x196f84 (1627Kb)
DDR: Writing 3842688 bytes at offset 0x0 buffer 70000000....ABCDEF
Closing...
Formatting UBI device... done
Getting tftp://192.168.16.1/wrs-firmware.tar ... done
Extracting filesystem...  done
@end smallexample

The longest steps are erasing @i{dataflash} (it takes 2 minutes) and
the last three steps: formatting, tftp and extraction; each of them takes
around 1 minute.

Please note that the IP address used in the TFTP transfer depends on
the DHCP handshake: the value above is what your developers use.
The name @t{wrs-firmware.tar}, on the other hand, is hardwired:
it matches the result of a firmware build, and the file name used
within the archive of official binaries we ship at release time.

It is suggested to look at the CPU's serial port during programming,
where you will see messages like these:

@smallexample
   -I- Statup: PMC_MCKR 1202 MCK = 100000000 command = 0
   -I- -- EXTRAM ISP Applet 2.9 --
   -I- -- AT91SAM9G45-EK
   [...]
   -I-     End of applet (command : 2 --- status : 0)
[...]
barebox 2014.04.0 #1 Tue Jun 24 09:05:43 CEST 2014
Board: White Rabbit Switch
[...]
Booting kernel for NAND flashing procedure
100Mbps full duplex link detected
DHCP client bound to address 192.168.16.246
[...]
Uncompressing Linux... done, booting the kernel.
[...]
/etc/init.d/wrs-boot-procedure: Running

Formatting UBI device... [...] done
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    129024 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
[...]
Getting tftp://192.168.16.1/wrs-firmware.tar ... done
UBIFS: mounted UBI device 0, volume 1, name "usr"
Extracting filesystem...  done
Requesting system reboot
Restarting system.
@end smallexample

Please note, however, that many more messages flow, as
formatting/mounting/umounting UBI devices is very verbose in the
kernel.  The sequence above is a summary of what happens at
installation time.

@c -------------------------------------------------------------------
@node Flash Script Description
@subsection Flash Script Description

@c FIXME: flash script
@b{Note}: this section may be slightly outdated, it needs review.

The @code{flash-wrs} script can be used to quickly flash the White Rabbit switch
as seen above. However it admits other functionalities detailed in this chapter.
You might also want to check its embedded documentations using:

@smallexample
$ ./build/flash-wrs --help
Usage: ./build/flash-wrs [options] [<firmware>.tar.gz] [DEV]

MAC:	 MAC address in hexadecimal separated by ':' (i.e, AB:CD:EF:01:23:45)
<firmware>.tar.gz: Use the file in the firmware to flash the device
DEV:	 The usb device (by default it is /dev/ttyACM0)
Options: 
  -h|--help	 Show this help message
  -m|--mode	 can be: default (df and nf), df (dataflash),
		 nf (nandflash), ddr (ddr memories).
  -g|--gateware	 Select the gateware: 18p (18 ports, default), 8p (8 ports)
  -e 		 Completely erase the memory (Can erase your configuration)
  -b|--build	 Use files that you have built in the WRS_OUTPUT_DIR

@end smallexample

The @i{DEV} is optional and the default is @code{/dev/ttyACM0}.
If your system maps the Atmel ROM to a different device name, please
pass the name on the command line. The script wants a full pathname 
starting with @code{/}.

If you want to flash the @i{at91boot.bin}, @i{barebox.bin}, @i{kernel}
or @i{file-system} that you just built, you can just call the script
from the build directory and use the @code{-b} option.

The official binaries for installation of version 5.0.1 of this package
are available in the @i{files} tab of this project in @t{ohwr.org}.
We don't provide a complete link here, but one is available in the
list of downloaded files: @t{build/download-info}.

You can select a mode using the @code{-m|--mode} flag to choose
to write in dataflash (df), nandflash (nf) or both (default).
The memory mode is used to select a partial re-flashing; this
is how the switch firmware is split among the two memories:
@itemize @bullet
@item dataflash: @emph{at91boot} and @emph{barebox} binaries
@item nandflash: @emph{kernel}, @emph{initramfs} and /usr @emph{file-system}
@end itemize

You can select which type of gateware you want to flash on your 
switch. Currently we only support LX240T (the current circuit doesn't
fit in the LX130T any more).  8-port images are sometimes used for
testing. And you can select this option like this:

@smallexample
$ ./build/flash-wrs --gateware 8p <...>
@end smallexample

You can also erase the dataflash memory before writing your binaries; to do this
add the option @code{-e}.   There is no need to especially erase
nand flash, because the installation procedure does the right thing with
it in any case.

The script performs the following steps:

@itemize @bullet
@item It compiles the loader (@i{usb-loader/} subdir).
@item It checks if the SAMBA bootloader is present.
@item It picks the correct binaries according to the options.
@item Optionally, it changes the default MAC addresses in @i{barebox}
      default environment, so you can use a different MAC for each switch.
@item Optionally, it erases the dataflash memory.
@item It places a magic string in RAM, to tell barebox we are installing
@item It loads the kernel and filesytem to RAM and boots them
@item It reads /dev/ttyACM0 to report the messages printed during flashing
@end itemize

The boot loader being booted finds the the magic string in memory, and
changes the kernel command line to force installation-mode.  The
kernel and filesystem being booted in the switch are the same images
for installation and run-time. (Releases before 4.0 built a special
installation filesystem, but now the procedure is simplified).

@c -------------------------------------------------------------------
@node Rebuilding Sam-ba Applets
@subsection Rebuilding Sam-ba Applets

The loader depends on code by the CPU vendor, which is very bad
quality as typical in the field.   If, by unlucky chance, you need to
rebuild the applets, you need a specific version of the cross-compiler,
and everything else will spit horrible errors.

A binary copy is uploaded in the @i{Files} sections of the @sc{ohwr}
project. The direct link is
@url{http://www.ohwr.org/attachments/download/3138/cd-g__lite.tar.gz}
(the name was @t{cd-g++lite.tar.gz}, but @sc{ohwr} changed the @t{+}
into @t{_}).

To build, you can run Benoit's script
@t{usb-loader/samba_applets/isp-project/build.sh}.

@c ##########################################################################
@node Flash Memory Use in WRS
@chapter Flash Memory Use in WRS

This is a summary of how we used the two internal flash
memories in the switch, when programmed with the official firmware
binaries. It is meant for people who want to better understand the
boot procedure and possibly customize stuff using higher-level tools,
like erasing and rewriting flash-memory areas from Linux itself.

Unfortunately, the CPU is not able to boot from NAND memory directly,
so the first steps of booting are performed from the @i{dataflash}
device.  Such an SPI memory is used to host the IPL (@i{at91boot}) and
the executable code of the @i{barebox} boot loader.  The user is not expected to
ever erase this memory; if it happens, the system won't boot and
you'll be forced to re-flash it entirely, which requires access to the back
side of the switch.

NAND memory is used for user-data: the boot loader configuration, the
kernel and the filesystem.

This is how the memory is used:

@example
   0x0000.0000 - 0x0010.0000    Barebox-environment-backup
   0x0010.0000 - 0x2000.0000    UBIfied-NAND
@end example

The first area is used to save the boot loader's configuration (if ever
changed from the default and saved), and the second one is later split
in UBI volumes.  In the future we plan to move the barebox
environment to dataflash memory, where a partition is currently reserved
but is not being used.

The @i{dataflash} is partitioned too, in the following way:

@example
   0x0000.0000 - 0x0000.8400   at91boot
   0x0000.8400 - 0x0008.c400   Barebox
   0x0008.c400 - 0x0009.4800   Barebox-Environment
   0x0009.4800 - 0x0009.5040   hwinfo
   0x0009.5040 - 0x0084.0000   Available-dataflash
@end example

All those partitions are available as @i{mtd} partitions in @i{/dev};
thus, for example, you can replace @t{barebox.bin} by just writing it
to the right device file.  The @i{hwinfo} partition is only available
as a read-only file (@i{/dev/mtd5ro}) because neither users nor
developers are ever expected to change the hardware information data.

Overall, the following is the content of @file{/proc/mtd}
after boot.  It is divided in stanzas for a better reading:
NAND partitioning, dataflash partitioning and the UBI volumes
stored withing ``UBIfied-NAND'' are thus shown in separate blocks:

@example
   dev:    size   erasesize  name

   mtd0: 00100000 00020000 "Barebox-environment-backup"
   mtd1: 1ff00000 00020000 "UBIfied-NAND"

   mtd2: 00008400 00000420 "at91boot"
   mtd3: 00084000 00000420 "Barebox"
   mtd4: 00008400 00000420 "Barebox-Environment"
   mtd5: 00000840 00000420 "hwinfo"
   mtd6: 007aafc0 00000420 "Available-dataflash"

   mtd7: 0201d800 0001f800 "boot"
   mtd8: 0961e000 0001f800 "usr"
   mtd9: 0961e000 0001f800 "update"
@end example

If you are customizing the switch, you may use the UBI commands to
change volumes: the commands are installed in the system,
within the @i{initramfs} image so they can be used before the flash
is accessed.

This is the role of the three UBI volumes (you can change the size
of the volumes or add new ones, but these three names appear in the
boot scripts):

@table @code

@item boot

	The boot volume hosts the kernel (@t{zImage}) and initial
        RAM disk (@t{wrs-initramfs.gz}).
        It is mounted by the boot loader for the default boot procedure,

@item usr

	This is the main filesystem, mounted under @t{/usr} during normal
        operation.  Both @t{/wr} and @t{/var} point to @t{/usr/wr} and
        @t{/usr/var}.  Moreover, the boot procedure copies @t{/usr/etc}
        to @t{/etc} as a first step, so any on-flash configuration is
        actually used by the running system.

@item update

	The volume is a storage place for firmware upgrade. If you
        copy @t{wrs-firmware.tar} in this volume, the next boot will
        completely replace @t{/usr} with this new image. If the tar
        file includes them, the kernel and @i{initramfs} image are
        replaced as well.  Developers can copy individual files,
        to upgrade only one of boot loader, kernel, @i{initramfs}
        and @i{/usr}.

        Additionally there are two files stored on @t{update} volume. File @t{last_update} contains date when last update was performed.
        Second file @t{saved_date} contains date of last gently shut down of a switch. Please note that both dates are saved with best effort. None of them is guaranteed.
        For more details please refer to @ref{Time keeping over restarts}.

@end table

For further details on the update procedure and exact file names to be
used in partial updates during development, please see
@t{/etc/init.d/wrs-boot-procedure} (in the source archive it is
distributed in @t{userspace/rootfs_override/}.

@c ==========================================================================
@node Restore default Barebox environment
@section Restore default Barebox environment

In some cases it might be necessary to restore default Barebox's environment.

Please note that so far switch uses only "Barebox-environment-backup"
partition. It doesn't use "Barebox-Environment".

Erasing can be performed from Barebox's commnad line:
@smallexample
   erase /dev/env0
@end smallexample
or from Linux:
@smallexample
   flash_erase /dev/mtd0 0 0
@end smallexample


@c ##########################################################################
@node WRS Internals
@chapter WRS Internals

@c ==========================================================================
@node Inter-Process Communication
@section Inter-Process Communication

This chapter described the network of IPC/RPC communications that are
active inside the switch.

Currently there are two mechanisms in place: a simple RPC library
(``Remote Procedure Call'', so a process can ask another process
to perform an action, and a shared memory mechanism, so status of
each @sc{wrs} process can be shared with other processes.

Initially, and up to  release 4.1 of
@t{wr-switch-sw} everything was RPC-based, including the passing
of status information.  Starting in November 2014 we introduced
shared memory, to lower the CPU usage and increase the ability
to monitor overall Switch status.

@c --------------------------------------------------------------------------
@node mini-rpc
@subsection mini-rpc

The RPC mechanism in the switch relies on the @i{mini-rpc} package.
The package is a submodule, currently at this commit:

@smallexample
  4c87062d userspace/mini-rpc (v1.0-5-g28dff05)
@end smallexample

The library is documented in its own doc/ directory. The basic
idea is that an RPC server exports stuff through a Unix socket;
the library can return the file descriptor to be used for @i{select}/@i{poll}
and all RPC is handled by calling a specific function.  The client
connects to the server and makes function calls using a library helper.

Each procedure being exported is described by a structure, that lists
the number and types of all arguments and the return type of the
function.  This is, unfortunately, heavier than we initially expected.

If you are studying the code and making sense of the RPC calls,
the important things to look for are @t{struct minipc_pd} (procedure
definition), @t{minipc_server_create}, @t{minipc_client_create}. To see
what procedure are exported, look for the function @t{minipc_export}.

As a special case, the RPC mechanism can happen over shared memory;
this is how the user-space programs can interact with the soft-pll
(the so-called @i{rt subsystem}).

The library is built from a @i{git} submodule in
@t{userspace/mini-rpc}.  Previous versions of this package included
more copies of the library: one within @i{ptp-noposix} and a subset in
@i{rt/ipc}.  Since version 4.1 (Oct 2014) of @i{wr-switch-sw} we
stopped supporting @i{ptp-noposix}, and the @i{rt} subsystem is built
from @i{wrpc-sw} since version 4.0 (Aug 2014).

@c --------------------------------------------------------------------------
@node RPC Sockets and Communication
@subsection RPC Sockets and Communication

This section describes the network of RPC calls that exist in the
White Rabbit Switch.

RPC servers are created in the following places:

@table @code
@item userspace/wrsw_rtud/rtud_exports.c

	The socket is called @t{rtud} and is used by @i{wrs_vlans}
        to request actual actions.

@item userspace/ppsi/arch-wrs/wrs-startup.c
@c FIXME: ppsi should use shmem

	The @t{ptpd} channel is created to report PTP status information,
        but @i{ppsi} is already instrumented to export using shared
        memory too. Thus, this RPC server will be removed soon.

@item userspace/wrsw_hal/hal_exports.c

	This channel is called @t{wrsw_hal} but the code uses is through
        the macro @t{WRSW_HAL_SERVER_ADDR}. The RPC server is used
        only for commands. All clients use shared memory to get status
        information.

@item wrpc-sw::ipc/rt_ipc.c

	This is a memory-mapped channel, at address 0x10007000
        (@t{RTS_MAILBOX_ADDR}).

@end table

Clients are created in the following places:

@table @code

@item userspace/tools/wrs_vlans.c

	@t{wrs_vlans} connects to the @i{rtud} to request configuration
        actions related to vlan setup. All status information is passed
        through shared memory.

@item userspace/tools/wr_mon.c

	The tty-based monitoring interface connects to @i{ptpd} (@i{ppsi})
        to enable or disable tracking. All status information is passed
        from @i{HAL} and @i{ppsi} through shared memory.

@item userspace/libwr/hal_client.c

	The library allows connecting to the HAL socket to ask
        for various actions (all librarized). The function
        @t{halexp_client_try_connect()} is called directly by
        @i{wr_phytool}.

@item userspace/libwr/rt_client.c

	The hal connects to the @i{rt} subsystem to control the
        soft-pll.

@end table

Functions exported by the HAL and PTP are defined in two headers:
@t{hal_exports.h} and @t{ptpd_exports.h}. The former exists
in two places, because @i{ppsi} is a separate package; the two copies
are identical and are expected to remain so. The same applies to @t{rt_ipc.h},
which appears both here and in @i{wrpc-sw}.

@c --------------------------------------------------------------------------
@node The Functions being Exported
@subsection The Functions being Exported

This section lists all functions that are being exported to @sc{rpc} by
the processes (excluding the @sc{rt} subsystem).

@table @code

@item ppsi::get_sync_state
@c FIXME: shmem this

	Called by @i{wr_mon} to fill a @t{ptpdexp_sync_state_t}
        structure.  This can be moved to shared memory.

@item ppsi::cmd

	Called by @i{wr_mon} to ask for two wr-servo operations:
        enable or disable tracking and adjust phase to a specified
        value.
        @c see wr_servo_enable_tracking
        @c see wr_servo_man_adjust_phase

@item hal::pps_cmd

	Called by library code (@t{halexp_pps_cmd}), in
        turn called by @t{ptpd_netif_adjust_counters}, called
        by @i{wr_phytool}.
        Called by @i{ppsi} directly (no @i{libwr}) in @t{wrs-time.c}.

@item hal::lock_cmd

	Called by library code (@t{halexp_lock_cmd}) but not used.
        Called by @i{ppsi} directly in @t{wrs-time.c}.

@item  rtud::clear_entries
@itemx rtud::add_entry
@itemx rtud::remove_entry

	Called by @i{rtu_stat} when performing actions.

@item rtud::learning_process

	Called by @i{rtu_stat} to enable/disable learning process on a set
	of ports.

@item rtud::unrec

	Called by @i{rtu_stat} to enable/disable dropping of frames which
	destination MAC address does not match to present RTU rules.

@item rtud::vlan_entry

	Called by both @i{rtu_stat} and @i{wrs_vlans}. It is
        used to pass a number of parameters to @i{rtud} and
        make it perform actions.

@item rtud::hp_mask

	Called by @i{rtu_stat} when setting which priorities are considered
	High Priority (this only concerns the traffic which is fast-forwarded).

@end table

@c --------------------------------------------------------------------------
@node The RT Subsystem
@subsection The RT Subsystem

The in-FPGA processor running the real-time subsystem of the switch
is using a shared memory connection for communication. The details
are documented in the @t{mini-rpc} manual.  Only the hal process
sends commands to the @sc{rt} subsystem.

@c --------------------------------------------------------------------------
@node WRS Shared Memory
@subsection WRS Shared Memory

The White Rabbit Switch has a shared memory system, with librarized
functions to access it.  Each process exports status
information in its own file, withing @i{/dev/shm/}; each file has
the same structure, but the size allocated in shared memory is
process-specific.

The initial part of each process' area is a @t{stuct wrs_shm_head}, which
allows to make some sense of the overall area.  The structure
is filled by library functions and accessed by shared memory users.
You can see how it is used in @t{tools/wrs_dump_shmem.c}.

The following functions are defined. Please look in the source code
for details about how they are used:

@table @code
@item struct wrs_shm_head *wrs_shm_get(enum wrs_shm_name name_id, char *name, unsigned long flags);
@itemx int wrs_shm_put(struct wrs_shm_head *head);

	Request access to a shared memory area, and stop using it.
        Flags are  @t{WRS_SHM_WRITE}, @t{WRS_SHM_READ} and @t{WRS_SHM_LOCKED}.
        If @t{WRS_SHM_WRITE} is set, head is properly initialized.
        If @t{WRS_SHM_LOCKED} is set, a  writer will mark the area as locked
        before writing its own @t{pid}, and a reader will wait for the @t{pid}
        to be valid (i.e. it waits for a writer to be there). A writer using
        the ``locked'' flag must release the lock after initialization
        by calling @t{wrs_shmem_write(WRS_SHM_WRITE_END)}.
        On error NULL is returned, and errno is set to @t{EINVAL},
        @t{ETIMEDOUT} or to the error returned by underlying system calls
        (for example, @t{EPERM} if the file cannot be mapped).

@item int wrs_shm_get_and_check(enum wrs_shm_name shm_name, struct wrs_shm_head **head);

	Small helper function for opening shmem and checking if initial data is
	already populated. Returns different errors when opening shmem failed,
	when version is not initialized (version equals to 0) or when data in
	shmem is inconsistent.

@item void *wrs_shm_alloc(struct wrs_shm_head *head, size_t size);

	Allocate data space within the shared memory area. The returned
        pointer can be used directly. Only writers should allocate
        but the code is not checking for this. The function is used,
        for example, in @t{wrsw_hal/hal_ports.c}.

@item void *wrs_shm_follow(struct wrs_shm_head *head, void *ptr);

	A reader can follow a pointer using this function.  The writer
        can allocate shared memory with @t{wrs_shm_alloc} and store the
        pointer in the same shared memory area, thus instantiating
        structures that point to other structures.  But the reader processes
        map the shared memory at a different address: this function
        can be used to convert a pointer in the writer's address space
        to a pointer in the reader's address space, or NULL if on error.
        Please see @t{tools/wrs_dump_shmem.c} about how this is used.

@item void wrs_shm_write(struct wrs_shm_head *head, int flags);
@itemx void wrs_shm_write_caller(struct wrs_shm_head *head, int flags, const char *caller);

	@t{flags} is @t{WRS_SHM_WRITE_BEGIN} or @t{WRS_SHM_WRITE_END}.
	Whenever internal consistency of data structure is needed, the
        writer should call this function before modifying shared structures,
        and also
        when all modifications are done and data is internally consistent.
        It is recommended to use @t{wrs_shm_write} version which is implemented
        as a macro, which calls @t{wrs_shm_write_caller} with a callee's
        function name as last parameter. Such approach is can be used for
        tracking source of write begin and write ends.

        A call with @t{WRS_SHM_WRITE_END} is mandatory for writers that
        used @t{WRS_SHM_LOCKED} at @t{wrs_shm_get} time.

@item unsigned wrs_shm_seqbegin(struct wrs_shm_head *head);
@itemx int wrs_shm_seqretry(struct wrs_shm_head *head, unsigned start);

	A reader can use these functions to ensure it reads
        internally-consistent data from a shared structure. It relies
        on proper use of @t{wrs_shm_write()} by the writer.

@item int wrs_shm_age(struct wrs_shm_head *head);

	The function returns the age, in seconds, of the last
        modification to the memory area.  It relies
        on proper use of @t{wrs_shm_write()} by the writer.

@item void *wrs_shm_data(struct wrs_shm_head *head, unsigned version);

	Returns a pointer to data after the @t{struct wrs_shm_head}.

@item wrs_shm_set_path(char *new_path);

	This function can be used to open shmem files from different location
	than @t{/dev/shm}. For example, these functions are used by a
	@t{wrs_dump_shmem} tool to allow opening shmem files copied from
	another switch.

@item void wrs_shm_ignore_flag_locked(int ignore_flag);

	This function complements function @t{wrs_shm_set_path}. It allows to
	ignore the flag @t{WRS_SHM_LOCKED}. If such flag is not ignored then
	function @t{wrs_shm_get_and_check} is not able to open shmem
	successfully due to lack of process running with the given pid.

@end table

@c ==========================================================================
@node The HAL Process
@section The HAL Process

In the initial days of White Rabbit Switch, developers tried to have
everything managed by a single ``Hardware Abstraction Layer'' process,
the HAL.  The process still exists but its role is smaller than it
used to, so some of the code is likely unused.  After release 4.1
we started a serious audit of the @sc{hal}.

The implementation is currently split between @i{userspace/wrsw_hal}
and @i{userspace/libwr}.  Some of the @i{libwr} functions are
used by other processes, but some are just librarized parts of
the @sc{hal} process (again, we are not very clean and tidy).

If you run @t{ps} in the switch, you'll see there are two @sc{hal}
process, one being a child of the other.  The child is forked by the
@i{mini-rpc} library, to poll shared memory (the communication channel
with the @i{rt} subsystem).  @i{mini-rpc} has one file descriptor per
channel, so users can run @i{select()} or @i{poll()}, thus this trick
to be able to turn a shared-memory channel into a file descriptor.
This is documented in @i{mini-rpc} documentation.  The polling time is
currently 25ms.

The @sc{hal} process is in charge of replying to IPC requests
(@i{hal_exports.c}), driving the fan speed according to temperature
(@i{libwr/fan.c}) and monitoring ports (this is spread in several
files related to @i{i2c} communication like reading SFPs' eeprom, lighting
proper LED in the front panel, etc.).

@c ==========================================================================
@node Reboot/Reset Diagnostics
@section Reboot/Reset Diagnostics

For management and diagnostics, the @sc{wrs} keeps track of the number
of boots since power on. It does so relying on four 32-bit @sc{cpu}
registers that are unpredictable at power-up and remain unchanged over
reboots. They are called ``backup registers'' in vendor documentation
(@t{GPBR}): they actually read 0 on power-up but the documentation
doesn't say anything so we can't count on it.

Our kernel patches use the 16 bytes in the following way; all values
are little-endian.

@table @code

@item bytes 0..2

	A magic number: if not found, then this is a power-on boot.
	(Note: we may use the "reset reason" register as an alternative)

@item byte 3

	Another magic number: it is `@t{R}' (0x52) if the last reboot
	has been requested by an operator.  The value is reset to
	`@t{U}' (for ``unknown'') right after boot.

@item bytes 4..5

	The number of boots since power on. This is at least 1. If zero,
	the code is not working.

@item bytes 6..7

	The number of soft reboots. This is incremented by the
	@i{reboot} system call, and not by accidental reboots
	(i.e. panic or other yet-unforeseen situation).  A healthy
	system should feature one soft-reboot less than total boots.

@item bytes 8..11

	(Not implemented yet).
        The fault address of the last panic. This is the instruction
	pointer normally printed by the stack backtrace. The register
	is zeroed at first boot and only modified within @i{panic}.

@item bytes 12..15

	(Not implemented yet).
	The "link register" register at last panic, like above.
	This is usually the caller of the function that failed,
	but it may be a local register if the failing function
	saved @t{lr} to the stack and used it as a scratch register.

@end table

The kernel exports the current status in @t{/proc/wrs-bootcount}
in a tagged text format for easy parsing.

This is an example, after power-on, one @t{reboot} command and
one hard reset (by pressing the button on the @sc{scb}):

@smallexample
   # cat /proc/wrs-bootcount
   boot_count: 3
   reboot_count: 1
   last_is_reboot: 0
   fault_ip: 0x00000000
   fault_lr: 0x00000000
@end smallexample

@sp 1

We will likely introduce support for the hardware watchdog, but it
is not supported at this time.

We may also consider to use one of the kernel's mechanisms for persistent
storage of information across reboots, to recover panic messages
after a reboot.

@b{Note:} Values of these registers can be read remotely via SNMP in objects:
@t{wrsBootCnt}, @t{wrsRebootCnt}, @t{wrsRestartReason}, @t{wrsFaultIP} and
@t{wrsFaultLR}.

@c ==========================================================================
@node Time keeping over restarts
@section Time keeping over restarts

At normal restart, current time is saved in @t{/update/saved_date}.
Later, at boot switch tries to retrieve correct time from ntp server
if configured in dot-config.
If no correct date can be retrieved via ntp, switch tries to set a date stored
in @t{/update/saved_date}. Please note that at crash, power down or
reset by a reset button no date information is saved. After such restart switch
will set date to last gentle reboot or to 1st of January 1970 if there were no
gentle restarts before.

Date set from file @t{/update/saved_date} is never correct, but is based on
best effort principle.

@c ==========================================================================
@node Supervision of running processes
@section Supervision of running processes

During normal operation @t{monit} supervises several processes running on a wrs
switch. Check is done every 10 seconds. As for now supervised processes are:
@t{wrsw_rtud},
@t{wrsw_hal},
@t{ppsi},
@t{wrs_watchdog},
@t{lighttpd},
@t{sshd},
@t{snmpd},
@t{lldpd},
@t{nslcd}.

In case any of the supervised processes does not run anymore (because of a crash,
exit etc.), @t{monit} restarts missing process. If 5 restarts of a particular
process occurs during 10 cycles (10*10 seconds), the entire switch is
restarted.
The process' name causing a restart is saved in the file
@t{/update/monit_restart_reason} on the flash partition. After next boot this
file is moved to @t{/tmp/monit_restart_reason}, where can be read by i.e. SNMP.
Since it is @t{/tmp} partition, file with restart reason is lost after next
boot.

Since @t{monit} is started from the inittab, even if @t{monit} crashes for some
reason it will be re-spawned by the @t{init}.

@c ==========================================================================
@node Disabling monit
@subsection Disabling monit

In some cases, especially during development it is convenient to disable
@t{monit} to avoid annoying re-spawns of the processes and restarts of the
entire switch.
@t{monit} can be disabled with command:
@example
/etc/init.d/monit.sh stop
@end example
which will send STOP signal to @t{monit}. Additionally @t{monit} will not start
after the boot if there is a config item @t{CONFIG_MONIT_DISABLE=y} in
the dot-config.

To re-enable @t{monit} first make sure there is no @t{CONFIG_MONIT_DISABLE=y} in dot-config, then execute command:
@example
/etc/init.d/monit.sh start
@end example

@b{Note:} Even when @t{monit} is disabled there is a process @t{/usr/bin/monit}
in a process list, but its state is "stopped" (T).

@c ==========================================================================
@node SDB and Hardware Information
@section SDB and Hardware Information

This chapter describes how hardware information is stored and retrieved
in version 4.1 and later. 

There are a number of information items that should be known to the
software, like the MAC addresses, the serial number of the device, and
what FPGA model is mounted on the PCB.  This information must be
available in some storage device, written at manufacture time and
never modified.

The storage device of choice, given the hardware architecture of the
White Rabbit Switch, is the @i{dataflash}.  The data format we chose
is @sc{sdb}, that we are using in a number of situations.

A description of @sc{sdb} is to be found in the @t{ohwr.org} project
called @i{fpga-config-space}. The @i{Self Describing Bus} was born
to describe address spaces (i.e. the cores that are part of an @sc{fpga}
design) but is also a good way to implement a small filesystem
in limited storage.

@c ==========================================================================
@node Hwinfo Placement in Dataflash
@subsection Hwinfo Placement in Dataflash

The @i{hwinfo} @sc{sdb} image in the White Rabbit Switch lives at offset
0x94800 in @i{dataflash}, and is 2112 bytes long (0x840: two or eight
erase regions, according to the device in use).

The area is available as @t{/dev/mtd5} from the Linux kernel, and can
be accessed as a partition from @i{barebox} (the default boot scripts
create it as @t{/dev/dataflash0.hwinfo}).

The binary image includes 4 files, stored as an @sc{sdb} filesystem:

@table @code
@item manufacturer

	The manufacturer name, as an ASCII string.

@item scb_version

	The ``Switch Core Board'' version, which is in the digit.digit
        form, like @t{3.3} or @t{3.4}.

@item eth0.ethaddr

	The MAC address for the management Ethernet port (RJ45, 100Mb/s).

@item wri1.ethaddr

	The MAC address for the first fiber port (SFP, 1Gb/s). Other ports
        are assigned sequential addresses starting from this one.

        @b{Note:} In the switches which were produced before v5.0 firmware
        was released this file contained @t{wr0.ethaddr}.

@item hw_info

	A line-oriented ASCII file including other ``@t{tag: value}''
        information.

@end table

@c ==========================================================================
@node Creating the Hwinfo File
@subsection Creating the Hwinfo File

The @i{hwinfo} file is created using @i{gensdbfs}. The tool is part of
@i{fpga-config-space} and is not included in @t{wr-switch-sw}, because
the package ships a pre-built base image that is then edited in-place.

To re-build the image, please follow the instructions included as
comments in the configuration file, @t{hwinfo-sdb/--SDB-CONFIG--}
and the respective commit message.  You most likely won't need to
rebuild the image, unless you want to add data files or change
the manufacturer name from the default.

A pre-built image is included as @t{binaries/sdb-for-dataflash}.

The script @i{build/wrs_hwinfo} can be used to edit the file
upgrading the MAC addresses and the tagged text file.  The tool
creates a copy of the base file and modifies it in @t{/tmp}.
It finally prints the new file name on @i{stdout}.

The following example with ``strange'' values by design shows how to
use the script, assuming @t{/tftpboot} is the public directory
accessed by the @sc{tftp} server.

@smallexample

  F=$(./build/wrs_hwinfo \
     -m1 00:02:04:06:08:0a \
     -m2 22:33:44:55:66:77 \
     -v 3.3 \
     -x "fpga: LX240T" -n 12345)

  chmod a+r $F
  cp $F /tftpboot
@end smallexample

Please check the source code for details about the command-line options.
The @i{version} argument is mandatory, because the software must know
what version the SCB is (this is not really needed to identify  3.4 from
3.4, but we don't know if we will be able to auto detect 3.5 or 4.0).

Some information items are not really mandatory (the script will
not fail if the are not specified), but should be defined anyways
because @sc{snmp} code retrieves them to tell network administrators.
Currently this only applies to the serial number (@t{-n}).

@c ==========================================================================
@node Storing Hwinfo in a White Rabbit Switch
@subsection Storing Hwinfo in a White Rabbit Switch

You can store your @i{hwinfo} using either the @sc{wrs} shell
or the boot loader. The former technique can be performed remotely,
the latter requires access to the serial
console, on the backplane. 

@c --------------------------------------------------------------------------
@node From the Linux Shell
@subsection From the Linux Shell

The information lives in partition @t{mtd5}, as shown in @t{/proc/mtd}
(Memory Technology Device). To avoid accidental erasure, our
filesystem doesn't include the device file @t{/dev/mtd5}, but only
the read-only counterpart, @t{/dev/mtd5ro}.

If you built your own @i{hwinfo}, with your MAC addresses, and you
copied it to @i{/update} (a good staging place for files copied
over the network), the following commands will place the information
in place:

@smallexample
   test -c /dev/mtd5 || mknod /dev/mtd5 c 90 10
   flash_erase /dev/mtd5 0 0
   cat /update/hwinfo > /dev/mtd5
   rm /dev/mtd5
@end smallexample

The commands create the device file if missing (to prevent an ugly
error message if you already created it), erase and overwrite the
data area, and finally remove the device file.

You new MAC addresses will be effective at the next boot.

@c --------------------------------------------------------------------------
@node From the Barebox Shell
@subsection From the Barebox Shell

If you prefer creating or replacing @i{hwinfo} from the boot loader,
this is the procedure. You can't run it with software version 4.0 or
earlier, because our boot loader was unable to access
@i{dataflash}. V4.1 fixed this problem and introduced the @t{sdb}
command in the bootloader to access your @i{hwinfo} structure.

@smallexample
   erase /dev/dataflash0.hwinfo
   dhcp 5
   tftp hwinfo /dev/dataflash0.hwinfo
@end smallexample

The code above assumes that the @i{hwinfo} file is available from your
@sc{dhcp/tftp} server. According to your configuration one or more
commands may be redundant.

The commands rely on a partition being there, and the default
script already created it using the following command:

@smallexample
   addpart /dev/dataflash0 0x840@@0x94800(hwinfo)
@end smallexample

You can verify successful writing by running @i{sdb ls}, as shown
below.

@c ==========================================================================
@node Accessing Hwinfo at Run Time
@subsection Accessing Hwinfo at Run Time

You can access the hardware information from @i{barebox} using the new
@i{sdbinfo}, @i{sdbread} and @i{sdbset} commands. The following
example shows an example session, using the file built in
@ref{Creating the Hwinfo File} and featuring a simplified @i{barebox}
prompt of ``@t{bb> }''. Please note that most of this is already
in the boot scripts, as we now extract the mac addresses from sdb.

@smallexample
  bb> addpart /dev/dataflash0 0x840@@0x94800(hwinfo)
  bb> sdb ls /dev/dataflash0.hwinfo
  46696c6544617461:2e202020 @ 00000000-0000083f .
  46696c6544617461:7363625f @ 00000240-00000243 scb_version
  46696c6544617461:7772302e @ 00000220-00000231 wri1.ethaddr
  46696c6544617461:6d616e75 @ 00000260-0000026f manufacturer
  46696c6544617461:68775f69 @ 00000420-0000083f hw_info
  46696c6544617461:65746830 @ 00000200-00000211 eth0.ethaddr

  bb> sdb cat /dev/dataflash0.hwinfo manufacturer
  Seven Solutions
  bb> sdb cat /dev/dataflash0.hwinfo hw_info
  fpga: LX240T
  scb_serial: 12345
  bb> sdb set /dev/dataflash0.hwinfo wraddr wri1.ethaddr
  bb> echo $wraddr
  22:33:44:55:66:77

  bb> sdb set /dev/dataflash0.hwinfo eth0.ethaddr
  00:02:04:06:08:0a
@end smallexample

@i{Barebox} passes the MAC address information to the Linux kernel by
setting proper environment variables using @i{sdbset}.

After boot, @i{sdb} can be accessed using the @i{sdb-read} command
(that this package copied from @t{fpga-config-space}.  The @sc{snmp}
code accesses the files directly, by linking the @i{libsdb} code base
-- again, from @t{fpga-config-space}.


@c ##########################################################################
@node Schematics are Available
@appendix Schematics are Available

The switch schematics for all PCB versions (3.x of the
SCB as well as both 3.1, 3.2 and 3.3 of the backplane)
are available on the Open Hardware Repository, at
@uref{http://www.ohwr.org/documents/180}, which can also be reached
from the @i{Documents} tab of the @i{White Rabbit} project.

Please note that only version 3.2 and 3.3 of both the motherboard and 
the backplane has been shipped commercially; you are interested in 
previous versions only if you are an early developer and have one of 
those in your hands.

@c ==========================================================================
@node DIP Switch HW version
@section DIP Switch HW version

Since v3.3, the backplane includes a DIP switch configured by the 
manufacturer in order to define a specific SCB and backplane 
version. This setup is then read by the software in order to load 
the correct FPGA binaries and use the proper I/Os. Please be aware 
that if you upgrade your SCB from LX130T to LX240T but keep the same 
backplane you might need to change the DIP switch configuration. 
Check the code from @code{userspace/libwr/i2c_io.c} code to 
know how to reconfigure the DIP switch for you upgraded device.

For example, the v3.3 backplane with v3.3 LX240T SCB must be configured as bellow:
@example
+--------------+---+---+---+---+
| DIP position | 1 | 2 | 3 | 4 |
+==============+===+===+===+===+
| DIP value    | 1 | 1 | 1 | 0 |
+--------------+---+---+---+---+
@end example

@c ##########################################################################
@node Installing from Jtag
@appendix Installing from Jtag

As an alternative to the serial flasher, you can take control of the
system with a JTAG debugger. Please note that 
the @i{USB Flasher} is @b{really} the preferred technique, but in case
it doesn't work for you, JTAG is the only way to
communicate with the switch.

Previous versions of this manual included detailed instructions about
such recovery procedure, but we have not been using JTAG for a long
while, so we didn't update the information to the V4 filesystem layout.

If you need to boot from JTAG, please refer to documentation in
version 3.3 or earlier of @t{wr-switch-sw} for generic ideas, knowing the
details are different.

@c ##########################################################################
@node Bugs and Troubleshooting
@appendix Bugs and Troubleshooting

Even if the package is already released and used in production,
some details can be
suboptimal, while some procedures may be tricky and need more explanation.
We are collecting all those issues in our project pages. Please visit:
@itemize
   @item Frequently Asked Questions: @url{http://www.ohwr.org/projects/white-rabbit/wiki/FAQswitch}
   @item Issues for WR Switch SW project: @url{http://www.ohwr.org/projects/wr-switch-sw/issues}
   @item Issues for WR Switch HDL project: @url{http://www.ohwr.org/projects/wr-switch-hdl/issues}
@end itemize
If you have any problem with this firmware and you don't find help in the above
links, feel free to reach us on the @i{white-rabbit-dev} mailing list.

@c ##########################################################################
@bye


@c  LocalWords:  gnudd titlepage iftex texinfo CERN timestamping smallexample
@c  LocalWords:  LocalWords ietf timestamp misc timestamps ttstamp onestamp
@c  LocalWords:  Tomasz Wlostowski buildroot distclean defconfig wrswitch REPO
@c  LocalWords:  menuconfig config dataflash whiterabbit stdout stderr svnsync
@c  LocalWords:  filesystem diff ohwr http mkdir linux rubini itemize PTPd VHDL
@c  LocalWords:  noposix ptpd userspace libwr DataFlash NAND barebox FPGA
@c  LocalWords:  Atmel Kconfig minicom tinyserial ttyUSB bootloader logfile
@c  LocalWords:  nandflash gateware TFTP init wrboot wiki pathname
@c  LocalWords:  subdirectory filename configs pathnames multicast
@c  LocalWords:  DHCP submodule minipc ppsi vlan shmem hwinfo ethaddr
