Test Suite for end-to-end performance tesing of access networks
From 2004-2008, I was member of the Board of MUSE, a huge European consortium focussing on the next generation of broadband access networks.
It brough many system Vendors, Operators, Research Institutes and Academia together from all over Europe, with a very good spirit of cooperation.
The consortium was funded from the FP6 program of the EU. This funding enabled an involvement of about 100 personyear per year, during a period of 4 years.
As a result, each MUSE meeting (4 times a year) had a size and organisation of a large conference.
Such meetings involved multiple sessions in parallel, divided over several Workpackages and Taskforces.
In that period, I was chairing a multidisciplinary team of experts, covering all layers of the OSI model.
They worked together in a taskforce (TF4) to identify good methods for analysing and verifying the end-to-end performances and capabilities
of an access network as a whole.
This joined effort resulted in a TestSuite of more than 670 pages, as disclosed below.
MUSE Test Suite for
Full Service End-to-End Analysis of Access Solutions
The demand for delivering high bandwidth multi-media
applications to end users, via multiple service providers, network
operators and technologies, has driven the world wide development of a
new generation of access networks.
A wide range of “knowledge tools” for analyzing the
strong and weak points of an access network.
A single source of reference to relevant standards,
methods, techniques and terminology.
Wide scope: covers the full OSI stack and much more:
access networks have to meet so many requirements that they have become
very complex. To verify if these networks are adequate for their
purpose, you cannot limit yourself to a set of isolated tests on
individual devices. In those cases a full service end-to-end approach
should be followed to test the system as a whole. So how to analyse the
most relevant capabilities and shortcomings of such a system?
this question becomes relevant, for instance, when someone needs to
make migration decisions toward new network solutions. In such a case
he/she wants to benchmark one solution against many alternatives.
Usually, this starts with defining functional requirements, followed by
the creation of RFI/RFQ documents.
But do those requirements
address the problem as a whole and in sufficient detail? And how to
find the most relevant selection criteria, how to identify which
additional tests are needed, how to interpret these test results, and
how to transform that into a well balanced evaluation of the offered
solution? In those cases, you definitely could use some help.
answers are not obvious. If a system has to deliver services to end
users via channels that are sometimes overloaded with other traffic, or
via unreliable channels, the system may be adequate for downloading
video files but inadequate for streaming video. This emphasises the
need for a full-service end-to-end approach to evaluate a system.
the public release of the MUSE Test Suite, no single source of
reference existed to analyse a system as a whole, from so many
technological viewpoints. The guidance from this Test Suite offers you
a holistic top-down view on what characteristics of a system are really
relevant, and how you can identify them.
Some of the details can
also be found in standards. This may be true for testing physical layer
aspects of a system or for testing individual elements or sub systems.
However, when higher layer characteristics are to be analysed, the
required information becomes scarce or is not available. And since
there are so many standards out there, where should one start?
MUSE Test Suite was designed specifically to address such challenges:
what to test and how to test. It is a comprehensive document that
provides you with a holistic system view on testing and to evaluate the
capabilities of access systems. It describes a wide range of test
objectives in significant detail, and guides the reader through
state-of-the art views and many standards (when appropriate).
A test suite in two parts:
• Part 1,
is on test objectives, and
identifies what to test (over 300 pages).
• Part 2,
is on test methods, and
identifies how to test (over 375 pages).
first part provides lots of guidance to explain the network
characteristics that should be analysed, and to summarize what
information is already available in various standards. It outlines
several classification and description models of access networks from
different perspectives (the well-known OSI layer model can cover only a
limited set of functionalities). It summarizes a variety of service
requirements as well as to introduce the various chapters on testing.
The second part discusses detailed test procedures and methods to
analyze the characteristics identified in part 1.
reference for the assessment of the QoS for various services, as
perceived by users. The first part describes which aspects should be
tested when assessing the performance of VoIP, Streaming Video,
Videoconference, Interactive Gaming en web browsing. The second part
discusses, in detail, tests for obtaining the perceived QoS for these
services. Read more ....
A top-down view on
connectivity, starting from a service point of view. Multiple classes
of services are described and for each class an example service (e.g.
high speed internet, multicast streaming, VoIP) has been used to show
how service testing can be done from the service down to the network
layer. Of course, this chapter is not exhaustive on all services. Table of contents of part 1 and part 2
bottom-up view on connectivity, starting from the architecture. It
describes relevant tests and standards for the network layer and
upward. It concentrates on functionality and performance tests without
detailing on specific test procedures; it excludes the physical layer.
As it is focused specifically on the MUSE architecture, it is a lot
more specific than the chapters on service connectivity. Table of contents of part 1 and part 2.
testing of RGW’s
reference for network and higher layer tests on residential gateways.
It identifies detailed objectives for testing IGMP, IGMP proxy, DHCP
and PPPoE functionality and conformance. These objectives are based on
requirements gathered from DSL-forum and IETF.
The second part
describes the test procedures in more detail, as they were used and
verified while implementing an automatic test suite on residential
gateways. Table of contents of part 1 and part 2.
A detailed mapping of the
TMN management functions described in ITU-T M.3400, into an exhaustive
list of requirements. The readers are made familiar with the TMN model
and its management functions, so that they can understand why MUSE has
chosen TMN as its reference model for testing Management systems. Read more ....
comprehensive overview on the xDSL subjects that are really relevant,
plus the motivation why. It covers all flavours of xDSL, and gives an
up-to-date guidance through the huge amount of available tests in
standards (ITU, DSLF, ETSI).
Part two refers to existing standard
tests and concentrates on describing missing details. Furthermore, it
adds methods on recent topics, such as means for loop qualification
(DELT/SELT), mitigating impulsive noise, and protecting legacy systems
(PSD shaping). Read more ....
A top-down view
on testing of generic system aspects that is characteristic to
fibre-based access networks. It concentrates on optical signals and on
the performance of an access system as a whole (leaving tests for
underlying optical components out of scope). Systems are differentiated
with respect to their interfaces (analogue, digital, framed digital)
and topologies (point-to-point, point-to-multipoint). Test
methodologies are identified for each of them. Specifically, it covers
tools and procedures available to measure relevant physical layer
parameters, referencing appropriate standards. Read more ....
is a European consortium of vendors, operators and universities, active
from January 2004 - March 2008. The aim is cooperation on research and
development of future, low cost, multi-service access networks.
MUSE is partly funded from the FP6 programme of the European Commission
and this Test Suite is one of its deliverables (DTF4.4).
each application we have listed the factors that influence the perceived
QoS. Following ITU-T P.800, the perceived QoS will
be rated on a five point scale, the so-called Mean
Opinion Score (MOS),
see the table below.
systematic way of testing the management capabilities of a system as a
whole is to follow the structure of the TMN
model, defined by the
ITU. This model provides an overall architecture framework for
analying management functions (vertical view) as well as TMN
The horizontal view follows the traditional way of describing element
management functions (see figure 1). This method is the so-called
FCAPS method, and separates between management functions for Faults,
Configuration, Accounting, Performance and Security.
vertical view covers various management levels, starting from the
of service providers to support their business processes, downto the
of individual network elements. The MUSE management model is a mixture
of the so
called eTOM model
(for higher layers) and the M.3400 model (for lower layers) and
distincts between Business management, Service management, Network
management, Element management and the individual network elements.
The Test Suite provides the reader with an exhaustive framework for
testing, that is well prepared for future developments, and not
restricted to MUSE-only architectures.
Figure 1. TMN and eTOM models
1 of the Test Suite
starts with the motivations behind the selection of the TMN model as a
provides an introductory part,
focusing on the
underlying FCAPS concepts, and prepare for a detailed step-by-step
Management Testing approach. It summarizes a range of test objectives,
covering all capabilities needed to manage a system as a whole.
2 starts with a summary
management functions that are described in
ITU-T M.3400. It maps all its requirements to an exhaustive list
of functionalities that are to be evaluated. The structure of
proposed approach assures that no aspect should be missed. In fact,
part 2 identifies all test steps needed to test
the objectives summarized in part 1.
parts are complementary in nature, and have common
goals. The aim is to validate management capabilities according to TMN
model, and to provide an adequate framework for this. The approach is
generic in nature (not restricted to MUSE-only), and enables a setup
implementation that facilitates rapid tesing of management
functions for all practical cases.
The prime objective of the xDSL chapter in part 1 is to give a
overview on testing xDSL, written in a tutorial style. The reader is
guided through all the topics that are really relevant, leaving
details and complexity to the various xDSL standards. The document is
to the standards; adding new tests, giving additional explanation and
provides references to further reading where meaningful.
Figure 1: xDSL brings broadband
connections to millions of European households.
Part 1 starts with the different
approaches of analyzing xDSL systems,
followed by a step-by-step guideline along relevant test
in terms of functional tests. Functional tests are to identify if a
specific xDSL based
solution can serve as an adequate transport platform for a full
service, end to end access network. Possible applications for
functional tests are (a) to debug solutions that are under development,
(b) to identify strong and weak points of a solution to enable
strategic decisions on migration scenarios, or (c) to select one
solution out of many offers from different vendors.
Figure 2: Basic test set-up for
testing under differential mode injected impulsive noise.
Part 2 is very different from part 1. This part is written in a
procedural style for guiding the test engeneer, step by step, through
the associated xDSL
test methods. Its style is similar to what is common
for various DSL
Forum test documents, with the difference that part 2 concentrates on
functional tests. The (comprehensive) DSL forum documents have in
common that they mainly concentrate on conformance
All objectives identified in part 1 have been elaborated in
Figure 3: Photo of an
enlarged Dutch street cabinet, that demonstrates how xDSL systems are
moving closer to the customer location. The enlargement was required to
accomodate VDSL2 equipment.
Some of the new topics that are discussed in both parts are:
The need for PSD shaping
to enable VDSL2 to coexist with ADSL in the same cable, and how
the compliance of shaping with access rules can be verified.
Why the demand for transporting video services has
made test of the immunity against impulse
noise so important.
Developments on new capabilities
to manage the advanced features that have been
introduced since ADSL2+ and/or VDSL2. Examples are PSD shaping, Impulse
noise protection and Loop diagnostics.
The organisation of tests for analyzing fiber-based systems follows a
strict top-down approach. Tests that are relevant for systems that are
to transport analog TV channels over fibre, may not be suitable for
digital systems with passive optical point-to-multipoint (P2MP)
To cover all these systems by a single Test Suite,
the description of tests objectives has been grouped in part 1 as
tests are tests that can be applied to
all fibre-based systems. They are to analyse capabilities of
a system to manage and monitor the link, to survive from fibre breaks
(due to redundancy) and to prevent security related issues.
A second group of tests are those that are dedicated
to systems with specific external
It may be obvious that systems with analog, digitial and/or framed
digital interfaces have very different requirements.
Another group of tests are dedicated to systems with
specific internal (optical)
interfaces. Think of different
modulation formats, wavelengths schemes and network architectures (P2P,
P2MP). These tests take into account that the optical signal quality
evaluations and/or the total transmission performance can deteriorate
by different phenomena.
What all the above tests have in common is that they aim to analyze
fibre-based networks from a system point of view. A full
characterisation of individual elements (such as devices, modules and
subsystems) is considered as out of scope.
2 summarizes a variety of test
methodologies to verify the capabilities identified in part 1. It
provides the reader with many references to test methods that have been
described in public documents, appropriate standards and application
1: Impact of ORL (produced by different configurations of splitters in
P2MP architecture) on ONT sensitivity
To enable a fibre-based network to perform above pre-defined
performance requirements, it is essential to characterize several
performance-limiting characteristics of the submodules within a link.
involves measurements of all kinds of key parameters of transmitters and receivers, as well
as power budget
calculations of the link. It also involves measurements on
limiting characteristics of passive optical
fibre plants, such as for instance the spectral response of optical
filters (see figure 2).
2: Spectral characteristics of ODN- Transmission performance of an 18
channel CWDM filter (ITU G.694.2)
The general survey in the fibre-optic chapters provides
support to vendors, operators, students, designers and experts that
need to analyse access
solutions as a whole rather than stand alone testing of individual