Josef Vojtech, Stanislav ma, Jan Radil,

1
Dark Fibre Facilities for Research and
Experimentation
www.ces.net

• Josef Vojtech, Stanislav íma, Jan Radil,
• Lada Altmannová
• josef.vojtech_at_cesnet.cz

2
Dark Fibre Facilities for Research and
Experimentation
Authors participate on CESNET research
program (www.ces.net), GÉANT2/3 project
(www.gean.net), Phosphorus project
(www.ist-phosphorus.eu/about.php) Presented
content do not necessarily reflect an official
opinion of any institution or project.
3
Dark Fibre Facilities for Research and
Experimentation Outline

• Introduction
• Experimentation on Physical Layer
• Concepts Used
• Dark Fiber
• Nothing in Line
• Open Photonic Devices
• CESNET2
• Experimental Facility Applications
• Connecting Dispersed Users to EFs
• Conclusions

4
Dark Fibre Facilities for Research and
Experimentation The Czech Republic

• Area
• 78 866km2
• 30 450 sq mi
• (South Carolina)
• Population
• 10 349 372
• (Michigan)

source http//en.wikipedia.org/wiki/Czech_Republi
c
5
Dark Fibre Facilities for Research and
Experimentation CESNET

• CESNET - Czech Educational and Scientific Network
• Established as Association of Legal Entities
  (z.s.p.o.) in 1996 not for profit
• All public universities Czech Academy of
  Sciences
• 52 staff members in Prague over 150 part-time
  staff working on projects
• Funded by Ministry of Education, Youth and Sports
  of the Czech Republic and association members
  under research plan Optical High Speed National
  Research Network and its New Applications
  (2004-2010)

6
Dark Fibre Facilities for Research and
ExperimentationExperimentation on Physical Layer

• Understanding of physical layer is crucial
• There are no applications without physical layer.
• Technology used in physical layer limits
  overlying applications by transmission speed,
  latency, non-determinism etc.
• There is electronic processing speed limitation
  (about 100 Gb/s per port), optical processing
  speed limitation will be about 1 Tb/s per port.
  Optical processing has significantly lower energy
  consumption. See for example presentation Tetsuya
  Miyazaki Node and Link Technologies for New
  Generation Networks, NICT, http//www.ict-firewo
  rks.eu/fileadmin/events/9-10_June_2008_Brussels/Pr
  esentations/09-06-08/5B-Photonic_Networks/15-TETSU
  YA_MIYAZAKI_-_Photonic_Networks.pdf

7
Dark Fibre Facilities for Research and
ExperimentationExperimentation based on physical
layer

• Should be based on technology advances.
• Many possibilities to enhance applications can be
  find in physical layer.
• In many cases we can gauge before network users,
  what new physical layer possibilities will bring
  improvements in future.
• In Research and Development driven by
  experiments, it is important to verify and
  evaluate real possibilities, create new offer of
  services for users, and find early adopters.
• Applications can be enhancement by connecting
  dispersed users to experimental facilities.

8
Dark Fibre Facilities for Research and
Experimentation Concepts used

• Concepts used for building
• Operational National Research and Educational
  Network (NREN) - CESNET2
• Experimental Facility (EF) CzechLight, part of
  Global Lambda Integrated Facility (GLIF), see
  http//www.glif.is
• Dark Fiber (DF) Customer Empowered Fiber (CEF)
  Networks
• Nothing in Line Approach (NIL)
• Cross Border Fibers (CBF)
• Family of open photonic devices CzechLight
  Family

9
Dark Fibre Facilities for Research and
Experimentation Dark Fibres, Customer Empowered
Fibres

• Migration from telco lambda services to dark
  fibers brings
• Much more capacity
• Freedom in network design
• DFs really needed for E2E services
• First mile issues with fibers in many places
• Wireless can be used too but limitations may be
  rather significant (no WDM, 10G predicted, but
  higher speeds?, reliability and availability...)
• First DF in CESNET in 2000, 2.5 Gb/s PoS
• Approximately 300km (186mi), 3 regenerators
• More DFs have been acquired since then
• First Erbium Doped Fiber Amplifiers (EDFAs)
  operated DF in CESNET in 2002, 1 GE.
• Approximately 190km (118mi), NIL concept
• DF line in service till upgrade to 10 Gbps O-DWDM
  system in 2008
• Ethernet-based E2E services, shared capacity,
  VLANs
• Transition of backbone to DFs was finished in 2004

10
Dark Fibre Facilities for Research and
Experimentation Nothing in Line

• Nothing in Line (NIL), hut skipping
• Original approach, hut each 80km (50mi) - not
  necessary always and everywhere
• Distances can be extended, even with commercially
  available equipment
• Reduction in CAPEXOPEX (less HW, housing, power)
• Examples NIL operational lines
• 235km (146mi) EDFA only originally 1GE, upgraded
  to 10Gbps O-DWDM
• 308km (191mi) EDFA Raman

11
Dark Fibre Facilities for Research and
Experimentation Cross Border Fibres

• Many projects and traffic with your neighbor?
  Acquire/rent a DF - latency will be low.
• In CESNET
• CBF SANET Slovakia 1GE in 2003, upgrade to Open
  photonic DWDM (O-DWDM) in 2006
• CBF PIONIER Poland 1GE in 2004, upgrade to DWDM
  in 2007
• CBF ACONET Austria O-DWDM, in 2006 O-DWDM

12
Dark Fibre Facilities for Research and
ExperimentationOpen photonic devices for network
research and experimentation

• CESNET is developing family of open photonic
  devices (CL family), enabling early adoption of
  leading edge photonic technology
• Devices support network research and
  experimentation on dark fibre level, fast testing
  and prototyping, support experimental
  applications requested by users or field
  experiments on dark fibre lines, verifying
  feasibility of network improvements.
• Proved very useful for CESNET2 development,
  Experimental Facility development, CBF lighting,
  GLIF applications development (especially for
  dispersed end users), improvement of
  interoperability on the physical layer, as well
  as remote monitoring and control, low latency and
  deterministic multicast (will be addressed
  further).
• Main advantages offer freedom of design, ability
  to meet user needs, easy to modify if change
  necessary, avoiding delays in innovation (low
  needs to save investments), photonic transmission
  and processing speeds, low cost, saving energy,
  and space.

13
Dark Fibre Facilities for Research and
Experimentation Family of Open Photonics Devices

• First Open Photonic devices in CESNET2 in 2004
• Family of open photonic devices (DWDM and other
  photonic tasks)
• EDFAs (terminal, inline applications, CATV)
• Raman amplifiers including TDM (amplification in
  long NIL)
• Tunable CD compensators (long NIL, 40G)
• Reconfigurable Optical Add/Drop Multiplexers
  (ROADMs)
• Variable Mux/Demuxes (VMUX)
• Wavelength selective switches (WSS)
• Optical channel monitors (OCM)
• All optical wavelength converters
• Photonic path switches (both mechanically and non
  mech. based)
• Photonic path switches with multicast option
  (mech. non mech.)
• Manufactured by FTTx companies/vendors.

in experimental regime
14
Dark Fibre Facilities for Research and
Experimentation Progress in CESNET networks

• Evolution of the CESNET network 1999 2007 (not
  with all details).

2004
2000
2002
2005
2007
15
Dark Fibre Facilities for Research and
Experimentation Up-to-date situation in CESNET2

• CESNET2 network 2009
• 4700km (2920mi) of DFs
• Incl. 980km (609mi)
• single fiber DF links
• Mixed architecture coexistence of commercial
  and open DWDM systems at speeds up to 10Gbps

16
Dark Fibre Facilities for Research and
Experimentation Up-to-date situation in CESNET
WAN

• 10 lines Cisco 15454 MSTP n x 10 Gb/s DWDM
  transmission system (with ROADM).
• 10 lines deployed with open photonic n x 10 Gb/s
  DWDM transmission systems, including 2
  international CBF connections to Bratislava and
  Vienna
• No bottleneck on international links anymore.
• 6 single fibre lines deployed with open photonic
  DWDM
• Other 4 links will be lit in 2009 (including 3
  single fibre lines).

17
Dark Fibre Facilities for Research and
Experimentation Experimental facility WAN
18
Dark Fibre Facilities for Research and
ExperimentationExperimental facility -
connections
CBF 8 x 10 Gb/s
Masaryk University BRNO
GOLE CzechLight Praha
N x 1GE over SONET
CBF 4 x 10 Gb/s
ONS15454
ONS15454
OC-192c
40 x 10 Gb/s
N x 1 GE
CBF 8 x 10 Gb/s
Ethernet VLANs
Cisco6506
E300
OC-192c
N x 110 Gb/s
CESNET2 DWDM backbone
BigIron
Cisco7609
19
Dark Fibre Facilities for Research and
Experimentation GLIF
20
Dark Fibre Facilities for Research and
ExperimentationConnecting dispersed users to EFs

• Preferred approach is based on dark fibres
  connecting end users and EF via photonic
  (all-optical) lightpaths (i.e. implemented
  without OEO conversions).
• Tunneling through CWDM was successfully
  demonstrated (but not yet fully published) for
  example in GLIF2007 workshop in Prague as
  connection between Charles University and EF,
  used for all demos.
• Photonic lightpath connected to EF are available
  in CESNET2 NREN and in CESNET Experimental
  Facility, including CBFs.

21
Dark Fibre Facilities for Research and
Experimentation Experimental facility
22
Dark Fibre Facilities for Research and
Experimentation GLIF 2007 demos solution
N x 10GE
?1 ?2 ?3 ?4
?1 ?2 ?3 ?4
GOLE CzechLight Praha
PASNET
DWDM MUX
CWDM MUX
CWDM DEMUX
DWDM DEMUX
Carolinum
CLA PB01
Dark Fibre, G.652, 13 dB (DWDM over CWDM)
?1 ?2 ?3 ?4
?1 ?2 ?3 ?4
Demos PCs with XFP Switches
E300
DWDM DEMUX
CWDM DEMUX
CWDM MUX
DWDM MUX
OC-192c
OC-192c

• Combination of
• 10G international lambdas, CWDM, DWDM
• Open and big vendor transmission systems

23
Multicasting device demonstration
Dark Fibre Facilities for Research and
Experimentation

• CLM (Multicast Switch) has
• photonic part multicasting (replicating) optical
  signal bands
• electronic part for remote switching control via
  web interface
• Multicast by CLM is transmission speed agnostic,
  deterministic without jitter and delay, i.e.
  without OEO conversion, without store and forward
  processing and without loading of switch by
  multiple tasks
• Operation demonstrated at GLIF 2007 at Prague
• Operation demonstrated outside CESNET on
  University of Washington (8th Annual GLIF
  Workshop on 1 October 2008 at Seattle), CLM was
  located at StarLight and managed remotely

24
Dark Fibre Facilities for Research and
Experimentation Photonic multicast with CLM
25
Dark Fibre Facilities for Research and
Experimentation
PNWGP
7609
CAVEwave (2155)
EVL e600
CESNET (trunk 440,441)
CzechLight
6/2
Praha E300
CW 6506
6/6
6/10
6/14
(441)
(445)
(2155)
CW 6506
TX only
RX only
RX only
C-Wave (2155)
RX only
GLIF Mog HD RX
(2155)
O1
I1
O2
CESNET CLM
Brno C6506
O3
IP 10.200.200.200 MAC 101010101010
C-Wave (2155)
O4
(440)
(441)
HD RX
HD TX
StarLight
IP 10.200.200.200 MAC 101010101010
UCSD Calit2

• Combination of
• 10G international lambdas, DWDM
• Open and big vendor transmission systems

CW 6506
6509
e1200
GLIF 2008 CLM Demonstration in Cinegrid demos
HD RX
IP 10.200.200.200 MAC 101010101010
26
Dark Fibre Facilities for Research and
Experimentation Applications over EF

• GLIF demo on 8th Annual Global LambdaGrid
  Workshop 2008, Seattle, USA.
• HD video with real optical multicast.
• GLIF demos on 7th Annual Global LambdaGrid
  Workshop 2007, Praha, Czech Republic.
• HD, 4k video transmissions.
• Multiple 10 Gb/s links to a medieval building
  with open DWDM transmission system.
• HEP - data access and processing for ATLAS and
  ALICE on LHC, D0 on TEVATRON, STAR on RHIC.
• First VINI sites in Europe (Praha, Plzen).
• http//www.vini-veritas.net/
• Intercontinental Remote Education on High
  Performance Computing between Masaryk University
  Brno and Louisiana State University.

27
Dark Fibre Facilities for Research and
Experimentation GLIF Lightpaths (E2E) to HEP
projects participants in the Czech Republic
GOLE NetherLight Amsterdam
AS The Academy of Sciences of the Czech
Republic CTU Czech Technical University,
Praha CU Charles University, Praha
n x 1GE over SONET
ONS15454
OC-192c
Access 1GE
Nuclear Physics Institute (AS) Re
Trunk 10GE
DF
TAIPEI
BNL
FNAL
Faculty of Nuclear Sciences and Physical
Engineering (CTU) Brehová, Praha
Institute of Physics (IoP AS), Na Slovance, Praha
CWDM
Faculty of Mathematics and Physics (CU), Trója,
Praha
DF
Cisco6506
E300
10GE
CWDM
CWDM
Institute of Experimental and Applied Physics
(CTU) Horská, Praha
CWDM
GOLE CzechLight Praha

• Combination of
• 10G international lambdas, CWDM
• Open and big vendor transmission systems

DF
Nuclear Physics Institute (AS) Bulovka, Praha
28
Dark Fibre Facilities for Research and
Experimentation
GOLE CzechLight Praha
vini1.cesnet.cz
1GE
E300
1GE
GOLE NetherLight Amsterdam
vinix.cesnet.cz
1GE over SONET
1GE
1GE over SONET
OC-192c
ONS15454
1GE
OC-192c
simple extension possibilities
Lightpath Chicago/Amsterdam/Praha for VINI
CESNET2 PoP Praha
GOLE StarLight Chicago
CESNET EF n x 10Gb/s
CESNET2 n x 10Gb/s
Plzen
Brno

• Combination of
• 10G international lambdas, DWDM
• Open and big vendor transmission systems

CESNET ONS15454
CESNET2 ONS15454
CESNET2 ONS15454
vini network in the USA
1GE
1GE
1GE
viniy.cesnet.cz
vini2.cesnet.cz
viniz.cesnet.cz
29
Dark Fibre Facilities for Research and
Experimentation Conclusions I

• Attention to photonic transmission and processing
  should be increased
• Speed limits of electrical processing, above
  certain limit the only possibility
• Ability to make savings on energy consumption,
  housing space. Should be addressed by the
  physical layer network design the possibility of
  addressing them on the higher layers is very
  limited
• Design of E2E lightpaths multidomain task
• Necessity of interoperability at physical layer
• Theoretical understanding and experimental
  verification (in DF experimental facilities)
  necessary to verify feasibility of deployment in
  the production network
• Cooperation with FTTx vendors and operators
• Open photonic systems
• Can enable early adoption of the new photonic
  technology in the production
• Enable and support experimental applications
  requested by users or field experiments on dark
  fiber lines
• Can offer beneficial cost advantages

30
Dark Fibre Facilities for Research and
Experimentation Conclusions II

• The development and interconnection
  (federalization) of experimental facilities (open
  testbeds) enabling field technology testing and
  user participation should be supported
• Give researchers an experimental environment for
  validating innovative and potentially
  disruptive architectures and technologies
  including physical layer. Many issues are only
  discovered when technology, devices or systems
  are deployed in "real-life" situations.

31
Dark Fibre Facilities for Research and
Experimentation Call for cooperation

• Interconnection of (dark fiber) experimental
  facilities, enabling researchers access to these
• Experimental verification and deployment of
  cutting edge solutions and technology

32
Dark Fibre Facilities for Research and
Experimentation Kyoto Price 4K transmission (via
CESNET)

• The Kyoto Prize is an international award to
  honor those who have contributed significantly to
  the scientific, cultural, and spiritual
  betterment of mankind.
• 4K uncompressed live streaming (6 Gbit/s)
• Kyoto to Stockholm, L2-10GbE over 21,000km.
• 4K compressed multicast streaming (500 Mbit/s)
• Multicast by hardware packet replicator
• Worlds First Trans-Pacific and Trans-Atlantic
  (21,000km) Real Time Switching and Streaming
  Transmission of Uncompressed 4K Motion Pictures,
  Nov 10-11, 2007 from Kyoto to Stockholm via
  Chicago and Prague
• http//www.dmc.keio.ac.jp/en/topics/071126-4K.html
  

33
Dark Fibre Facilities for Research and
Experimentation References

• 1 Petr Holub, Josef Vojtech, Jan Radil, et.
  al., Pure Optical (Photonic) Multicast, GLIF
  2007 Demo, Prague, 2007.
• 2 Jan Radil, Stanislav íma, Customized
  Approaches to Fibre-based E2E Services, TERENA
  1st E2E Workshop, Amsterdam, 2008.
• 3 Stanislav íma, et. al., LTTx Lightpaths
  to the application, From GOLEs to dispersed end
  users , GLIF 2008 Workshop, Seattle, 2008.
• 4 Josef Vojtech, Jan Radil, Transparent all
  optical switching devices in CESNET, 25th APAN
  meeting, Honolulu, 2008.

34
Dark Fibre Facilities for Research and
Experimentation Acknowedgement

• Jan Gruntorád, Miloslav Hula, Jirí Navrátil, Jan
  Nejman, Václav Novák

35
Dark Fibre Facilities for Research and
Experimentation Thank you for your
attention.Questions?
36
Dark Fibre Facilities for Research and
Experimentation Invitation to Future Internet
Conference 11-13 May 2009, CLARION CONGRESS
HOTEL Prague
Organized by the EC and CESNET during the Czech
Republic presidencyhttp//www.future-internet.eu
/events/eventview/article/eu-conference-the-future
-of-the-internet.html
37
Dark Fibre Facilities for Research and
Experimentation Invitation to 5th Customer
Empowered Fiber Network WorkshopMay 14 15,
2009 Praha, Organized by CESNET
Previous workshops http//www.ces.net/doc/seminars
/cef2007/
Universitas Carolina, founded 1348.
38
Dark Fibre Facilities for Research and
Experimentation GLIF 2007 photonic multicast
with CLM