MAX experience with capacity planning

1
MAX experience with capacity planning

• Dan Magorian, MAX
• Director of Engineering and Operations
• magorian_at_maxgigapop.net

2
Planning?? You expect us to be able to plan in
this environment?!!
3
No, seriously

• Actually, I wanted to capture a conversation I
  had with Tim Lance (Nysernet) yesterday,
  following his presentation on construction of
  their 32 AoA NYC pop, and following Thurs
  business model panel.
• Most RONs and XPs are run as businesses. However
  much were part of higher ed, however much of a
  mission we have to facilitate research, in the
  long term our growth and sustainability come down
  to making good business decisions.
• We have income from our participants, and
  expenditures building and running our networks.
  Those better at least balance or you wont be
  around long, unless youre a funded research
  project.
• Whether youre trying to decide whether to build
  out a new pop, or set service pricing, are both
  fundamentally business decisions.

4
So what is a business decision?

• A fancy term for an educated gamble.
• You try to figure what you need to do, then you
  go for it (or not) and make it happen. At the
  end of the day, you see whether you were right or
  wrong. During the day, therere sometimes
  options to change or bail, and sometimes not.
  But theres always exposure.
• For example, 3 years ago we pulled the trigger on
  a new suite at Level3 in Mclean. We were in a
  dilemma because only a tiny suite was left on the
  first floor, and the second floor didnt have dc
  power. We decided to go for it, which in 20/20
  hindsight was the wrong decision, because we
  rapidly outgrew it and now are facing a complex
  migration to an expansion space. Also, we
  decided to place a T640 in Mclean, which some
  people opposed because where NewNet would be was
  unknown at the time, but which turned out to have
  been a right guess. Lose some, win some.

5
The point about planning is

• Its the process of educating yourself about the
  business decisions (gambles) youre going to
  take, so that you win more than you lose
• Whether its 500k for a router or 12M for a new
  pop, you have financial exposure upfront before
  you know whether you guessed right or wrong.
  This is the part most of us dont like very much.
  While people in higher ed arent usually fired
  for white elephants, in companies execs making
  wrong 1B merger decisions are.
• So whatever the amount, you sometimes find
  yourself betting the farm. Gambling a lot of
  your organizations cash on a business decision
  that looks good to the best of your planning
  ability, but which may not turn out to be right,
  often for reasons not under your control. Build
  it, and hope that you guessed right that theyll
  come.

6
So what does this philosophy about business
decisions have to do with optical capacity
planning?

• A lot, actually.
• MAX runs a regional optical network in the
  Washington/ Baltimore/Northern VA area with 7
  pop locations and 400 fiber miles in 4
  production and 5 research overlapping rings, lit
  by 4 different dwdm systems. Weve done this
  since 2000. 
• We cover a small footprint, but we're very
  complex.
• RE nets are not a traditional layer 1 capacity
  planning environment, where transport engineers
  provision based on requirements thrown over the
  wall by data engineers whose network traffic
  doubles every 18 months. 
• These are the revenue-based networking folks
  who come to NANOG (revenue-based and cost-based
  are Dave Farmers terms).

7
So is R E cost-based networking or
politics-based networking?

• Our RE environment often needs massive
  overprovisioning to accommodate big flows with
  good performance.
• That needs to be accomplished despite
  understaffing and often chronic underfunding.
  Were not telcos trying to gouge customers for
  every Mb, with staffs of thousands to push
  paperwork.
• So often capacity planning is more about
  horsetrading fiber and other resources to get to
  where youre guessing you need to be next year,
  as much for politics as for actual capacity
  growth. 
• This leads to far more interesting heterogeneity
  and complexity than would be the case for a
  revenue-based network able to spend 500M to
  build out a shiny new homogeneous optical
  network, who knows what their customers need and
  builds based on traffic.

8
Enough philosophy, how about some methodology?

• Max sells ip, mpls, and lambda transport services
  over 4 production dwdm rings. Since we have
  rings, we try to provision lambdas with optical
  protection where possible. Fiber cuts happen.
• Recently weve had high growth in 10G lambda
  demand. Due to dropping 10G xponder costs, we
  now only provision 10G lambdas, and will serve
  gige lambdas via muxponders (8 giges to a 10G).
  This works because people rarely want OC48 2.5Gs,
  usually giges
• Due to Movaz low 10G chassis density (2 10Gs
  with protect per 4U RayExpress), we are using
  add-in dwdm from Ekinops, a French startup.
  These are muxed into the Movaz as protected
  external waves (aliens). Still in test because
  their software is early, and unclear still
  whether space savings trades against complexity.

9
Dwdm system cost tradeoffs

• Interesting comparison of "system" vs "component"
  dwdm costs. A pair of 10G dwdm xponders is
  currently 6k-10k/pr bought as components (or as
  parts of unknown-brand dwdm systems )
• But we're currently paying 24k/pr for
  system-integrated xponders from Movaz and
  Zhone/Luxn. This says that there's a "systems
  integration tax" of 250-400 that you pay for
  stuff like Movaz developing all their software
  inc GMPLS, which is paid for by their hardware
  costs.
• Nothing new there Cisco, Juniper etc all do
  that, it's how the industry works.

10
More dwdm system cost tradeoffs

• But there are also "bare bones" dwdm systems that
  have very little integration or software, eg MRV
  and JDSU. These are basically all sfps and xfps,
  that have a lower markup over the component
  costs, and offer fewer bells and whistles like
  variable attenuation etc.
• More data would be interesting to characterize
  vendors on this scale from components on one end
  to "economy" like JDSU MRV to "midrange" like
  Movaz to "Cadillacs" like Cisco 454s and Ciena
  Corestreams.
• Personally I'm a fan of DIY optical systems, but
  on production nets it's hard to get away with
  anything ad hoc. Your price of failure is so
  high that it dwarfs any difference in xponder
  costs. In the R E world, people are more
  concerned with the diff between what the optical
  systems guys are charging, which are considered
  cheap, vs the router guys who are still trying to
  get 75k for one 10G int.

11
So where are we going? 

• With emerging next-gen ROADMs signalling to
  tunable lasers for capacity on demand, the
  research picture for dynamic resources looks
  good. Heard talks about DRAGON, GLIF, and
  others.
• Getting this technology migrated from DRAGON and
  HOPI test networks out into the real world of
  production where things have to work well, is
  our next major challenge.
• Make no mistake transitioning from the current
  world of statically provisioned net resources to
  dynamic is a major sea change.
• But the world is changing, and we have to
  position ourselves in that emerging world. All
  while betting the farm on business decisions that
  will hopefully work out right and create growth.