NOC Theory and Practice

1
NOC Theory and Practice

• Part II The Nuts and Bolts
• (The Practice Of Making a Good NOC)

2
DISCLAIMER

• I am not affiliated with any corporation. My
  views are my own.

3
Intro - Whats In A NOC?

• Purpose - What Type of Company Are You?
• People will always be your key asset.
• Systems make the job of people easier.
• Managers assist in identifying gaps in both.
• Processes allow for flexibility in operations
  while still maintaining standards.
• And, last but not least, the facility is where
  you do your work.

4
The Facility - Think Data Center

• Location, location, location.
• Security
• Environmental Concerns
• This applies to both your inside and outside
  environment.
• Ergonomics
• Reliability
• Sources of Labor

5
The Goals of a NOC

• Our conception is
• At the bare minimum, you will receive requests
  from your customers
• You will process those requests in some fashion
  (engineering, handoff, ...)
• You will communicate resolution to the customer,
  and/or keep customer updated on status

6
Goals II

• NOC as first, second, or third tier support.
• Conception of NOC as do everything group
• The engineers should be smart at designing,
  configuring, etc. - NOC should be divided into
  categories of engineers able to resolve issues
• A smart issue routing process makes this a very
  effective model

7
The Process

• Receive events from your inputs (phone calls,
  network monitoring, etc)
• Process inputs into a problem to be resolved
• Route the problem to the correct individual
• Resolve the problem as appropriate
• Communicate status continually.
• Record the problem for later review and training.

8
The Key to Success Is

• By constant re-examination of your operation,
  your employees, and what inputs you are getting,
  you can reach incredible efficiency. Communicate
  this to other teams and your operations will
  prosper.

9
The Facility (cont.)

• Security applies to keeping other people out, but
  it also applies to making a safe arrival for your
  employees, at any hour.
• Environmental concerns Just like a datacenter,
  consider climate control and earthquake/act of
  god resistance. Disaster planning rules should
  be in effect.
• Sources of labor Not just engineers, but
  builders.

10
A Word on Portability

• As Matt suggests, portability is critical.
• Obvious resources Computers, communications,
  phones.
• Not obvious resources Vehicles and quick
  transport/assembly, good cases for shipping or
  relocation, breakdown/reassembly schedules.
• Multiple everything.
• Make sure your resources are distributed. If
  your network is large, your footprint of mobility
  should be large.

11
Hiring the Right People

• Skillset
• but more importantly, flexibility
• Stress
• Scheduling
• Stability
• NOCs are hard things to staff appropriately. And
  good people are hard to retain.

12
Hiring (cont.)

• Make sure they have lives. But know what you
  cant ask.
• Can they talk? How are their soft skills?
• Do they get flustered easily?
• And remember, org charts suck. Classify people
  according to their skills, not according to some
  hierarchy which has no meaning outside of some
  middle-management mind.

13
Keeping Your People Happy

• Environment
• Ergonomics
• Creature Comforts
• Were not kidding.
• Give some thought to a 24x7 outlook. This means
  beds, showers, food, and climate.

14
Environment - People

• Climate controlled centers get cold at night.
  Make sure you have 24H control over your space.
• When thinking about climate, keep in mind your
  working conditions are slightly different in a
  NOC.
• 200AM to 400AM are the critical hours.

15
Ergonomics - People

• Hire a good ergonomic consultant.
• Positions of lights, phones, keyboards, monitors,
  pens - all this matters.
• Observation is key. Your engineers will show you
  what the problems are. (Cameras)
• Seating. If youre in a chair for eight hours at
  a stretch, dont you want it to be a comfortable
  one?
• Screen space.

16
About People. About NOCs.

• Its a messy business. Its high stress. Its
  easy to let things get out of hand.
• Good management qualities in a NOC
• cool heads
• methodical approach to solving problems
• having been on the customer side
• micromanagement sucks.

17
Lets go over retention.

• Retention of NOC staff is 100 harder. Very few
  people live for this kind of job.
• So, innovate. Think outside the box.
• People want to grow. Know what your people want
  to do when they grow up.
• Remember your core focus. Do operations, not
  sales, or account services, or anything else.
• At the same time, make sure your folks are
  informed on everything. Everything comes out in
  the wash. All the dirty clothes come to NOC.

18
Interviewing NOC folks

• You are interested in three things.
• The candidates willingness to learn.
• The candidates 1-2 year outlook on their career.
• The candidates ability to work funny schedules.
• They are interested in three things.
• Will you offer them growth potential?
• Can you give them enough perks to enjoy their
  work?
• Do you have your stuff together?
• Are the procedures well-documented?
• Will they be called in outside their shift?
• Will they ever wonder who to go to?

19
Productivity

• Momentum is key. Keeping your people busy is the
  most important thing towards maintaining
  productivity.
• The other component to momentum is keeping them
  working on different things. This makes them
  better engineers, and helps feed ego.
• Everything at your fingertips - your engineers
  should never have to make a customer wait more
  than four minutes on hold for anything. If it
  has to wait longer than that, tell the customer
  why, and mark it as something to fix.

20
The View of the Customer

• A customer wants
• Professionalism your engineers must treat the
  customer as if they are paying you a lot of money
  and the engineer wants the customer to continue
  to do that.
• ETR Your customer wants to know when the problem
  will be fixed. I dont know is not an
  acceptable answer. The ETR is the first thing
  you give to a customer and the one thing you
  update him on at least every hour (or less if
  they prefer).

21
The View of the Customer II

• A description Concise or not, the customer
  wants to know what the problem appears to be.
• Remember your focus. It is not (typically) your
  responsibility to engineer your customers
  network for maximum reliability. Your
  responsibility is the operation of that network.
  The only thing you should do as a NOC is fix
  problems when they arise in the most expedient
  and professional manner possible. A customer
  wants to be comfortable, and assured that that
  will happen.

22
Things we never, EVER allow a customer to hear

• I dont know when it will be fixed.
• I cant help you.
• Im not qualified to answer that question.
• Let me transfer you to someone else.

23
Its not always ideal. But -

• Where possible, the first person to take the call
  should be the person who follows it to
  completion. The customer hates being thrown
  between different engineers - but not having an
  answer is worse.

24
A Little About Systems

• Dont just focus on ticketing.
• Make sure your people have information on
  everything they need to operate.
• If all the architects fell off the planet today,
  could you rebuild?
• Its about agility.

25
Policies and Procedures

• If you dont write it down, it isnt a procedure
  or a policy. There are no unwritten rules in a
  NOC. People cycle.
• Focus on procedures that affect the handling of a
  customer. The soft skills are usually the
  hardest to learn.
• Full disclosure?
• Troubleshooting Techniques

26
And a word on incidents.

• Incidents, tickets, call it what you will.
• Whatever happens in the NOC
• gets summarized to the account exec if the issue
  is of appropriate severity
• gets recorded - nothing EVER gets deleted from a
  customer log
• gets archived
• gets resolved - how much time will you spend on
  issues that never get fixed?

27
Some final notes - Conclusion

• Its easy to throw a bunch of people in a room.
  Its hard to build an efficient, productive, and
  well-oiled NOC (the same goes for networks)
• Record everything. Write everything down.
• Try to remain as flexible as possible.
• Look to your employees for cues on what their
  ideal environment should be.
• Remember your focus, and know your customer.
• Know what not to say.

28
Credits

• As far as the practice of NOC building goes, we
  must thank
• herb_at_tomobiki.urusei.net - thanks for great
  private discussions on NOC building
• nanog subscribers - for talking about operational
  stuff

29
And finally

• Sean Donelan - for giving network operations
  folks an impetus for existing (due to his backhoe
  skills)
• Shouts to EFNet IRC nanog you know who you are

30
Part III Evaluation Criteria

• Social engineer your own NOC.
• We like to rate NOC folks on
• Do they sound professional? Do they have the
  answers?
• Do they follow us all the way through to
  resolution?
• Can they tackle difficult as well as simple
  problems?

31
Part III Evaluation Criteria (cont).

• How many calls a day does your NOC take?
• Do you survey your customers, Cisco-TAC style?
• The three Ps
• Perception
• Passion (Emotion)
• Presentation

32
Evaluation Criteria

• They (customer) must perceive you as the people
  that will help.
• You (NOC) must present your solution with total
  assurance that you can handle the problem.
• You (NOC) must have passion in dealing with the
  customer. Make it friendly, even if its a
  network down emergency. It will come back to pay
  off.

33
Remember.

• Your goal is to be professional, calm, cool and
  collected, even in the face of serious adversity.
• If you dont have an answer, you know where to
  get it and how long it will take.
• You have to communicate to the customer and keep
  the customers viewpoint in mind, no matter what.
• As a NOC manager/builder, you have to make sure
  your people have the right tools.