Metrics%20for%20Evaluating%20ICEBERG

1
Metrics for Evaluating ICEBERG

• ICEBERG Retreat Breakout Session
• Jan 11, 2000
• Coordinators Chen-Nee Chuah Jimmy Shih

2
Goal of This Breakout Session

• We claim in ICEBERG that an IP-based core network
  can be better than the traditional
  telephony-based core network in terms of
• Usability.
• Performance.
• Scalability.
• Robust Execution at a Reduced Cost.
• Our challenge for this breakout session is to
  come up with a set of metrics for evaluating
  ICEBERG against these claims.

3
Usability

• Preference Registry.
• How easy to setup and update preferences?
• Personal Activity Coordinator.
• How easy to interact with it?
• Personal Mobility.
• How easy to achieve this goal?
• Potentially Any Network Service (PANS).
• How easy to provide PANS given the limitations of
  different devices and networks?
• Service Creation.
• How easy to create new services? How much
  infrastructure needs to be there to allow new
  service creation?

4
Performance

• Operation in the Wide-area.
• End to end performance.
• Call set up time.
• Service Discovery Service/ Personal Activity
  Coordinator/ Preference Registry lookup time.
• Resource reservation set up time.
• Performance when services span multiple
  administrative domains.
• Performance when services span different types of
  networks and devices.
• Service Mobility.
• Performance during service handoffs.

5
Scalability

• Does ICEBERG scale to a large user base?
• How does the signaling complexity/overhead grow
  with the number of users? The number of service
  providers?
• Amount of state information needed to be stored
  at the ICEBERG point of presence(IPOP)? At the
  subsequent routers?
• Diversity and size of access networks that
  ICEBERG can provide services to?
• How many simultaneous calls per second can be
  handled?

6
Robust Execution at a Reduced Cost

• Fault Tolerance.
• Component failures. If a Call Agent breaks down,
  how long does it take to regenerate another Call
  Agent?
• Time to discover faults, time to recover from
  faults, loss of information due to faults.
• Availability.
• Network partitions. Percentage of time IPOP is
  available/accessible. Dependencies between
  components.
• Operation and Maintenance.
• Add, update and delete software agents or cached
  data (e.g. user preference).
• Security, Authentication and Privacy.
• Trusted computing base, time to discover attacks,
  damage due to attacks.

7
Deployment Issues

• How many people would use the ICEBERG
  communication services?
• How easy to develop new communication services?
• How easy to setup the system?
• How much does the system cost?
• How difficult to administer the system?
• How easy to extend an existing system?

8
Testing Platforms

• What is the reasonable analytical and simulation
  framework to study scalability performance?
• Experience learnt from web server studies
  emulating multiple clients
• What about RMI based server? IPOP?