Washed by the Very Same Rain: System Administration Research

1
Washed by the Very Same Rain System
Administration Research

• Alva L. Couch
• Tufts University
• couch_at_cs.tufts.edu

2
Part I

• What is research?

3
Who am I?

• Overseer of LISA chair of steering committee,
  board liaison (since 2005)
• 14 LISA papers since 1996 ( 2 students who
  submitted sole-author papers)
• 2 LISA best paper awards and 1 best student
  paper award since 1996.
• 2003 SAGE Professional Service Award (with Mark
  Burgess and Paul Anderson).

4
What is research?

• We all think we know, but
• popular accounts of the nature of research are
  misleading, and
• remain misleading throughout recorded history!

5
A popular misconception

• Einstein created the theory of relativity out of
  thin air.
• No one else could have done it but Einstein.
• Not!
• Einsteins work began in a context of what was
  already known, and
• Several mathematicians (notably Minkowski) were
  working in the same context and concurrently
  trying to come up with their own explanations!

6
A song and a woven cloth

• This presentation is structured like the folksong
  same rain by American folk singer Pat
  Humphries, which has been considered by some as a
  paradigm for research and exploration.
• It is a cloth woven from threads inspired by my
  wifes mentor Prof. Philip Morrison (of MIT) and
  his TV series The Ring of Truth, which
  discusses how scientists develop their ideas.

7
We're all living in a great big dipper

• All research occurs in a context, that includes
• What work has been done before.
• What community is interested.
• What problems remain to be solved.
• Context is a moving target that can change
  rapidly over time.

8
Were all washed by the very same rain

• By definition, research doesnt occur in a
  vacuum.
• If you see something important, chances are that
  a number of other people have seen the same
  thing.
• Difference is whether you do something about
  understanding what you see!
• Edison 1 inspiration, 99 perspiration.

9
We are swimming in the stream together

• Research is not about working alone, but rather
  about communicating ideas to a community that is
  exploring similar directions.
• Most important step is to identify your community
  (or communities).
• Who are you swimming with?

10
Some in power and some in pain

• Failure is a crucial part of research.
• Ones hypothesis can be invalid.
• Even after one has believed it for years.
• Only by failing can one learn.
• Only by being open to failure can one become
  objective.
• I have more wrong ideas than right ones!

11
The usual formula for how to do research

• Determine context of the problem.
• Survey proposed solutions.
• Determine new directions to explore.
• Choose one direction to explore.
• Develop a hypothesis about the direction.
• Test that hypothesis.
• Evaluate the results of the test.
• Refine the hypothesis, and repeat.

12
Sufficient but not necessary

• The formula is one way to accomplish research.
• But people can become all-too-dependent upon the
  formula.
• Better understand the important parts and how
  they fit together.

13
Key elements of the formula

• Context maintaining an idea of what you know and
  dont know about a problem.
• History keeping track of what you learn over
  time.
• Evidence how what you see supports or refutes
  what you might think.
• Conversation the ability to explain what you see
  to others.

14
An alternative formula

• Get excited about something.
• Commit to learning all that can be understood
  about it.
• Choose some small part of it to understand
  better.
• Write down your specific ideas about the nature
  of this part. This is your hypothesis.
• Test your understanding with observation. This is
  your experiment.
• Remain doubtful of unconvincing evidence, and
  curious about contradictory evidence.
• Refine yourself and then repeat!

15
Research versus learning

• Too often, research is mischaracterized as a
  discovery product, like finding a piece of gold
  in a gold mine.
• Most research is instead a learning process,
  where you learn something new about something you
  already see.
• The gold is not what you see, but what you learn.

16
Process versus product

• Accordingly, research is a process of learning
  rather than a product of searching.
• A negative result (refutation of your hypothesis)
  is as important as a positive result (support for
  your hypothesis).
• In either case, you learned something.

17
Research redefined

• An active learning process
• In which you explore what happens, and learn from
  the world
• In a continuing conversation with a community of
  learning
• In a changing and evolving context of observed
  phenomena and human needs
• In which one risks being wrong, but learns and
  evolves from ones mistakes.

18
Good research and good science

• There are two main axes to the problem
• Good research doing things that are useful and
  interesting to a target community.
• Good science doing things in a way that is
  rational and acceptable to the scientific
  community as a whole.

19
Good research is

• New gives some new insight.
• Useful someone can use the insight.
• Relevant people can see how it helps.

20
Good science is

• Reproducible others can study the same problem
  and come to the same conclusions.
• Objective does not depend upon who is doing it.
• Evident everything arises from direct evidence.
• Defendable you can explain your conclusions
  compellingly to others.

21
Expectations and learning

• The largest hurdle to learning is expectation.
• The greatest aid to learning is doubt.
• If you think you know whether your hypothesis is
  true, you arent doing research.
• If, on the other hand, you admit that you dont
  know the answer, you are already on the road to
  finding it.

22
The Ph.D.

• There once was a young man from Esser,
• whose knowledge grew lesser and lesser.
• It once grew so small,
• he knew nothing at all,
• so now hes a college professor!
• (Getting a Ph.D. is not just about gaining
  knowledge, but about learning not to misrepresent
  what you know and dont know!)

23
The Ring of Truth

• My wife was the researcher for the TV series The
  Ring of Truth, which discusses the nature of
  science.
• Each show concentrates on some aspect of the
  scientific method Looking, Change, Mapping,
  Clues, Atoms, and Doubt.
• Lets map these ideas into system administration
  terms!

24
My version of the Ring of Truth (for system
administration research)

• Looking the skill of seeing and recognizing the
  unknown within the context of the known.
• Change the fact that nothing is constant, and
  knowledge always transforms the receiver.
• Mapping the ability to see analogies between one
  part of the world and another, and to make models
  that explain and inform.
• Clues the ability to recognize and collect
  evidence, even if it contradicts intuition.
• Atoms the ability to recognize the indivisible,
  and come to grips with what is knowable and
  unknowable.
• Doubt the ability to look at something without
  preconceptions, and accept confusion.

25
Looking

• The ability to look at something familiar and see
  something new.
• Burch and Cheswick,Tracing Anonymous Packets to
  Their Approximate Source, Proc. LISA 2000.
• A denial of service (DoS) attack is not always a
  bad thing, and one can use a structured DoS to
  identify perpetrators of other DoSs!

26
Change

• The ability to embrace the idea that ones
  understanding of the world and the world
  changes and improves over time.
• Finke, Manage People, Not Userids, Proc. LISA
  2005.
• A revisitation of the same authors previous
  paper on the subject, in which he explains how
  his understanding and practice improved over time
  and reversed some prior decisions.

27
Mapping

• The ability to use models and abstraction to
  understand the world.
• Couch, Wu, and Susanto, Toward a cost model for
  system administration, Proc. LISA 2005.
• A model of cost for helpdesks shows through
  simulation that helpdesks running near the limit
  of staff capacity experience chaotic changes in
  total value.

28
Clues

• The ability to look for and see clues toward new
  and different explanations of phenomena.
• Gross and Rosson, Looking for Trouble
  Understanding End-User Security Management, Proc.
  CHIMIT 2007.
• The windows firewall message do you want to
  allow this connection is semantically equivalent
  in the minds of most users with do you want
  to get your work done or not?

29
Atoms

• The ability to come to grips with what is
  knowable and what is unknowable.
• Burgess, Computer Immunology, Proc. LISA 1998.
• Centralized control systems depend upon knowing
  the unknowable, whereas physical systems such as
  the human body depend upon distributed and more
  knowable notions.

30
Doubt

• The ability to face and embrace ones lack of
  understanding of complex phenomena.
• Evard, An Analysis of UNIX System Configuration,
  Proc. LISA 1997.
• Configuration management is often conceptualized
  as a simple choice between tools, but involves a
  more complex conflict between technical methods
  and human needs.

31
Belief and research

• Belief has little part in research. Research is
  about doubt and evidence, while belief is about
  what one does without evidence.
• My grandfather used to ask me whether I believe
  in the theory of evolution.
• I never managed to explain to him that a theory
  is not a thing to be believed, but rather, a
  thing to be tested, weighed, and doubted.

32
Most common mistakes in research

• Hypothesis is too limited only reasonable
  conclusion is no.
• Hypothesis is too broad only reasonable
  conclusion is maybe.
• Not accounting for all variables something
  important is missed.
• No target community result is not important.
• Lack of objectivity, reproducibility, or
  evidence result is not plausible.

33
Part II

• Steps toward engaging in research

34
Parts of becoming a researcher

• Engaging in active learning.
• Being open to doubt.
• Finding and maintaining context.

35
Aids to effective learning

• Keeping a personal journal of ideas, directions,
  hypotheses, experiments, conclusions, references.
  
• Breadth documenting every idea you get.
• Depth exploring one new direction at a time.
• Documenting each hypothesis and the evidence for
  and against it as soon as possible.

36
Persistence of memory?

• Dont rely on your memory, no matter how good it
  is.
• Your understanding of the problem is a moving
  target.
• To teach other people what you learned, you need
  to recall what you didnt know before!

37
Example my journal

• Dated entries describe hypotheses, tests,
  results, ideas.
• In electronic form (plaintext).
• Ideas often turn out to be wrong.
• I never delete or edit an entry!
• This is not a publication it is a starting point
  for one.
• It is more important to have a record than to be
  correct.

38
Being open to doubt

• Doing research is about accepting that absolutely
  any idea you write down is
• subject to continual validation and
• can turn out to be invalid at any time in the
  future.
• Each entry in the journal is a starting point for
  discussion, and not a fact.
• In mine, the invalidated entries outnumber the
  validated ones.

39
Finding context and community

• Several resources can aid you in beginning
• The Anderson taxonomy of system administration
  topics. Anderson and Patterson, A Retrospective
  on Twelve Years of LISA Proceedings, Proc. LISA
  1999.
• Book Selected Papers in Network and System
  Administration (based upon the Anderson
  Taxonomy).
• Book Handbook of Network and System
  Administration (beyond the Anderson taxonomy).
• USENIX compendium of best papers (a testament to
  the most interesting topics and approaches).
• Google can help, but only if you already know the
  proper keywords!

40
Just as important find community

• Your community the people in this room.
• One often chooses a problem for a community
  rather than the other way around.

41
Essential skills of the researcher

• Focused reading
• Documenting biases.
• Collecting evidence.
• Being open to surprises.

42
Focused reading

• A researcher doesnt read a paper like a regular
  person.
• Reading occurs in a context.
• To answer specific questions.

43
The typical questions

• Relevance is this work relevant to what I want
  to understand?
• Context where did their understanding start
  (when their work began)?
• Results where did their understanding end (when
  they finished this paper)?
• Doubt what unknowns did they find?

44
Questions evolve!

• These are just a starting point.
• As you focus upon a topic, reading becomes more
  focused as well.
• E.g., Is this relevant becomes a question about
  a specific kind of relevance.

45
Part III

• Examples

46
(Ahem)

• The original idea for this talk was to describe
  the whole landscape of system administration
  research and where things are today.
• I thought about this a bit and decided that it
  was too broad an objective.
• And it sounded a bit boring.
• So instead, I am going to show you several
  examples of how to build your own landscape of
  whats important to you.
• And then, Ill take requests!

47
How to build your own landscape

• Express your preconceptions honestly.
• Use focused reading to find evidence for or
  against your preconceptions.
• Weigh the evidence, reevaluate your
  preconceptions.
• When the literature fails to support or refute,
  its time to do your own experiment.

48
Some parts of the current landscape (some of
whats hot)

• Power-aware systems
• Adoption of automation tools versus writing your
  own tools.
• Balancing security and business objectives.
• Integrated management of systems, knowledge,
  security, audit data.
• Dealing with various (existing and new) forms of
  spam.
• (and many others).

49
Power-aware systems

• No paper at LISA as yet.
• Two important posters at HotPower 2008
• Srikantaiah, Kansal, and Zhao, Energy Aware
  Consolidation for Cloud Computing.
• Lu and Varman, Workload Decomposition for Power
  Efficient Storage Systems,
• Focused reading
• What is the problem?
• What are the challenges?
• How could this apply to system administration?

50
Adoption of automation tools

• This is a hard one.
• Lets go digging
• Mentioned in my LISA 2005 talk What is this
  thing called configuration management?.
• Lots of hallway conversations.
• Lots of very indirect evidence.
• Evidence scattered all over the universe, one
  sentence at a time.
• I didnt say this was always easy.

51
Balancing security and business objectives

• Very few writings, but very controversial. One
  example
• Beattie, Arnold, Cowan, Wagle, Wright, and
  Shostack, Timing the application of security
  patches for optimal uptime, Proc. LISA 2002.
• Focused reading
• What questions remain?
• Are there analogies with other best practices?

52
Integrated management

• Lots of references with scattered ideas. One
  example
• Wang, Verbowski, Dunagan, Chen, Wang, Yuan, and
  Zhang, STRIDER A Black-box, State-based Approach
  to Change and Configuration Management and
  Support, Proc. LISA 2003.
• Focused reading
• What is the problem?
• How does their approach work?
• Can it be applied to Linux?

53
Spam

• A huge number of references with different
  strategies. One example
• Singaraju and Kang, RepuScore Collaborative
  Reputation Management Framework for Email
  Infrastructure, Proc. LISA 2007.
• Focused reading
• What kind of spam does this prevent?
• What requirements are there?
• What limitations are there?

54
And the votes are in!

• Anomaly detection and correction
• Networking and IT Infrastructure
• Configuration management (3)
• Databases and Information Storage (3)
• Heterogeneity
• IP telephony

• Managing mobile and wireless computing (3)
• Network and Information Security (3)
• Remote administration
• Scaling problems large or high-volume (2)
• User management
• Virtualization (5)

55
So, the next topic is rather obvious

• I happen to know a bit about virtualization
• Alva Couch, System administration thermodynamics,
  login magazine, Oct 2008.

56
Kinds of virtualization

• Whole operating system (XEN, VMWare, etc).
• I/O virtualization virtualize access to files,
  devices, etc, but not the operating system.
• Monica Lam
• Virtualization of configuration management
• (NSDI Shards system)

57
Requests?

• (Feel free to put me on the spot)

58
Part IV

• Epilogue

59
The Pat Humphries song upon which I patterned
this presentation

• "We're all living in a great big dipper.
• We're all washed by the very same rain.
• We are swimming in the stream together,
• Some in power and some in pain.
• We can worship this ground we walk on,
• Cherishing the dreams that lie deep inside.
• Loving spirits will live forever.
• We're all swimming to the other side.

60
But the last verse is most relevant

• When we get there we'll discover
• All the gifts we've been given to share
• Have been with us since life's beginning
• And we never noticed they were there.
• We can balance at the brink of wisdom
• Never recognizing that we've arrived.
• Loving spirits will live together.
• We're all swimming to the other side.

61
Pat Humphries said, about same rain

• This did not just come out of me. This came
  from a lot of different people and different
  places, and I just happened to be here at the
  right time for it to flow through my pen, my tape
  recorder.
• I would say the same thing about my own
  research.

62
Washed by the Very Same Rain System
Administration ResearchThe End

• Alva L. Couch
• Tufts University
• couch_at_cs.tufts.edu