Historic Integrity in Peer-to-Peer Systems

1
Historic Integrity inPeer-to-Peer Systems

• Petros Maniatis, Mary Baker
• Computer Science Department
• Stanford University

2
Talk Outline

• Motivation
• The Problem
• Secure Timelines
• Timeline Entanglement
• Evaluation
• The Future

3
Motivation

• Time and history are really important!

4
Time and History in Distributed Systems

• Time
• A clock logical or physical assigning time
  to system events
• Time stamp
• The time when a system event occurs
• History
• All system events that ever happened
• The time stamps of those events

5
Who Cares About Time?

• Resource management / mutual exclusion
• Two requesters vie for the same resource
• Earlier requester wins
• Prior art issues and time stamping
• Two parties invent the same thing
• He who publishes it first wins (a patent, glory,
  a PhD, etc.)

6
Who Cares About History?

• Backups
• Checkpoints
• Archival storage
• Version control systems (CVS)
• Versioned file systems (VMS, Plan 9, Elephant)
• Accounting

7
Who Cares About History? (2)

• Reputation systems
• Was the reputation of this account accrued or
  magically built overnight?
• Accountable systems
• E.g., p2p storage
• At time t, node A accepted document D
• At time t1gtgtt, node A claims it never saw
  document D
• Can I catch the fraud?

8
The Problem

• Whats there and what isnt

9
Approaches

• Logical clocks Lamport 1978
• Partial temporal ordering of all events globally
• Everyone is well-behaved
• Network synchronization
• Keeps all well-behaved nodes synchronized
• Assumes well-known bounds on misfortune
• Secure time stamping
• Trusted third party maintains ordering of events
• It gives authoritative answers on what happened
  when

10
Why Is That Not Enough?

• P2P systems are vulnerable to malice
• They are chaotic
• They are fragile
• They are very dynamic
• They are large
• Past approaches are
• Vulnerable to malicious attacks, and/or
• Unscalable, and/or
• Not survivable

11
Our Approach

• We solve a narrower, more selfish problem
• Every peer
• Maintains a local view of global time and history
• Safeguards the integrity of portions of global
  history it cares about
• Only trusts itself or provable information
• Requirements
• Efficient, scalable, survivable, aggressively
  decentralized
• Timeweave a tamper-evident historic record for a
  distributed system

12
Secure Timelines

• The fingerprints of history

13
Logical Clocks

• A counter of the state changes of a system
• The clock advances by one tick
• Every time the system state changes
• Every time a message is sent
• Every time a message is received
• I can change my record of old states and nobody
  knows
• I can move my clock backwards and nobody knows

14
Secure Timelines

• A secure timeline is
• An authenticated logical clock, and
• A historic record of the state transitions of a
  system
• Every time step has
• A logical time (a counter of time steps)
• An annotation (a fingerprint of the current
  system state)
• An authenticator

15
Secure Timelines (2)

• Authenticator is a one-way hash of
• Previous logical time
• Previous authenticator
• Previous state

16
One-Way Functions andThe Arrow of Time

• A function f is one-way if
• Given f(a) it is intractable to find a (preimage)
• Given lta,f(a)gt it is intractable to find b such
  that f(b)f(a) (second preimage)
• It is intractable to find lta, bgt such that
  f(a)f(b) (collision)
• SHA1, for example, is considered one-way
• If f(a)b, then I can safely assume that a was
  known before b
• Otherwise, someone must have found the
  appropriate a for b, either intentionally or by
  luck

17
Secure Timelines Are Tamper-Evident

• If authenticator 5 is known
• I can request a proof that the previous state was
  state 4
• Proof is the application of the one-way function
  to state 4, yielding authenticator 5
• Similar proofs for previous states or
  authenticators
• There is no way that the system can prove any
  other state as its state at time 4 or before

18
How Do Timelines Work?

• Holders of timeline authenticators can ask
• Was State3 the state of the system at time 3?
• Did State2 precede State4?
• Answers are provable
• A proof is a trace of the one-way functions
  applied
• If it leads to the current authenticator, its
  valid

19
How Do Timelines Work? (2)
20
What Is a Timeline Good for?

• Did event E happen at time t?
• Find the state at time t
• Check if event E is included in the state
• Did event E happen before event E?
• Find the state S including event E
• Find the state S including event E
• If S precedes S, E happened before E
• Only meaningful within a single system
• How can I extract temporal orderings of events
  across different systems?

21
Timeline Entanglement

• A bridge over trust boundaries

22
Distributed Timelines

• A secure timeline spans the states of a single
  system
• How do we transition to a distributed system of
  mutually distrustful peers?
• Major security requirement
• Correctness If a precedes b, no peer can
  convince me that b precedes a
• Timeline Entanglement
• Include authenticators from one timeline into the
  state of all others

23
Timeline Entanglement

• Every node stores samples of the timeline of
  every other node in the group
• All-to-all entanglement
• Inclusion of my sample in a foreign timeline
  builds a precedence
• From the past of my timeline
• To the future of the foreign timeline
• Done regularly (not only when messages are sent)
• Say every 10 minutes

24
Timeline Entanglement (2)

• A sends B a timeline thread
• Signed ltA,i,authenticatorigt
• B includes the thread in its system state
• In a thread archive
• B sends A an entanglement receipt
• Contains a proof of thread storage
• Shows that As thread is included in the
  computation of Bs next authenticator

25
Entanglement Checks

• Continuity
• B checks that As thread follows the same
  timeline as last time they talked
• Acknowledgement
• Receipt shows that B archived As thread in its
  current state
• Reverse continuity
• A checks Bs receipt against past threads from B

26
Entanglement Properties

• Commitment
• After A tells B its authenticator, it cant
  change its timelines past
• After B acknowledges As thread, it cant change
  its thread archive
• Cross-domain ordering
• Events before ltA,3gt precede events after ltB,2gt
  provably

27
Local View of Global History

• Through entanglement, A samples Bs timeline

28
Local View of Global History

• Through entanglement, A samples Bs timeline
• This gives A a coarse view of Bs history

29
Local View of Global History

• Through entanglement, A samples Bs timeline
• This gives A a coarse view of Bs history
• Coarseness results in loss of temporal resolution
• X, Y and Z all appear to A temporally
  indistinguishable

30
Local View of Global History

• Through entanglement, A samples Bs timeline
• This gives A a coarse view of Bs history
• Coarseness results in loss of temporal resolution
• X, Y and Z all appear to A temporally
  indistinguishable
• Entanglement frequency determines granularity

31
Entangled Node Architecture

• Secure timeline
• Timeline thread archive
• Contains all threads received from peers
• Included in the nodes accountable state
• Necessary to prove I have archived a thread
• Entanglement receipt archive
• Holds receipts received for outgoing threads
• Not included in accountable state
• I dont need to prove Ive archived the receipts
  for my threads

32
How Does Timeline Entanglement Work?
33
Evaluation

• Benefits, performance and scalability

34
Entanglement Benefits

• Secure Temporal Mapping
• I can map a remote logical time onto my own trust
  domain
• Historic Integrity
• I am a witness to all remote timelines
• If I care about a remote event and it changes,
  Ill catch the fraud
• Survivability
• I can prove the temporal ordering of local events
  with remote events, even after the remote
  timeline dies

35
Secure Temporal Mapping

• A whishes to know when Bs time was 2
• It looks in its outgoing threads for a thread
  arriving at B at or before 2
• It looks in its incoming threads for a thread
  sent from B after 2
• These map B,2 to As interval (1,7)
• X maps to interval (1,7) and, therefore, precedes
  Y

36
Why Is Temporal Mapping Useful?

• Any event at remote timelines can be mapped to
  the local timeline
• Any ordering among events is reduced to a locally
  verifiable ordering
• I dont need to trust the remote timelines to
  decide what came first
• Limitation mapping loses some temporal resolution

37
Performance

• How large a problem can we solve before the
  solution becomes unmanageable?
• Free variables
• Size of entangled set (group size)
• Entanglements per time step per node
• Metrics
• Maintenance overhead
• CPU Disk I/O used by thread production,
  verification, location and storage
• Communication overhead
• Sending data rate

38
Performance for Increasing Entanglement Load

• 10-minute, 1-hour and 6-hour entanglement
  resolutions

39
Performance for Increasing Group Size

• 0.5, 1 and 2 entanglements per time step

40
Future Directions

• Limitations, extensions, applications

41
Complete and Connected Entanglement Graphs

• Entanglement graph is complete
• Every peer entangles with every other peer
• Complete entanglement has limited scalability
• For high resolution (10 minutes), a group size of
  600 nodes takes 100ms per step
• To scale to larger groups, move to a less dense
  graph

42
Transitive Entanglement

• Nodes only entangle with their immediate
  neighbors
• B only entangles with A,C, D, G and J
• Arbitrary mappings are performed through
  successive time hops
• To map from Bs timeline to Is timeline, first
  map to G and then to I
• Routing algorithms use the aggregate temporal
  loss metric
• To choose between B-G-I and B-J-I, check which
  induces the least temporal resolution loss in the
  mapping

43
Globally-Authenticated Historic File Systems

• Technique
• Retain history of file system changes (e.g., Plan
  9)
• Maintain a secure timeline of the file system
  state
• Entangle timelines across a very large file
  server population
• Benefit
• Automatic global-scale file time stamping
• Very difficult to delete unobtrusively documents
  that were originally included in the system

44
Conclusion

• Time and history are important concepts for
  current and future p2p systems
• We extend the notion of logical clocks to build a
  tamper evident history of a systems states
• We combine logical clocks of many systems through
  timeline entanglement
• An efficient mechanism to maintain a consistent,
  tamper-evident history of a large system

45
The End

• Technical report and other info at
• http//identiscape.stanford.edu

46
Attacks against Time and History

• Time failures
• Going back in time
• Reporting different time to different peers
• History failures
• Changing history
• Forgetting history
• Reporting different versions of history to
  different peers

47
Lamport Clock Properties

• Observable Precedence
• If a and b happen in the same system and a comes
  before b, then a precedes b
• Message transmission always precedes message
  reception
• In an observably consistent clock, if a
  precedes b, then clock(a)ltclock(b)

48
Lamports Logical Clocks

• Invented by Lamport in 1978
• They help maintain partial ordering of events in
  distributed systems
• Every system maintains a logical clock
• Messages communicate the senders logical time to
  the receiver
• Receiver adjust his clock to maintain temporal
  ordering

49
Lamport Clock Operation

• Every system increments its logical time after
  every event
• Every message sent contains the current time
• When a system receives a message, it sets its
  time to be greater than the maximum of its time
  and the local clock

50
Scope of Lamport Clocks

• Everyone is trusted to perform correctly
• When I get a message, I dutifully adjust my clock
• I always advance my clock between events
• Failures are benign
• Crashes
• Network disconnection

51
Authenticated Search Trees

• Each node is labeled
• Root Hash is the digest
• Inclusion proof is based on
• The path from the root to the node
• All siblings of the nodes ancestors

52
Maintaining Archive Snapshots

• Version 1 has keys a, b, c d
• Version 2 is Version 1 without key a
• If I time stamp root 2, I time stamp the snapshot
  of the tree without a, i.e., as removal

53
RBB Trees
54
Persistence in RBB-trees