Information Theory and Security

1
Information Theory and Security
2
Lecture Motivation

• Up to this point we have seen
• Classical Crypto
• Symmetric Crypto
• Asymmetric Crypto
• These systems have focused on issues of
  confidentiality Ensuring that an adversary
  cannot infer the original plaintext message, or
  cannot learn any information about the original
  plaintext from the ciphertext.
• But what does information mean?
• In this lecture and the next we will put a more
  formal framework around the notion of what
  information is, and use this to provide a
  definition of security from an information-theoret
  ic point of view.

3
Lecture Outline

• Probability Review Conditional Probability and
  Bayes
• Entropy
• Desired properties and definition
• Chain Rule and conditioning
• Coding and Information Theory
• Huffman codes
• General source coding results
• Secrecy and Information Theory
• Probabilistic definitions of a cryptosystem
• Perfect Secrecy

4
The Basic Idea

• Suppose we roll a 6-sided dice.
• Let A be the event that the number of dots is
  odd.
• Let B be the event that the number of dots is at
  least 3.
• A 1, 3, 5
• B 3, 4, 5, 6
• I tell you the roll belongs to both A and B then
  you know there are only two possibilities 3, 5
• In this sense tells you more than
  just A or just B.
• That is, there is less uncertainty in
  than in A or B.
• Information is closely linked with this idea of
  uncertainty Information increases when
  uncertainty decreases.

5
Probability Review, pg. 1

• A random variable (event) is an experiment whose
  outcomes are mapped to real numbers.
• For our discussion we will deal with
  discrete-valued random variables.
• Probability We denote pX(x) Pr(X x).
• For a subset A,
• Joint Probability Sometimes we want to consider
  more than two events at the same time, in which
  we case we lump them together into a joint random
  variable, e.g. Z (X,Y).
• Independence We say that two events are
  independent if

6
Probability Review, pg. 2

• Conditional Probability We will often ask
  questions about the probability of events Y given
  that we have observed Xx. In particular, we
  define the conditional probability of Yy given
  Xx by
• Independence We immediately get
• Bayess Theorem If pX(x)gt0 and pY(y)gt0 then

7
Example

• Example Suppose we draw a card from a standard
  deck. Let X be the random variable describing the
  suit (e.g. clubs, diamonds, hearts, spades). Let
  Y be the value of the card (e.g. two, three, ,
  ace). Then Z(X,Y) gives the 52 possibilities for
  the card.
• P( (X,Y) (x,y) ) P(Xx, Yy) 1/52
• P(Xclubs) 13/52 ¼
• P(Y3) 4/52 1/13

8
Entropy and Uncertainty

• We are concerned with how much uncertainty a
  random event has, but how do we define or measure
  uncertainty?
• We want our measure to have the following
  properties
• To each set of nonnegative numbers
  with
  , we define the uncertainty by
  .
• should be a continuous function A
  slight change in p should not drastically
  change
• 
  for all ngt0. Uncertainty increases when there are
  more outcomes.
• If 0ltqlt1, then

9
Entropy, pg. 2

• We define the entropy of a random variable by
• Example Consider a fair coin toss. There are two
  outcomes, with probability ½ each. The entropy is
• Example Consider a non-fair coin toss X with
  probability p of getting heads and 1-p of getting
  tails. The entropy is
• The entropy is maximum when p ½.

10
Entropy, pg. 3

• Entropy may be thought of as the number of yes-no
  questions needed to accurately determine the
  outcome of a random event.
• Example Flip two coins, and let X be the number
  of heads. The possibilities are 0,1,2 and the
  probabilities are 1/4, 1/2, 1/4. The Entropy is
• So how can we relate this to questions?
• First, ask Is there exactly one head? You will
  half the time get the right answer
• Next, ask Are there two heads?
• Half the time you needed one question, half you
  needed two

11
Entropy, pg. 4

• Suppose we have two random variables X and Y, the
  joint entropy H(X,Y) is given by
• Conditional Entropy In security, we ask
  questions of whether an observation reduces the
  uncertainty in something else. In particular, we
  want a notion of conditional entropy. Given that
  we observe event X, how much uncertainty is left
  in Y?

12
Entropy, pg. 5

• Chain Rule The Chain Rule allows us to relate
  joint entropy to conditional entropy via H(X,Y)
  H(YX)H(X).
• (Remaining details will be provided on the white
  board)
• Meaning Uncertainty in (X,Y) is the uncertainty
  of X plus whatever uncertainty remains in Y given
  we observe X.

13
Entropy, pg. 6

• Main Theorem
• Entropy is non-negative.
• where denotes the
  number of elements in the sample space of X.
• (Conditioning reduces entropy)
• with equality if and only if X and Y are
  independent.

14
Entropy and Source Coding Theory

• There is a close relationship between entropy and
  representing information.
• Entropy captures the notion of how many Yes-No
  questions are needed to accurately identify a
  piece of information that is, how many bits are
  needed!
• One of the main focus areas in the field of
  information theory is on the issue of
  source-coding
• How to efficiently (Compress) information into
  as few bits as possible.
• We will talk about one such technique, Huffman
  Coding.
• Huffman coding is for a simple scenario, where
  the source is a stationary stochastic process
  with independence between successive source
  symbols

15
Huffman Coding, pg. 1

• Suppose we have an alphabet with four letters A,
  B, C, D with frequencies
• We could represent this with A00, B01, C10,
  D11. This would mean we use an average of 2 bits
  per letter.
• On the other hand, we could use the following
  representation A1, B01, C001, D000. Then the
  average number of bits per letter becomes
• (0.5)1(0.3)2(0.1)3(0.1)3 1.7
• Hence, this representation, on average, is more
  efficient.

A B C D 0.5 0.3 0.1 0.1
16
Huffman Coding, pg. 2

• Huffman Coding is an algorithm that produces a
  representation for a source.
• The Algorithm
• List all outputs and their probabilities
• Assign a 1 and 0 to smallest two, and combine to
  form an output with probability equal to the sum
• Sort List according to probabilities and repeat
  the process
• The binary strings are then obtained by reading
  backwards through the procedure

1
A
0.5
1.0
1
B
0.3
0.5
1
0
C
0.1
0.2
0
D
0.1
0
Symbol Representations A 1 B 01 C 001 D 000
17
Huffman Coding, pg. 3

• In the previous example, we used probabilities.
  We may directly use event counts.
• Example Consider 8 symbols, and suppose we have
  counted how many times they have occurred in an
  output sample.
• We may derive the Huffman Tree
• The corresponding length vector is
  (2,2,3,3,3,4,5,5)
• The average codelength is 2.83. If we had used a
  full-balanced tree representation (i.e. the
  straight-forward representation) we would have
  had an average codelength of 3.

S1 S2 S3 S4 S5 S6 S7 S8 28 25 20 16 15 8 7 5
18
Huffman Coding, pg. 4

• We would like to quantify the average amount of
  bits needed in terms of entropy.
• Theorem Let L be the average number of bits per
  output for Huffman encoding of a random variable
  X, then
• Here, lx length of codeword assigned to symbol
  x.
• Example Lets look back at the 4 symbol example
• Our average codelength was 1.7 bits.

19
Next Time

• We will look at how entropy is related to
  security
• Generalized definition of encryption
• Perfect Secrecy
• Manipulating entropy relationships
• The next computer project will also be handed out