random variables

1
UNIT -II

• Topics covered from
• Chapter 1
• (Communication Systems-Simon Hykin)

Prof. Parjane M.A.
2
Why do we need Probability?

• We have several graphical and numerical
  statistics for summarizing our data
• We want to make probability statements about the
  significance of our statistics
• Eg. In Stat111, mean(height) 66.7 inches
• What is the chance that the true height of Penn
  students is between 60 and 70 inches?
• Eg. r -0.22 for draft order and birthday
• What is the chance that the true correlation is
  significantly different from zero?

3
Deterministic vs. Random Processes

• In deterministic processes, the outcome can be
  predicted exactly in advance
• Eg. Force mass x acceleration. If we are
  given values for mass and acceleration, we
  exactly know the value of force
• In random processes, the outcome is not known
  exactly, but we can still describe the
  probability distribution of possible outcomes
• Eg. 10 coin tosses we dont know exactly how
  many heads we will get, but we can calculate the
  probability of getting a certain number of heads

4
Events

• An event is an outcome or a set of outcomes of a
  random process
• Example Tossing a coin three times
• Event A getting exactly two heads HTH, HHT,
  THH
• Example Picking real number X between 1 and 20
• Event A chosen number is at most 8.23 X
  8.23
• Example Tossing a fair dice
• Event A result is an even number 2, 4, 6
• Notation P(A) Probability of event A
• Probability Rule 1
• 0 P(A) 1 for any event A

5
Sample Space

• The sample space S of a random process is the set
  of all possible outcomes
• Example one coin toss
• S H,T
• Example three coin tosses
• S HHH, HTH, HHT, TTT, HTT, THT, TTH, THH
• Example roll a six-sided dice
• S 1, 2, 3, 4, 5, 6
• Example Pick a real number X between 1 and 20
• S all real numbers between 1 and 20
• Probability Rule 2 The probability of the whole
  sample space is 1
• P(S) 1

6
Combinations of Events

• The complement Ac of an event A is the event that
  A does not occur
• Probability Rule 3
• P(Ac) 1 - P(A)
• The union of two events A and B is the event that
  either A or B or both occurs
• The intersection of two events A and B is the
  event that both A and B occur

Event A
Complement of A
Union of A and B
Intersection of A and B
7
Disjoint Events

• Two events are called disjoint if they can not
  happen at the same time
• Events A and B are disjoint means that the
  intersection of A and B is zero
• Example coin is tossed twice
• S HH,TH,HT,TT
• Events AHH and BTT are disjoint
• Events AHH,HT and B HH are not disjoint
• Probability Rule 4 If A and B are disjoint
  events then
• P(A or B) P(A) P(B)

8
Independent events

• Events A and B are independent if knowing that A
  occurs does not affect the probability that B
  occurs
• Example tossing two coins
• Event A first coin is a head
• Event B second coin is a head
• Disjoint events cannot be independent!
• If A and B can not occur together (disjoint),
  then knowing that A occurs does change
  probability that B occurs
• Probability Rule 5 If A and B are independent
• P(A and B) P(A) x P(B)

Independent
multiplication rule for independent events
9
Equally Likely Outcomes Rule

• If all possible outcomes from a random process
  have the same probability, then
• P(A) ( of outcomes in A)/( of outcomes in S)
• Example One Dice Tossed
• P(even number) 2,4,6 / 1,2,3,4,5,6
• Note equal outcomes rule only works if the
  number of outcomes is countable
• Eg. of an uncountable process is sampling any
  fraction between 0 and 1. Impossible to count
  all possible fractions !

10
Combining Probability Rules Together

• Initial screening for HIV in the blood first uses
  an enzyme immunoassay test (EIA)
• Even if an individual is HIV-negative, EIA has
  probability of 0.006 of giving a positive result
• Suppose 100 people are tested who are all
  HIV-negative. What is probability that at least
  one will show positive on the test?
• First, use complement rule
• P(at least one positive) 1 - P(all negative)

11
Combining Probability Rules Together

• Now, we assume that each individual is
  independent and use the multiplication rule for
  independent events
• P(all negative) P(test 1 negative) P(test
  100 negative)
• P(test negative) 1 - P(test positive) 0.994
• P(all negative) 0.994 0.994 (0.994)100
• So, we finally we have
• P(at least one positive) 1- (0.994)100 0.452

12
Conditional Probabilities

• The notion of conditional probability can be
  found in many different types of problems
• Eg. imperfect diagnostic test for a disease
• What is probability that a person has the
  disease? Answer 40/100 0.4
• What is the probability that a person has the
  disease given that they tested positive?
• More Complicated !

Disease Disease - Total
Test 30 10 40
Test - 10 50 60
Total 40 60 100
13
Definition Conditional Probability

• Let A and B be two events in sample space
• The conditional probability that event B occurs
  given that event A has occurred is
• P(AB) P(A and B) / P(B)
• Eg. probability of disease given test positive
• P(disease test ) P(disease and test ) /
  P(test ) (30/100)/(40/100) .75

14
Independent vs. Non-independent Events

• If A and B are independent, then
• P(A and B) P(A) x P(B)
• which means that conditional probability is
• P(B A) P(A and B) / P(A) P(A)P(B)/P(A)
  P(B)
• We have a more general multiplication rule for
  events that are not independent
• P(A and B) P(B A) P(A)

15
Random variables

• A random variable is a numerical outcome of a
  random process or random event
• Example three tosses of a coin
• S HHH,THH,HTH,HHT,HTT,THT,TTH,TTT
• Random variable X number of observed tails
• Possible values for X 0,1, 2, 3
• Why do we need random variables?
• We use them as a model for our observed data

16
Discrete Random Variables

• A discrete random variable has a finite or
  countable number of distinct values
• Discrete random variables can be summarized by
  listing all values along with the probabilities
• Called a probability distribution
• Example number of members in US families

X 2 3 4 5 6 7
P(X) 0.413 0.236 0.211 0.090 0.032 0.018
17
Another Example

• Random variable X the sum of two dice
• X takes on values from 2 to 12
• Use equally-likely outcomes rule to calculate
  the probability distribution
• If discrete r.v. takes on many values, it is
  better to use a probability histogram

X 2 3 4 5 6 7 8 9 10 11 12
of Outcomes 1 2 3 4 5 6 5 4 3 2 1
P(X) 1/36 2/36 3/36 4/36 5/36 6/36 5/36 4/36 3/36 2/36 1/36
18
Probability Histograms

• Probability histogram of sum of two dice
• Using the disjoint addition rule, probabilities
  for discrete random variables are calculated by
  adding up the bars of this histogram
• P(sum gt 10) P(sum 11) P(sum 12) 3/36

19
Continuous Random Variables

• Continuous random variables have a non-countable
  number of values
• Cant list the entire probability distribution,
  so we use a density curve instead of a histogram
• Eg. Normal density curve

20
Calculating Continuous Probabilities

• Discrete case add up bars from probability
  histogram
• Continuous case we have to use integration to
  calculate the area under the density curve
• Although it seems more complicated, it is often
  easier to integrate than add up discrete bars
• If a discrete r.v. has many possible values, we
  often treat that variable as continuous instead

21
Example Normal Distribution

• We will use the normal distribution throughout
• this course for two reasons
• It is usually good approximation to real data
• We have tables of calculated areas under the
  normal curve, so we avoid doing integration!

22
Mean of a Random Variable

• Average of all possible values of a random
  variable (often called expected value)
• Notation dont want to confuse random variables
  with our collected data variables
• ? mean of random variable
• x mean of a data variable
• For continuous r.v, we again need integration to
  calculate the mean
• For discrete r.v., we can calculate the mean by
  hand since we can list all probabilities

23
Mean of Discrete random variables

• Mean is the sum of all possible values, with each
  value weighted by its probability
• µ S xiP(xi) x1P(x1) x12P(x12)
• Example X sum of two dice
• µ 2 (1/36) 3 (2/36) 4 (3/36) 12
  (1/36)
• 252/36 7

X 2 3 4 5 6 7 8 9 10 11 12
P(X) 1/36 2/36 3/36 4/36 5/36 6/36 5/36 4/36 3/36 2/36 1/36
24
Variance of a Random Variable

• Spread of all possible values of a random
  variable around its mean?
• Again, we dont want to confuse random variables
  with our collected data variables
• ?2 variance of random variable
• s2 variance of a data variable
• For continuous r.v, again need integration to
  calculate the variance
• For discrete r.v., can calculate the variance by
  hand since we can list all probabilities

25
Variance of Discrete r.v.s

• Variance is the sum of the squared deviations
  away from the mean of all possible values,
  weighted by the values probability
• µ S(xi-µ)P(xi) (x1-µ)P(x1)
  (x12-µ)P(x12)
• Example X sum of two dice
• s2 (2 - 7)2(1/36) (3- 7)2(2/36) (12 -
  7)2(1/36)
• 210/36 5.83

X 2 3 4 5 6 7 8 9 10 11 12
P(X) 1/36 2/36 3/36 4/36 5/36 6/36 5/36 4/36 3/36 2/36 1/36
26
Energy and Power Signals

• The performance of a communication system depends
  on the received signal energy higher energy
  signals are detected more reliably (with fewer
  errors) than are lower energy signals.
• An electrical signal can be represented as a
  voltage v(t) or a current i(t) with instantaneous
  power p(t) across a resistor defined by
• OR

27
Energy and Power Signals

• In communication systems, power is often
  normalized by assuming R to be 1.
• The normalization convention allows us to express
  the instantaneous power as
• where x(t) is either a voltage or a current
  signal.
• The energy dissipated during the time interval
  (-T/2, T/2) by a real signal with instantaneous
  power expressed by Equation (1.4) can then be
  written as
• The average power dissipated by the signal during
  the interval is

28
Energy and Power Signals

• We classify x(t) as an energy signal if, and only
  if, it has nonzero but finite energy (0 lt Ex lt 8)
  for all time, where
• An energy signal has finite energy but zero
  average power
• Signals that are both deterministic and
  non-periodic are termed as Energy Signals

29
Energy and Power Signals

• Power is the rate at which the energy is
  delivered
• We classify x(t) as an power signal if, and only
  if, it has nonzero but finite energy (0 lt Px lt 8)
  for all time, where
• A power signal has finite power but infinite
  energy
• Signals that are random or periodic termed as
  Power Signals

30
Random Variable

• Functions whose domain is a sample space and
  whose range is a some set of real numbers is
  called random variables.
• Type of RVs
• Discrete
• E.g. outcomes of flipping a coin etc
• Continuous
• E.g. amplitude of a noise voltage at a particular
  instant of time

31
Random Variables

• Random Variables
• All useful signals are random, i.e. the receiver
  does not know a priori what wave form is going to
  be sent by the transmitter
• Let a random variable X(A) represent the
  functional relationship between a random event A
  and a real number.
• The distribution function Fx(x) of the random
  variable X is given by

32
Random Variable

• A random variable is a mapping from the sample
  space to the set of real numbers.
• We shall denote random variables by boldface,
  i.e., x, y, etc., while individual or specific
  values of the mapping x are denoted by x(w).

33
Random process

• A random process is a collection of time
  functions, or signals, corresponding to various
  outcomes of a random experiment. For each
  outcome, there exists a deterministic function,
  which is called a sample function or a
  realization.

Random variables
Sample functions or realizations (deterministic
function)
34
Random Process

• A mapping from a sample space to a set of time
  functions.

35
Random Process contd

• Ensemble The set of possible time functions that
  one sees.
• Denote this set by x(t), where the time functions
  x1(t, w1), x2(t, w2), x3(t, w3), . . . are
  specific members of the ensemble.
• At any time instant, t tk, we have random
  variable x(tk).
• At any two time instants, say t1 and t2, we have
  two different random variables x(t1) and x(t2).
• Any realationship b/w any two random variables is
  called Joint PDF

36
Classification of Random Processes

• Based on whether its statistics change with time
  the process is non-stationary or stationary.
• Different levels of stationary
• Strictly stationary the joint pdf of any order
  is independent of a shift in time.
• Nth-order stationary the joint pdf does not
  depend on the time shift, but depends on time
  spacing

37
Cumulative Distribution Function (cdf)

• cdf gives a complete description of the random
  variable. It is defined as
• FX(x) P(E ? S X(E) x) P(X x).
• The cdf has the following properties
• 0 FX(x) 1 (this follows from Axiom 1 of the
  probability measure).
• Fx(x) is non-decreasing Fx(x1) Fx(x2) if x1
  x2 (this is because event x(E) x1 is contained
  in event x(E) x2).
• Fx(-8) 0 and Fx(8) 1 (x(E) -8 is the empty
  set, hence an impossible event, while x(E) 8 is
  the whole sample space, i.e., a certain event).
• P(a lt x b) Fx(b) - Fx(a).

38
Probability Density Function

• The pdf is defined as the derivative of the cdf
• fx(x) d/dx Fx(x)
• It follows that
• Note that, for all i, one has pi 0 and ?pi 1.

39
Cumulative Joint PDF

• Often encountered when dealing with combined
  experiments or repeated trials of a single
  experiment.
• Multiple random variables are basically
  multidimensional functions defined on a sample
  space of a combined experiment.
• Experiment 1
• S1 x1, x2, ,xm
• Experiment 2
• S2 y1, y2 , , yn
• If we take any one element from S1 and S2
• 0 lt P(xi, yj) lt 1 (Joint Probability of two or
  more outcomes)
• Marginal probabilty distributions
• Sum all j P(xi, yj) P(xi)
• Sum all i P(xi, yj) P(yi)

40
Expectation of Random Variables (Statistical
averages)

• Statistical averages, or moments, play an
  important role in the characterization of the
  random variable.
• The first moment of the probability distribution
  of a random variable X is called mean value mx or
  expected value of a random variable X
• The second moment of a probability distribution
  is mean-square value of X
• Central moments are the moments of the difference
  between X and mx, and second central moment is
  the variance of x.
• Variance is equal to the difference between the
  mean-square value and the square of the mean

41
Contd

• The variance provides a measure of the variables
  randomness.
• The mean and variance of a random variable give a
  partial description of its pdf.

42
Time Averaging and Ergodicity

• A process where any member of the ensemble
  exhibits the same statistical behavior as that of
  the whole ensemble.
• For an ergodic process To measure various
  statistical averages, it is sufficient to look at
  only one realization of the process and find the
  corresponding time average.
• For a process to be ergodic it must be
  stationary. The converse is not true.

43
Gaussian (or Normal) Random Variable (Process)

• A continuous random variable whose pdf is
• µ and are parameters. Usually denoted as
• N(µ, ) .
• Most important and frequently encountered random
  variable in communications.

44
Central Limit Theorem

• CLT provides justification for using Gaussian
  Process as a model based if
• The random variables are statistically
  independent
• The random variables have probability with same
  mean and variance

45
CLT

• The central limit theorem states that
• The probability distribution of Vn approaches a
  normalized Gaussian Distribution N(0, 1) in the
  limit as the number of random variables approach
  infinity
• At times when N is finite it may provide a poor
  approximation of for the actual probability
  distribution

46
Autocorrelation

• Autocorrelation of Energy Signals
• Correlation is a matching process
  autocorrelation refers to the matching of a
  signal with a delayed version of itself
• The autocorrelation function of a real-valued
  energy signal x(t) is defined as
• The autocorrelation function Rx(?) provides a
  measure of how closely the signal matches a copy
  of itself as the copy is shifted ? units in time.
• Rx(?) is not a function of time it is only a
  function of the time difference ? between the
  waveform and its shifted copy.

47
Autocorrelation

• symmetrical in ? about zero
• maximum value occurs at the origin
• autocorrelation and ESD form a Fourier transform
  pair, as designated by the double-headed arrows
• value at the origin is equal to the energy of the
  signal

48
AUTOCORRELATION OF A PERIODIC (POWER) SIGNAL

• The autocorrelation function of a real-valued
  power signal x(t) is defined as
• When the power signal x(t) is periodic with
  period T0, the autocorrelation function can be
  expressed as

49
Autocorrelation of power signals
The autocorrelation function of a real-valued
periodic signal has properties similar to those
of an energy signal

• symmetrical in ? about zero
• maximum value occurs at the origin
• autocorrelation and PSD form a Fourier transform
  pair, as designated by the double-headed arrows
• value at the origin is equal to the average power
  of the signal

50
(No Transcript)
51
(No Transcript)
52
Spectral Density
53
Matched Filters

• A very useful technique for detecting the
  presence of a signal of a certain shape in the
  presence of noise is the matched filter.
• The matched filter uses correlation to detect the
  signal so this filter is sometimes called a
  correlation filter
• It is often used to detect 1s and 0s in a
• binary data stream

54

• It has been shown that the optimal filter to
  detect a noisy signal is one whose impulse
  response is proportional to the time inverse of
  the signal. Here are some examples of wave shapes
  encoding 1s and 0s and the impulse responses of
  matched filters.

55

• Even in the presence of a large additive noise
  signal the matched filter indicates with a high
  response level the presence of a 1 and with a low
  response level the presence of a 0. Since the 1
  and 0 are encoded as the negatives of each other,
  one matched filter optimally detects both.

56
SPECTRAL DENSITY

• The spectral density of a signal characterizes
  the distribution of the signals energy or power,
  in the frequency domain
• This concept is particularly important when
  considering filtering in communication systems
  while evaluating the signal and noise at the
  filter output.
• The energy spectral density (ESD) or the power
  spectral density (PSD) is used in the evaluation.
• Need to determine how the average power or energy
  of the process is distributed in frequency.

57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60

• Power spectral density (PSD) applies to power
  signals in the same way that energy spectral
  density applies to energy signals.
• The PSD of a signal x is conventionally indicated
  by the notation, Gx(F) or Gx(f) .
• In an LTI system,
• Also, for a power signal, PSD and autocorrelation
  form a Fourier transform pair.

61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
Power Spectral Density
Let T T2 - T1 OR T1T2 - T
70
(No Transcript)
71
1.9 Noise
Noise unwanted signals that tend to disturb the
transmission and processing of signals.
72
Shot Noise

• Shot noise arises in electronic devices such as
  diodes and transistors because of the discrete
  nature of current flow in these devices.
• It is difficult to describe statistical
  characterization of the shot-noise process.

73
Thermal Noise
It is the name given to the electrical noise
arising from the random motion of electrons in a
conductor .
Mean-square value of the thermal noise voltage
k 1.38x10-23 joules, Boltzmanns constant T
absolute temperature in degrees Kelvin
T273Co
74
It is of interest to note that the number of
electrons in a resistor is very large and their
random motions inside the resistor are
statistically independent of each other, the
central limit theorem indicates that thermal
noise is Gaussian distributed with zero mean.
75
White Noise
Its power spectral density is independent of the
operating frequency.
The dimensions of No are in watts per Hertz.
Why is it called white noise ?
The adjective white is used in the sense that
white light contains equal amounts of all
frequencies within the visible band of
electromagnetic radiation.
76
N0 is usually referenced to the input stage of
the receiver of a communication system. It may be
expressed as
Where k is Boltzmanns constant and Te is the
equivalent noise temperature of the receiver.
Te is defined as the temperature at which a noisy
resistor has to be maintained such that, by
connecting the resistor to the input of a
noiseless version of a system, it produces the
same available noise power at the output of the
system as that produced by the sources of noise
in the actual system.
77
Figure 1.16 Characteristics of white noise. (a)
Power spectral density. (b) Autocorrelation
function.
78
Discussionwhite noise
Strictly speaking, white noise has infinite
average power and it is not physically
realizable.
Is it useful in practical system analysis?
As long as the bandwidth of a noise process at
the input of a system is appreciable larger than
that of the system itself, then we may model the
noise process as white noise.
79
Additive white Gaussian noise AWGN
Additive noise
Probability density function
Power spectral density
80
Figure 1.17 Characteristics of low-pass filtered
white noise. (a) Power spectral density. (b)
Autocorrelation function.
Example 1.10Ideal Low-Pass Filtered White Noise
81
Narrowband Noise
The noise process appearing at the output of a
narrow band filter is called narrowband noise.
Figure 1.18 (a) Power spectral density of
narrowband noise. (b) Sample function of
narrowband noise.
82
Representations of narrowband noise

1. Represented in terms of In-phase and quadrature
   components
2. Represented in terms of Envelope and phase

83
1.11 Representation of Narrowband Noise in
terms of In-Phase and Quadrature Components
nI(t) in-phase component nQ(t) quadrature
componentboth of them are low-pass signals.
84
Properties of Narrowband Noise

1. nI(t) and nQ(t) have zero mean.
2. If n(t) is Gaussian, then nI(t) and nQ(t) are
   jointly Gaussian.
3. If n(t) is stationary, then nI(t) and nQ(t) are
   jointly stationary.

5. nI(t) and nQ(t) have the same variance as n(t).
85
Example 1.12 Ideal Band-Pass Filtered White
Noise
86
Figure 1.20 Characteristics of ideal band-pass
filtered white noise. (a) Power spectral
density. (b) Autocorrelation function. (c) Power
spectral density of in-phase and quadrature
components.
87
1.12 Representation of Narrowband Noise in
terms of Envelope and Phase
88
Joint probability density function of r and ?
Phase ? is uniformly distributed
Envelope r is Rayleigh distributed