Invertibility, MaximumLikelihood

1
Chalmers 5 Dec. 2000
Invertibility, Maximum-Likelihood Estimation,
and Multiple Accessing
James L. Massey Prof. em. ETH-Zürich Adjunct
Professor, Lund University Trondhjemsgade 3,
2t.h. DK-2100 København Ø JamesMassey_at_compuserve.
com
2
ESTIMATION THEORY
Observation
Parameter of interest
Estimate
Assumption
Is known for every
.
in the support of
This support is known, but may itself not be
known.
3
MAP Estimation Rule
When
, choose
where
maximizes
Maximum-Likelihood (ML) Estimation Rule
When
, choose
where lies
in the support of and maximizes
4
An Example of ML Estimation
?
?
White Gaussian dim n, variance ?2
The support of
is the whole of Rn.
maximizes
But
implies that
so that
ML estimator is a straight wire!
It follows that
Same holds true for any noise density with max at
0.
5
?
But suppose that
is actually the system
LDS
Sequence Generator
and that we are
really interested in estimating rather than
HOW CAN WE DO THIS?
For whatever it may be worth, we note that the
above system is entirely equivalent to the system
LDS
?
6

• A principle is the recognition of a simple
  equivalence between apparently different things.
• Some Noteworthy examples
• Principle of interchangeability of the input and
  the unit-sample response of a linear
  time-invariant system.
• Principle of the conservation of momentum.
• Invertibility principle for maximum-likelihood
  estimation.

7
The invertibility principle for ML estimation
If where f is an invertible
function from the support of to the
support of then, for the same observation,
ML Estimator for
PROOF
and
are the same event
so that
8
N.B. The invertibility principle does not hold
for MAP estimation (nor Bayesian estimation, nor
LMMSE, etc.)
Parameter Source
?
Gaussian variance ?2
Suppose that for
Thus, for
for
9
For the situation
LDS
?
?
White Gaussian variance ?2
the invertibility principle tells us that the ML
estimate of can be made as follows
LDS
where is the unit-sample response of the
stable LDS with transfer function
, i.e., by the inverse filter.
10
Note that for the ML estimator of , we have
LDS
LDS
?
?
White Gaussian variance ?2
This shows that the ML estimator gives an
undistorted picture of . There are no
sidelobes in the response of the inverse filter
to the signal , such as there are in a
matched filter estimator, that would distort
the picture of .
11
If , then
so that the zeroes of inside
the unit circle determine for ,
while the zeroes outside the unit circle
determine for If has
zeroes on the unit circle, then the inverse
filter does not exist.
The inverse filter is usually non-causal, but so
what?
Q Because the inverse filter has the greatest
gain at those frequencies where the signal
is weakest, wont this ML estimate of be
much noisier than the matched filter estimate?
A Yesif we make a stupid choice of .
Noif we make a smart choice.
12
A simple, but slightly misleading example
One period of m-sequence
7
4
inverse
matched
0
k
-1
13
A fully honest example
1
k
-1
k
14
For the fully honest example, the matched filter
is
1
k
-1
J. Ruprecht has shown that, for well-chosen
long sequences , the degradation of the
signal-to-noise ratio of the inverse filter
compared to the matched filter is less than 1 dB!
15
(Rotational speed is 2B ? 2W buckets/second.)
Fouriers conveyor belt for Fourier Bandwidth W B
is the Shannon bandwidth.
Each bucket holds one amplitude-modulated signal
Ak?k(t), where ?k(t) -? lt k lt ? is an
orthonormal set of signals with Fourier bandwidth
W.
Equality holds here for shifted sinc signals
where
16
To say that a signal has Shannon Bandwidth B
means that 2B orthonormal signals per second are
required to represent the signal.
B gt 0 means that the signal cannot be
deterministic!
Example The signal U3? with a ?1 coin toss or a
0 every 3 samples, with 0s in between. A
typical sample function is
has the same Shannon (but not Fourier) bandwidth
as the signal
obtained by spreading with the sequence
17
The Fundamental Theorem of Bandwidth is just the
assertion that the Shannon Bandwidth B cannot
exceed the Fourier Bandwidth W, but equality can
be achieved.
B is the bandwidth needed by the signal. W is the
bandwidth used by the signal. The spreading
factor is the ratio W/B.

• Why should one spread a signal?
• To mitigate multipath effects.
• To hide signals
• ? Good ElectroMagnetic Compatibility (EMC)
• ? Multiple-Accessing Capability
• How should we spread for multiple accessing?

18
Should we use Code-Division Multiple Access
(CDMA)?
LDS s(1)
MF for s(1)
Delay ?1
LDS s(2)
MF for s(2)
Delay ?2
?
LDS s(M)
MF for s(M)
Delay ?M
WGN
Spreading
Receiver filters

• Advantages
• No synchronization of users required
• Equality of peak power and average power
• Graceful degradation with increasing number of
  users
• Disadvantages
• Many different receiver filters
• Inter-user interferencenear-far problem

19
Should we use CodeTime-Division Multiple Access
(CTDMA) in which users are separated by the times
they send their codes?
LDS
?1 0 chips
LDS
?2 ? chips
LDS
?
LDS
?M (M-1)?
WGN
Spreading
Receiver filter

• Advantages
• Only one receiver filter
• No inter-user interferenceno near-far problem
• Equality of peak power and average power
• Disadvantages
• Synchronization of users required
• Definite maximum number of users (M? lt n
  required)
• Requires a good choice of to minimize snr
  loss.

20
Example with M 2 and n 6.
? 3 chips
The principle of interchangeability of the input
and the unit-sample response of a linear
time-invariant system implies that and
can be interchanged. Very useful!
Provided the multipath spread is less than ?
chips, CTDMA provides an essentially free ML
estimate of the multipath.
LDS
Channel hi
LDS
Channel hi

21
DETECTION THEORY
Observation
Quantity of interest must be a discrete random
variable.
Decision
Assumption
Is known for every
.
in the support of
This support is known, but may itself not be
known.
22
MAP Decision Rule
When
, choose
where
maximizes
Maximum-Likelihood (ML) Decision Rule
When
, choose
where lies
in the support of and maximizes
23
The boring invertibility principle for ML
decisions
If where f is an invertible
function from the support of to the
support of then, for the same observation,
ML Decider for
PROOF
and
are the same event
so that
24
WHY BORING?
Because the invertibility principle also holds
for MAP decisions, Bayesian decisions, and
virtually every other kind of decision that makes
sense.
PROOF for MAP decisions
and
are the same event
so that
But this does not mean that the invertibility
principle for decisions is not useful!
25
Forneys decoder-equivalence nugget!
Every decoder
LINEAR ENCODER
DECODER
DECODED SEQUENCES
INFORMATION SEQUENCES
CODE WORD
RECEIVED DATA
CHANNEL
can be realized as
ENCODER INVERSE
LINEAR ENCODER
DECODED SEQUENCES
INFORMATION SEQUENCES
CODE WORD
CODE WORD DECISION
CODE WORD DECIDER
RECEIVED DATA
Same system with decoder in two parts.
26
However, for the random variables to be
well-defined, it is necessary that the encoder
inverse, which is an LDS over a finite field, be
stable, i.e., that it have a feedforward
inverse. This is why one cannot use
catastrophic encoders, i.e., encoders that do
not have a feedforward inverse in conjunction
with Viterbi (i.e., ML) decoding.
FEEDFORWARD ENCODER INVERSE
CONVOLUTIONAL ENCODER
ML DECODED SEQUENCES
INFORMATION SEQUENCES
CODE WORD
CODE WORD ML DECISION
VITERBI DECODER
RECEIVED DATA
27
Some relevant references J. Ruprecht,
Maximum-Likelihood Estimation of Multipath
Channels. Konstanz Hartung-Gorre Verlag, 1989.
(ETH Zürich Dissertation No. 8789.) G. Kramer,
U. Loher and J. Ruprecht, Code-Time Division
Multiple Access, Proc. GLOBECOM 96. J. L.
Massey, "Towards an Information Theory of
Spread-Spectrum Systems," in Code Division
Multiple Access Communications (Eds. S. G. Glisic
and P. A. Leppänen). Boston, Dordrecht and
London Kluwer, 1995, pp. 29-46. Send me an
e-mail request and I will send you a version with
the equations compiled! I will also be glad to
send you the PowerPoint file of these slides.