Computability

1
Computability Complexity I

• Chris Umans
• Caltech

2
Outline

• Turing Machines, languages
• Halting Problem
• reductions
• Rices Theorem
• Time Complexity Classes
• P vs. EXP

3
From previous lectures

• many equivalent models of computation
• we will standardize on TMs
• consider decision problems
• also called languages (sets of strings)

input tape
finite control

1
1
0
0
1
1
0
0
0
0
1
1
read/write head
q0
4
From previous lectures

• Can describe a TM to implement any procedure for
  which we could imagine writing a program or
  algorithm
• but perhaps tedious to write out
• Can construct a Universal TM that recognizes the
  language
• ltM, wgt M is a TM and M accepts w
• there is a general purpose TM whose input can be
  a program to run

5
Church-Turing Thesis

• the belief that TMs formalize our intuitive
  notion of an algorithm is
• Note this is a belief, not a theorem.

The Church-Turing Thesis everything we can
compute on a physical computer can be computed
on a Turing Machine
6
Deciding and Recognizing

• accept
• reject
• loop forever

TM M
input

• L(M) strings that M accepts
• M recognizes L(M)
• set of languages recognized by some TM is called
  Turing-recognizable or recursively enumerable
  (RE)
• if M rejects all x ? L(M) we say it decides L(M)
• set of languages decided by some TM is called
  Turing-decidable or decidable or recursive

7
Do all problems have an algorithm that solves
them?
decidable
all languages
RE

• decidable ? RE ? all languages
• are these containments proper?

8
Undecidability

• Definition of the Halting Problem
• HALT ltM, wgt TM M halts on input w
• HALT is Turing-recognizable (RE)
• proof?
• Theorem HALT is not decidable (undecidable).

9
The Halting Problem

• HALT ltM, wgt TM M halts on input w
• Proof
• suppose TM H decides HALT
• define new TM H on input M
• if H accepts ltM, Mgt then loop
• if H rejects ltM, Mgt then halt
• consider H on input H
• if it halts, then H rejects ltH, Hgt, which
  implies it cannot halt
• if it loops, then H accepts ltH, Hgt which
  implies it must halt
• contradiction.

10
Diagonalization
box (M, w) does M halt on w?
inputs
Y
Turing Machines
n
Y
The existence of H which tells us yes/no for each
box allows us to construct a TM H that cannot be
in the table.
n
n
Y
n
Y
n
Y
Y
n
n
Y
H
11
So far
decidable
all languages
RE
HALT

• Can we exhibit a natural language that is non-RE?
• on problem set

12
Reductions

• Given a new problem NEW, want to determine if it
  is easy or hard
• right now, easy means decidable
• right now, hard means undecidable
• One option
• prove from scratch that the problem is decidable,
  or
• prove from scratch that the problem is
  undecidable (dream up a diag. argument)

13
Reductions

• A better option
• to prove NEW is decidable, show how to transform
  it into a known decidable problem OLD so that
  solution to OLD can be used to solve NEW.
• to prove NEW is undecidable, show how to
  transform a known undecidable problem OLD into
  NEW so that solution to NEW can be used to solve
  OLD.
• called a reduction

14
Definition of reduction

• More refined notion of reduction
• many-one reduction

A
B
f
yes
yes
reduction from language A to language B
f
no
no
15
Definition of reduction
A
B
f

• YES maps
• to YES
• NO maps
• to NO
• function f should be computable
• Definition f S? S is computable if there
  exists a TM Mf such that on every w?S Mf halts
  on w with f(w) written on its tape.

yes
yes
f
no
no
16
Definition of reduction

• Notation A many-one reduces to B is written
• A m B
• Meaning B is at least as hard as A
• more accurate B at least as expressive as A

17
Using reductions

• Definition A m B if there is a computable
  function f such that for all w
• w ? A ? f(w) ? B
• Theorem if A m B and B is decidable then A is
  decidable
• Proof
• decider for A on input w, compute f(w), run
  decider for B, do whatever it does.

18
Using reductions

• Main use given language NEW, prove it is
  undecidable by showing OLD m NEW, where OLD
  known to be undecidable
• proof by contradiction
• if NEW decidable, then OLD decidable
• OLD undecidable. Contradiction.
• common to reduce in wrong direction.
• review this argument to check yourself.

19
Many-one reduction example

• Consider the language
• NONEMPTY ltMgt L(M) ? Ø

• f(ltM, wgt) ltMgt
• where M is TM that
• on input x, if x ? w, then reject
• else simulate M on x, and accept if M halts
• f clearly computable

f
yes
yes
f
no
no
NONEMPTY
HALT
20
Many-one reduction example
f

• f(ltM, wgt) ltMgt
• where M is TM that
• on input x, if x ? w, then reject
• else simulate M on x, and accept if M halts

yes
yes
f
no
no
NONEMPTY
HALT

• yes maps to yes?
• if ltM, wgt ? HALT then f(ltM, wgt) ? NONEMPTY
• no maps to no?
• if ltM, wgt ? HALT then f(ltM, wgt) ? NONEMPTY

21
Undecidable problems

• Theorem The language
• REGULAR ltMgt M is a TM and L(M) is a regular
  language
• is undecidable.
• a regular language is set of strings described by
  an expression built from a finite alphabet,
  concatenation, union, and
• fact 0,1 is regular
• fact 0n1n is not regular

22
Many-one reduction example
f
yes
yes
f
no
no
REGULAR
HALT

• HALT ltM, wgt TM M halts on input x
• REGULAR ltMgt M is a TM and L(M) is a regular
  language
• what should f(ltM, wgt) produce?

23
Many-one reduction example

• Proof
• f(ltM, wgt) ltMgt described below

• is f computable?
• YES maps to YES?
• ltM, wgt ? HALT ? f(M, w) ? REGULAR
• NO maps to NO?
• ltM, wgt ? HALT ? f(M, w) ? REGULAR

• on input x
• if x has form 0n1n, accept
• else simulate M on w and accept if M halts

24
Rices Theorem

• We have seen that the following properties of
  TMs are undecidable
• TM halts
• TM accepts a nonempty language
• TM accepts a regular language
• How widespread is undecidability phenomenon?

25
Rices Theorem

• Rices Theorem Every nontrivial TM property is
  undecidable.
• A TM property is a language P for which
• if L(M1) L(M2) then ltM1gt ? P iff ltM2gt ? P
• TM property P is nontrivial if
• there exists a TM M1 for which ltM1gt ? P, and
• there exists a TM M2 for which ltM2gt ? P.

26
Rices Theorem

• The setup
• let TØ be a TM for which L(TØ) Ø
• assume ltTØgt ? P
• technicality if ltTØgt ? P then work with property
  complement-of-P instead of P
• non-triviality ensures existence of TM M1 such
  that ltM1gt ? P

27
Rices Theorem

• Proof (know ltTØgt ? P and ltM1gt ? P)
• reduce from HALT (i.e. show HALT m P)
• what should f(ltM, wgt) produce?
• f(ltM, wgt) ltMgt described below

• f computable?
• YES maps to YES?
• ltM, wgt ? HALT ? L(f(M, w)) L(M1) ? f(ltM, wgt) ?
  P

• on input x,
• accept iff M halts on w and M1 accepts x

28
Rices Theorem

• Proof
• reduce from HALT (i.e. show HALT m P)
• what should f(ltM, wgt) produce?
• f(ltM, wgt) ltMgt described below

• NO maps to NO?
• ltM, wgt ? HALT ? L(f(M, w)) L(TØ) ? f(M, w) ? P

• on input x,
• accept iff M halts on w and M1 accepts x

29
Computability summary

• Main message
• some problems have no algorithms
• proof by diagonalization
• can use reductions from a known undecidable
  problem to a new problem to prove undecidability
  of the new problem
• undecidability a widespread phenomenon

30
Complexity

• So far we have classified problems by whether
  they have an algorithm at all.
• In real world, we have limited resources with
  which to run an algorithm
• time
• storage space
• need to further classify decidable problems
  according to resources they require

31
Complexity

• Complexity Theory study of what is
  computationally feasible (or tractable) with
  limited resources
• running time
• storage space
• number of random bits
• degree of parallelism
• rounds of interaction
• others

32
Worst-case analysis

• Always measure resource (e.g. running time) in
  the following way
• as a function of the input length
• function value is the maximum quantity of
  resource used over all inputs of given length
• called worst-case analysis

33
Time complexity

• Definition the running time (time complexity)
  of a TM M is a function
• fN ? N
• where f(n) is the maximum number of steps M uses
  on any input of length n.
• M runs in time f(n), M is a f(n) time TM

34
Time complexity

• We care about the behavior on large inputs. Why?
• general-purpose algorithm should be scalable
• overhead (e.g. for initialization) shouldnt
  matter in big picture

35
Time complexity

• Measure time complexity using asymptotic notation
  (big-oh notation)
• disregard lower-order terms in running time
• disregard coefficient on highest order term
• example
• f(n) 6n3 2n2 100n 102781
• f(n) is order n3
• write f(n) O(n3)

36
Time complexity

• Definition TIME(t(n)) L there exists a TM M
  that decides L in time O(t(n))
• Definition P or polynomial-time is
• P ?k 1 TIME(nk)
• Definition EXP or exponential-time is
• EXP ?k 1 TIME(2nk)

37
Time complexity

• interested in a course classification of
  problems. For this purpose,
• treat any polynomial running time as efficient
• problems in P are tractable
• treat any exponential running time as inefficient
  
• problems require exponential time are
  intractable

38
Time complexity

• Why polynomial-time?
• insensitive to particular deterministic model of
  computation chosen
• closed under modular composition
• empirically qualitative breakthrough to achieve
  polynomial running time is followed by
  quantitative improvements from impractical (e.g.
  n100) to practical (e.g. n3 or n2)

39
A puzzle

• Find an efficient algorithm to solve the
  following problem
• Input sequence of pairs of symbols
• e.g. (A, b), (E, D), (d, C), (B, a)
• Goal determine if it is possible to circle at
  least one symbol in each pair without circling
  upper and lower case of same symbol.

40
A puzzle

• Find an efficient algorithm to solve the
  following problem.
• Input sequence of pairs of symbols
• e.g. (A, b), (E, D), (d, C), (b, a)
• Goal determine if it is possible to circle at
  least one symbol in each pair without circling
  upper and lower case of same symbol.

41
2SAT

• This is a disguised version of the language
• 2SAT formulas in Conjunctive Normal Form with
  2 literals per clause for which there exists a
  satisfying truth assignment
• CNF AND of ORs
• (A, b), (E, D), (d, C), (b, a)
• (x1 ? ?x2)?(x5 ? x4)?(?x4 ? x3)?(?x2 ? ?x1)
• satisfying truth assignment assignment of
  TRUE/FALSE to each variable so that whole formula
  is TRUE

42
Algorithm for 2SAT

• Build a graph with separate nodes for each
  literal.
• add directed edge (x, y) iff formula includes
  clause (?x ? y) (equiv. to x ? y)

x4
?x4
?x3
?x1
x2
?x5
x3
x1
?x2
x5
e.g. (x1 ? ?x2)?(x5 ? x4)?(?x4 ? x3)?(?x2 ? ?x1)
43
Algorithm for 2SAT

• Claim formula is unsatisfiable iff there is some
  variable x with a path from x to ?x and a path
  from ?x to x in derived graph.
• Proof (?)
• edges represent implication ?. By transitivity of
  ?, a path from x to ?x means x ? ?x, and a path
  from ?x to x means ?x ? x.

44
Algorithm for 2SAT

• Proof (?)
• to construct a satisfying assign. (if no x with a
  path from x to ?x and a path from ?x to x)
• pick unassigned literal x with no path from x to
  ?x
• assign it TRUE, as well as all nodes reachable
  from it assign negations of these literals FALSE
• well-defined path from x to y and x to ?y
  implies path from ?y to ?x and y to ?x, implies
  path from x to ?x
• consistent path x to y (assigned FALSE) implies
  path from ?y (assigned TRUE) to ?x, so x already
  assigned at that point

45
Algorithm for 2SAT

• Algorithm
• build derived graph
• for every pair x, ?x check if there is a path
  from x to ?x and from ?x to x in the graph
• Running time of algorithm (input length n)
• O(n) to build graph
• O(n) to perform each check
• O(n) checks
• running time O(n2). 2SAT ? P.

46
Another puzzle

• Find an efficient algorithm to solve the
  following problem.
• Input sequence of triples of symbols
• e.g. (A, b, C), (E, D, b), (d, A, C), (c, b, a)
• Goal determine if it is possible to circle at
  least one symbol in each pair without circling
  upper and lower case of same symbol.

47
3SAT

• This is a disguised version of the language
• 3SAT formulas in Conjunctive Normal Form with
  3 literals per clause for which there exists a
  satisfying truth assignment
• dont know if this problem is in P
• much more on this later
• for now, observe that it is in TIME(2n)

48
decidable
all languages

• 3SAT ? EXP open whether it is in P.
• 2SAT ? P.
• Can we at least prove that P is different from
  EXP?

RE
P
EXP
49
Time Hierarchy Theorem

• Theorem For every proper complexity function
  f(n) n
• TIME(f(n)) ? TIME(f(2n)3).
• Note P ? TIME(2n) ? TIME(2(2n)3) ? EXP
• Most natural functions (and 2n in particular) are
  proper complexity functions.

50
Time Hierarchy Theorem

• Theorem For every proper complexity function
  f(n) n
• TIME(f(n)) ? TIME(f(2n)3).
• Proof idea
• use diagonalization to construct a language that
  is not in TIME(f(n)).
• constructed language comes with a TM that decides
  it and runs in time f(2n)3.

51
Recall The Halting Problem
box (M, x) does M halt on x?
inputs
Y
Turing Machines
n
Y
The existence of H which tells us yes/no for each
box allows us to construct a TM H that cannot be
in the table.
n
n
Y
n
Y
n
Y
Y
n
n
Y
H
52
Proof of Time Hierarchy Theorem
box (M, x) does M accept x in time f(n)?
inputs
Y
Turing Machines
n

• rows include all of TIME(f(n))
• TM SIM tells us yes/no for each box in time g(n)
• construct TM D running in time g(2n) that is not
  in table

Y
n
n
Y
n
Y
n
Y
Y
n
n
Y
D
53
Proof of Time Hierarchy Theorem

• Proof
• SIM is TM deciding language
• ltM, xgt M accepts x in f(x) steps
• Claim SIM runs in time g(n) f(n)3.
• define new TM D on input ltMgt
• if SIM accepts ltM, Mgt, reject
• if SIM rejects ltM, Mgt, accept
• D runs in time g(2n)

54
Proof of Time Hierarchy Theorem

• SIM decides ltM, xgt M accepts x in f(x)
  steps
• TM D on input ltMgt
• if SIM accepts ltM, Mgt, reject
• if SIM rejects ltM, Mgt, accept
• suppose M in TIME(f(n)) decides L(D)
• M(ltMgt) SIM(ltM, Mgt) ? D(ltMgt)
• but M(ltMgt) D(ltMgt)
• contradiction.

55
Complexity summary so far

• We have defined the complexity classes P
  (polynomial time), EXP (exponential time)

some language
decidable
all languages
RE
P
HALT
EXP