Algorithmic Problems

1
Algorithmic Problems

• Zeph Grunschlag

2
Agenda

• Viewing algorithms as languages
• Algorithmic Decision Problems
• ATM, ACFG , AREX, ETM, etc.

3
Knowing the Unknowable

• Arrived at important juncture in course. Now we
  will study one of the fundamental questions in
  Computer Science
• What is the ultimate power of computational
  machinery?
• Of equal importance in human quest to understand
  self
• What are the impenetrable limits of human
  knowledge?

4
Knowing the Unknowable

• Accepting the Church-Turing thesis implies that
  these are the same question! Church-Turing
  thesis is usually recognized as applying to human
  computations, and therefore human thought
  processes.
• Amazing Fact It is possible to give precise
  examples of problems which are beyond the reach
  of mechanical computation as well as human
  intellect. We in fact know the unknowable.

5
Decision Problems

• A decision problem is a set of yes/no questions,
  each called an instance. For the decision
  problem to be interesting, the set of questions
  should be infinite.
• EG Does z equal the sum of x and y ? is a
  good example. A computer program that can give
  the correct solution for any possible x,y,z
  intuitively must have figured out how to add
  numbers.
• EG Does 8 equal the sum of 3 and 5? is a bad
  example. Could easily write program for this
  instance without knowing how to add!

6
Decision Problems

• Q How about the problem Does God Exist? Can
  a computer solve this problem?

7
Decision Problems

• A In the way defined above, the answer is yes!
  We give two computer programs, both of which
  accept the input Does God exist?
• One of these programs is correct! Therefore,
  that computer program has solved the problem of
  deciding if God exists!
• Q Any misgivings?

A(String question) return No!
B(String question) return Yes!
8
Decision Problems

• A Many possible misgivings. Heres one
• Solution to existence of God surely not what is
  meant by solving the problem. We expected the
  solution to be arrived at by careful
  consideration of physical evidence, or by
  theological arguments, or some other intricate
  process.
• CS viewpoint Any algorithm which works only on
  a finite set of inputs is considered meaningless,
  as could always program a look-up table which
  gave correct solutions without using any insight.

9
Decision Problems

• So to solve the deity existence problem would
  have to give a more general solution. EG
• Given A complete description of a physical
  universe and a complete description of a deity.
• Decide Does the deity necessarily exist for the
  given universe?
• Now there are infinitely many possible inputs, so
  cannot just use a look-up table and this is a
  meaningful problem, though hopeless

10
Decision Problems as Languages

• To study what it means for a decision problem to
  be solvable, need a method of standardizing such
  problems and making them easily understood by
  computers. Cant just come over to a computer
  and ask Does 2 2 4? and even if could,
  dealing with voice recognition, natural language
  processing, etc. takes us far afield to topics
  irrelevant to answering is the problem
  solvable?
• Solution represent each instance by a string
• Thus instance is the string Does 2 2 4? or
  more simply 224.
• Q Define solvability language theoretically?

11
Decision Problems as Languages

• A A decision problem P is solvable if its
  encoding by strings admits a computer program
  which inputs such encodings, always halts, and
  always outputs the correct answer. Furthermore,
  the set of encodings should be computable in its
  own right.
• Language theoretically Let S be the alphabet in
  which the encoded instances of P exist. Let E be
  the set of encoded instance and let L be the set
  corresponding to the YES instances.1 Then P is
  said to be decidable (or algorithmically solvable
  or computable) if L is decidable.

12
Decision ProblemsSensible Encodings

• Some common sense is needed The encoding scheme
  should be sensible but there is no way to define
  what sensible means.
• EG Is p a prime number?
• Encoding scheme which results in the positive
  instances L 2,3,5,7, is sensible.
• Encoding scheme which results in the positive
  instances L 1,4,6,8,9,10, is not!

13
Decision ProblemsSensible Encodings

• Sensible encoding of sensible decision problem
  would never result in an undecidable encoding set
  E but condition still needed so dont solve
  impossible problems unwittingly.
• EG Does a given Java program with no infinite
  loops print out Im not loopy! on some input?
• Natural encoding set E is over Unicode alphabet,
  and is just the set of Java programs with no
  infinite loops. But well see that E is itself
  undecidable so second part becomes meaningless.

14
Decision ProblemsBasic Problems

• Before tackling fundamental decision problems of
  CS, start introspectively by considering if there
  are algorithms for
• telling if a given string is accepted by a given
  language of a fixed type C such as DFAs, NFAs,
  PDAs etc. Acceptance Problem AC
• telling if a given language of type is C empty.
  Emptiness Problem EC
• telling if two given languages of the same type
  C are equal. Equivalence Problem EQC
• Each in turn, is encoded by a language.

15
Decision ProblemsEncoding Example

• Lets see how ADFA may be encoded.
• ADFA is the problem of deciding
• Given A DFA M and a string x.
• Decide Does M accept x ?
• The language L of interest is the language of
  encodings of pairs (M, x ) where M accepts x.
  Encoding is denoted using the angled braces
  notation . In practice, will never
  really work with encodings as too unwieldy.
  However, for once its worth to see how this
  might actually be carried out.

16
Decision ProblemsEncoding Example
a

• Consider the DFA
• Q How can you represent this by a string?

0,1
0,1
b
17
Decision ProblemsEncoding Example

• A DFAs are 5-tuples,
• so just write down the
• 5 tuple, spelling out all
• details. Therefore
• (a,b,0,1,(a,0,b),(a,1,b),(b,0,a),(b,1,a),a,
  a)

a
0,1
0,1
b
18
Decision ProblemsEncoding Example

• A DFAs are 5-tuples,
• so just write down the
• 5 tuple, spelling out all
• details. Therefore
• (a,b,0,1,(a,0,b),(a,1,b),(b,0,a),(b,1,a),a,
  a)
• Q

a
0,1
0,1
b
19
Decision ProblemsEncoding Example

• A DFAs are 5-tuples,
• so just write down the
• 5 tuple, spelling out all
• details. Therefore
• (a,b,0,1,(a,0,b),(a,1,b),(b,0,a),(b,1,a),a,
  a)
• S

a
0,1
0,1
b
20
Decision ProblemsEncoding Example

• A DFAs are 5-tuples,
• so just write down the
• 5 tuple, spelling out all
• details. Therefore
• (a,b,0,1,(a,0,b),(a,1,b),(b,0,a),(b,1,a),a,
  a)
• d

a
0,1
0,1
b
21
Decision ProblemsEncoding Example

• A DFAs are 5-tuples,
• so just write down the
• 5 tuple, spelling out all
• details. Therefore
• (a,b,0,1,(a,0,b),(a,1,b),(b,0,a),(b,1,a),a,
  a)
• q0

a
0,1
0,1
b
22
Decision ProblemsEncoding Example

• A DFAs are 5-tuples,
• so just write down the
• 5 tuple, spelling out all
• details. Therefore
• (a,b,0,1,(a,0,b),(a,1,b),(b,0,a),(b,1,a),a,
  a)
• F

a
0,1
0,1
b
23
Decision ProblemsEncoding Example

• A DFAs are 5-tuples,
• so just write down the
• 5 tuple, spelling out all
• details. Therefore
• (a,b,0,1,(a,0,b),(a,1,b),(b,0,a),(b,1,a),a,
  a)
• Encoding amounts to just adding the comma
  and the input string w.
• Q Whats the alphabet of encoding lang. E ?

a
0,1
0,1
b
24
Decision ProblemsEncoding Example

• A TOO BIG! The number of letters in alphabet
  and number of states is unbounded. Using
  different characters for each state/letter would
  require an infinite alphabet if want to be able
  to encode every possible DFA! Instead, should
  fix some alphabet (e.g. binary or ASCII) and
  refer to states/letters using numbers. Thus the
  state a is denoted by 1, since its the first
  state, the state b by 2 and so on

25
Decision ProblemsWhat You Really Do!

• After going to all this effort to define decision
  problems in terms of languages, Turing machines,
  encodings and so on, in practice, we use none of
  the above. The point was to argue that in
  principle this could be done, and remind
  ourselves that this ought to be achievable, when
  wondering whether a not purported pseudocode is
  really an algorithm.
• Sipsers approach tries to maintain the veneer of
  this approach. I find this to be artificial at
  times so will just work directly with the natural
  data structure inherited by the problem.

26
ADFA

• For example, the easiest way to view a DFA is as
  a labeled graph. Encoding the DFA by a string
  and trying to apply some graph theoretic
  algorithm to the representative string is overly
  burdensome. You basically have to recreate the
  whole graph on the TM tape, which in principle
  can be done, but in practice is silly, especially
  since the starting point was a graph!
• Q How do you solve ADFA ?

27
ADFA

• A Just follow the path labeled by the input
  string from the start state. Accept if end was
  in F. More precisely.
• DFAaccept(DFA M, String x1 x2 x3 xn )
• State q q0 //q0 defined by Ms 5-tuple
• for(i 1 to n)
• q d(q,xi) // d defined by Ms 5-tuple
• if(q ? F) // F defined by Ms 5-tuple
• return ACCEPT
• else
• return REJECT

28
ANFA

• Q How about ANFA?

29
ANFA

• A Just determinize first and use solution to
  ADFA.
• NFAaccept(NFA N, String x1 x2 x3 xn )
• DFA M determinize(N) //Subset Constrn
• return DFAaccept(M, x1 x2 x3 xn )
• Q1 Whats the running time?
• Q2 Whats not ideal about this algorithm?
• Q3 Whats the fix?

30
ANFA

• A1 O(2Nn) since determinizing might involve
  all the subsets of the states of N so is
  exponential, but once have determinized,
  simulating the DFA is O(n).
• Q2 The problem is the exponential running time.
• Q3 We already know the fix. Its just the game
  Determinize which keeps track of active state
  without explicitly constructing all the subsets.
  Running time of O(N 2 n) since each update of
  the active states may involve looking at O(N )
  active states each of which may activate O(N )
  other states.

31
ANFA

• NFAaccept2(NFA N, String x1 x2 x3 xn )
• StateSet S q0 // store S as bit string
• for(i 1 to n)
• S d(S,xi) // since NFA, d outputs a set
• if(S and F are not disjoint)
• return ACCEPT
• else
• return REJECT

32
EDFA

• Lets now focus on emptiness problem. We can use
  the pumping lemma to solve the emptiness problem
  Suppose we know what the pumping number p is for
  our DFA. Then if we found no accepted strings
  of length accepts no strings.
• Q Why is this true?

33
EDFA

• A Suppose language not empty. Since checked
  that all strings of length smallest accepted string s must be of length ? p.
  The string s is pumpable, so can be pumped down
  obtaining a shorter string s, which contradicts
  the minimality of s !
• This prompts the following algorithm

34
EDFA

• DFAempty( DFA M )
• integer p Q
• for (all strings x over S with length
• if( DFAaccepts(M,x) ACCEPT )
• return NONEMPTY
• //only got here if nothing was accepted
• return EMPTY
• Q Running time? Improvements?

35
EDFA

• A
• Running time O(SQ) exponential.
• Improvements Just see if any accept state can be
  reached from start state by performing a BFS or
  DFS as follows

36
EDFA

• DFAempty2( DFA M )
• State q q0
• Stack S // Initialize a LIFO stack
• Set V ? // Set of visited states
• S.push(q0)
• while(S ? ?)
• q S.pop()
• if (q ?F ) return NONEMPTY
• V V ? q
• for (States q satisfying q?q ) // an edge
• if (q ?V ) S.push(q )
• return EMPTY // Only got here if F unreachable
• Q Is this a BFS or DFS?

37
EDFA

• A This is a DFS algorithm since stack is used.
  Queue would give BFS.

38
ACFG

• Acceptance problem is also solvable. Theres a
  rather horrible algorithm involving Chomsky
  Normal Form. Later well see a much improved
  polynomial time algorithm (must have one,
  otherwise compilers would be useless!)
• Chomsky normal form gives the following fact
• LEMMA Suppose a grammar G is in Chomsky normal
  form. Let x be in L(G ), and let n be the length
  of x. Then x is generated by a derivation of
  length 2n-1, when n 0.

39
ACFG

• LEMMA Suppose a grammar G is in Chomsky normal
  form. Let x be in L(G ), and let n be the length
  of x. Then x is generated by a derivation of
  length 2n-1, when n 0.
• Proof. The first n-1 productions are of the form
  A ? BC and get us to the correct length. The
  last n production are unit terminating A ? a and
  derive x.
• This gives rise to the following algorithm

40
ACFG

• CFGaccept( CFG G, String xx1 x2 x3 xn )
• CFG G ChomskyNormalForm(G )
• for (all derivations from start variable
• of G of length ? 2n 1)
• if (derivation resulted in x)
• return ACCEPT
• return REJECT
• Q1 Why does this work for x e?
• Q2 Running time in terms of n ?

41
ACFG

• A1 Works for x e because of ? 2n1 clause.
• A2 Exponential. If we consider the blow-up of
  CNF conversion, this is a truly horrendous
  algorithm.

42
Blackboard Exercises

• EQDFA
• ECFG
• CARDDFA