We’ve arrived at Turing Machines, and a natural question might be: is there anywhere to go from here? Can we design a machine that is more powerful, i.e., able to accept an even larger class of languages? The Church-Turing Thesis answers this with a definitive no. The thesis itself goes something like this:
Any model of algorithmic computation is no more powerful than a Turing Machine.
Many such models exist - examples that all belong in the equivalence class with one-tape Turing machines include:
The implications of this are huge: for one, it means that your choice of programming language or model of computation does not affect computability. Anything you can do in Python, you can also compute in Racket or Assembly or with a Turing machine.
Why is this a thesis rather than a theorem? There are many theorems that support the thesis - one could prove a theorem showing that all Java programs can be converted to equivalent Turing Machines, and vice versa. But the Thesis is more wide-ranging: it says that for any computation approach you could come up with, such a theorem can be proven. It’s called a thesis because the notion of a “model of algorithmic computation” is not a formally defined concept - it can’t really be proven because the terms aren’t defined well enough. The thesis refers to our intuitive understanding of algorithmic models of computation; in some sense, the Thesis gives a definition for this: anything that is equivalent to a Turing machine.
Just because there’s no machine more powerful than a Turing machine doesn’t mean there’s nothing left to do when it comes to categorizing languages.
Last time we observed an important distinction when it comes to outcomes of a Turing machine. When a machine \(M\) is given an input \(w\),
This third possibility poses an interesting (and perhaps inconvenient) conundrum: there may exist strings that are not in a language \(L(M)\) (because, as we have defined it, it’s not in the language if the machine does not terminate), but \(M\) cannot definitively tell us so!
Notice that in this situation, it’s still the case that the machine \(M\) accepts this language, but \(M\) does not terminate on some inputs. If this is the case, we call the language \(L(M)\) not decidable (or alternatively, undecidable). Conversely, if \(M\) terminates on all possible inputs, then \(L(M)\) is called decidable. An alternative (synonymous) term for decidable is recursive.
Most of the languages we’ve seen so far are decidable. If a string is not of the form \(a^n b^n c^n\), we can find out using a finite amount of computation. However, there are a lot of undecidable languages (indeed, in a very mathematical sense, most languages are undecidable - see TC 5.6).
One type of language that helps clarify the boundary of decidability is languages of the form (yes, this is getting meta): \[ \{(M, w) : \text{ $M$ accepts $w$}\} \] for various types of automata \(M\). That is, can we build a Turing machine that takes as input a machine \(M\) and an input \(w\), and tells us whether the machine \(M\) accepts the input \(w\)?
In order to talk about this as a language acceptance problem, we need elements of the language to be strings; as written above, they’re not - they’re 2-tuples of (Tuple, String). To handle this, we will, basically sweep it under the rug by making the claim that: there exists some way of encoding a machine \(M\) into a finite-length string. We know this to be true because we write down representations of machines all the time. The precise detials of this representation are considered moot - for our purposes, it suffices to know that there is a way to do it; formally, there exists a bijective function mapping machines to strings. We’ll write a string representation of a machine \(M\) and input \(w\) as \(\langle M, w \rangle\).
In the following discussion, we’ll use the following (hopefully plausible) claim without proof: it is possible to design a Turing machine that takes as input \(\langle M, w \rangle\) and “simulates” the execution of machine \(M\) on input \(w\). This is true for any of the machines we’ve seen: a Turing machine can simulate a DFA, PDA, or a Turing Machine. Further, since the TM can go back and read the input multiple times, it can simulate nondeterminism by trying all possibilities in a nondeterministic machine.
Very briefly, let’s run through the types of machines we’ve seen and determine whether the language \(A_T = \{\langle M_T, w \rangle : M \text{ accepts } w\}\) is decidable for various machine types \(T\).
This is decidable because we can simply simulate the machine. A DFA will always terminate on a finite input, so the simulation will terminate.
This is decidable because we could try all possibilities; we could also encode the algorithm to convert an NFA to a DFA into our machine as a preprocessing phase, then run the DFA.
\(A_{PDA}\) is trickier because a PDA may not terminate, so a simulation of one may not terminate. However, because PDAs are equivalent to Context-Free Grammars, \(A_{PDA}\) must be decidable if and only if \(A_{CFG}\) is decidable - that is, given a string representation of a context-free grammar and a candidate string, is that string in the language of the grammar? Checking all possible derivations has the same problem as the PDA: it’s not clear how many substitution rules we need to apply, and “check all possibilities” could go on forever. However, if we convert the CFG to Chomsky Normal Form - which we know we can do - then we only need to check all derivations of length \(2n-1\) where \(n\) is the length of the input. This can be done in finite computation.
How about a Turing machine? Intuition suggests it’s not looking good - since \(M\) may not terminate on \(w\), there’s no point at which you can stop and say “no” as the simulation keeps running. This points back to the Halting Problem that we discussed way back when we introduced computability theory: if there were a way to tell whether \(M\) would eventually halt, then \(A_{TM}\) would be decidable. Since there isn’t (and we proved it) - \(A_{TM}\) is not decidable. To make a more formal argument, we’ll draw a more rigorous connection to the halting problem.
Recall from way back when we introduced computability that we discussed the Halting Problem. In that lecture, I phrased it in terms of “programs”: Can you write a program that tells whether any given terminates? Thanks to the Church-Turing Thesis, we can equivalently rephrase this as: Can you design a Turing Machine that tells whether a given Turing Machine terminates on a given input?
Here’s the argument we used to show that such a program (or machine) cannot exist:
An alternative - and more general - way to prove that a problem is not decidable is to reduce its solution to the solution of a different problem that’s known to be undecidable. This is a very common tool employed in both computability theory and complexity theory - you’ll see this again!
Let’s look at an example. We will give a more rigorous proof that \(A_{TM}\) is not decidable by reducing HALT to \(A_{TM}\).
The general idea here is to demonstrate that if we could solve \(A_{TM}\), then we could solve HALT. Since we’ve already demonstrated that HALT is not decidable, this is sufficient to prove that \(A_{TM}\) is not decidable.
Theorem: \(A_{TM}\) is not decidable.
Proof: Suppose \(A_{TM}\) is decidable; that is, there exists a Turing machine \(M_{TM}\) that takes as input a TM and a string \(\langle M, w \rangle\) and accepts if \(M\) accepts \(w\) and rejects if \(M\) does not accept \(w\); in particular, \(M_{TM}\) always terminates, even if \(M\) does not.
Construct a Turing machine \(M_H\) that takes a machine and string \(\langle M, x \rangle\) as input. \(M_H\) operates as follows:
Observe that \(M_H\) is a solution to HALT that always terminates. Since HALT is undecidable, no such solution can exist; this is a contradiction, so our assumption that \(A_{TM}\) is decidable must be incorrect. \(\blacksquare\)
The above proof reduced HALT to \(A_{TM}\) to show that the latter is not decidable. You can also apply reductions to show that something is decidable. Whereas above we said “if (thing in question) were decidable, we’d be able to solve (undecidable problem)”. We can use similar reasoning to say “we can solve (thing in question) as long as we’re able to solve (decidable problems)”. We did this, informally, when showing that \(A_{NFA}\) is decidable: we argued that we can use \(M_{DFA}\) to solve \(A_{NFA}\) by first converting the NFA to a DFA (a process that will always terminate) then running \(A_{DFA}\) (which we know to be decidable). In this case, we reduced \(A_{NFA}\) to \(A_{DFA}\).
Rice’s theorem says that determining any nontrivial property of a Turing machine is undecidable. Yikes!
Even among languages that are not decidable, there are further distinctions to be made! A language is enumerable if you can construct a Turing Machine (or computer program) to generate all strings in the language; obviously this won’t terminate, but for any specific string in the language it will be generated in finite time.