The Cipher Event – Recreating the Past
Take part and challenge Colossus!
The Cipher Event - Recreating the Past
On 15 and 16 November 2007 we shall celebrate the completion of the Colossus Mark 2 rebuild at Bletchley Park and recreate cryptographic history when an international team will:
1. Prepare three separate (secret) texts in German (the challenge plaintexts);
2. Encipher each of those plaintext messages using the Lorenz SZ42 cipher and prepare three paper tapes (the challenge ciphertexts);
3. Transmit each of those ciphertexts in turn using amateur radio operators in Germany;
4. Intercept the ciphertexts at a replica ëYí station in the UK;
5. Interpret the ciphertexts using an original undulator and transfer those interpreted ciphertexts on to paper tape;
6. Load the paper tape ciphertexts on the Colossus Mark 2 rebuild in Bletchley Park Block H. Run the Colossus to recover the Lorenz machine wheel settings used to encipher the plaintext;
7. Recover the secret plaintext messages;
8. Validate the result.
At the same time as the international team receives the enciphered messages, radio amateurs around the world will be able to receive the same radio broadcasts and try their hand at decrypting it. It will be fascinating to see who completes the job first!
Encipher the Secret Message
Representatives from Heinz Nixdorf MuseumForum in Paderborn, Germany under the guidance of Norbert Ryska will prepare the challenge plaintexts (in German).
Transmit the Ciphertext
Radio amateurs at the radio station DL0HNF located in the Heinz Nixdorf Museum Forum, Paderborn will transmit the challenge ciphertexts. Heinz-Peter Bleier will lead the team and plans to operate a special event station with a dedicated call sign specifically for the purpose.
Intercept the Ciphertext
The Paderborn transmissions will be intercepted by two teams at Bletchley Park ñ the first will be led by John Housego and comprise members of the Milton Keynes Amateur Radio Society operating amateur radio station callsign GB2BP. They will use current technology receivers and signal capture methods.
The second team will be led by John Pether who will use the same type of equipment as used in the ëYí (intercept) station in Knockholt in WWII. This equipment includes AR88 receivers with undulators connected for hardcopy output on strip tape.
Interpret the Ciphertext
John Pether's team will interpret the output of the undulators by hand and punch paper tape directly for loading on to the Colossus Mark 2.
Run the Colossus Mark 2
Tony Sale and his team will load the paper tape containing the ciphertext on to the Colossus and run it to find the Lorenz machine wheel settings.
Recover the Plaintext
Once the Colossus has reported the wheel settings, Tony and his team will use the Tunny simulator to recover the plaintext (the TUNNY rebuild is not yet complete).
Validate the Result
Once the plaintext has been recovered, members of the MKARS plan to email and transmit the plaintext back to Germany using RTTY for validation and scoring.
Cypher Challenge Winner
Read how it was done by the man who did it. Well done.
Joachim Schüth is a German amateur and a winner in an area where real competence was the only way to win. Here he tells us about it:-
Software for Code breaking of the Lorenz SZ42
On November 15/16 2007, the rebuild of Colossus at Bletchley Park was celebrated in a Cipher Event organized by Tony Sale, curator of the British National Museum of Computing, who also headed the Colossus rebuild project. Messages encrypted with a historic Lorenz SZ42 cipher machine were transmitted from the ham radio station at the Heinz Nixdorf Museum in Paderborn to receiving stations in in Great Britain. The rebuilt Colossus machine was to perform the code breaking tasks using the historic methods.
The announcement of the Cipher Event stated that "At the same time as the international team receives the enciphered messages, radio amateurs around the world will be able to receive the same radio broadcasts and try their hand at decrypting it. It will be fascinating to see who completes the job first!". As a radio amateur (call sign: DL2KCD) I was intreagued by this challenge. Hams have a culture of contesting. The outstanding work of the cryptographers at Bletchley Park was also important from nowadays point of view in Germany, as it helped to shorten the lifetime of the Nazi dictatorship. Therefore the Cipher Event deserves attention also from this country.
I became aware of the upcoming event in mid September 2007 and found that there would be enough time to prepare for it. I figured that standard teletype modems and software would not be of much use, because the cipher stream contains non-printable characters (control characters of the Baudot code) which might not be recorded properly, and also because the historical tone frequencies were going to be used. In addition, any loss of a character in the received cipher texts means that the following text is no longer in sync with the steps of the SZ42 cipher algorithm. Recording a wrong character from time to time does not prevent the code breaking, but loosing characters does. I figured that I would need a special reception software to analyse the audio data, with a robust method of clock recovery. To perform the subsequent cryptanalysis, I decided to also write my own software. The information on the site that commemorates the work of the great Alan Turing at www.alanturing.net provided excellent material on the SZ42 and its cryptographic weaknesses. After years of C programming and trying other languages (like C++, Java, Haskell, Python), I had started to learn Ada early in 2007. This powerful and beautiful language has become my favourite, and I decided to do the code development as a programming exercise in Ada.
During November 15, I could receive some of the Cipher Challenge messages and decrypt them after breaking the key (wheel settings). All source code used is provided below. The Ada sources were compiled with GCC/GNAT. The PC used was a laptop with 1.4 GHz CPU, using NetBSD as the operating system. I used the antenna system and the radio transceiver of the club station DL0OV in Bonn to receive the transmissions.
Putting Colossus in a competition with modern computers may be a bit unfair. Colossus was an ingenious construction and a landmark in the history of computing. But technology has very much evolved since: When fed with a usable cipher text, the quick_setting program provided below found the settings of all 12 wheels within 46 seconds. During the Cipher Event, I actually spent most of the time with the signal processing work (converting the noisy audio recording into the cipher text stream of Baudot symbols) rather than with the crypto tasks.
The software was written for my own use and is therefore not well documented, and also the user interface is tailored to a machinist looking under the hood; the programs basically form a chain of command line tools. But others might find the algorithms interesting too.
Just grab all the Ada source files from the table at the bottom of this page and compile the programs with a command like
for i in *.adb; do gnatmake -gnatwa -gnatwe -gnatv -Wuninitialized -Werror -O3 $i; done
and make sure the resulting executables are in your search path. A source tarball is provided at the bottom of this page.
Then, for each of the two transmissions provided below, put the three input files (A.mp3, process.sh, and key0.txt) in one directory. Then run the process.sh shell script. The complete toolchain is run until the plaintext is revealed. For downloading, you can use the tarballs provided at the bottom of this page.
To save space and network bandwidth, the audio data which were originally recorded as 16 bit signed linear with 8000 Hz sample rate are provided here as MP3 files. The scripts assume that you have sox installed to convert the data back to the original format.
Note that execution times may vary slightly because of the use of random numbers to modify the trial settings. Occasionally you may not get the correct result on the first run.
The plaintext contained two upshift characters before and two downshift characters after each space (represented as <<_>> in the plain.out files). This means that the same character was sent twice in sucession far more often than in a normal plaintext. This gives rise to pronounced statistical properties in the difference sequences that are considered during Chi setting when Motors and Psi are unknown. The plaintext was therefore rather code breaker-friendly. The up and down shifts around the spaces are not even necessary, as a space is a space in letter shift and in figure shift.
Transmission of Nov. 15, 2007, 12:00 UTC
This is the first transmission received strong enough in Bonn for decoding.
I mailed my original result to Tony Sale at 13:14 UTC, and to HNF at 13:42 UTC on Nov. 15, 2007.
Input files:
MP3 of 12:00 UTC recorded audio. Cipher text is transmitted using historic 6-tone signal. The cipher text transmission starts at 03:08 min after the beginning of the recording.
Shell script tailored to 16:00 UTC transmission.
Used to supply wheel patterns to quick_setting
Output files:
Cipher text file with some diagnostic data (first two lines omitted on input to quick_setting)
Output of quick_setting, the actual counterpart to the work of Colossus
New key file with settings found by quick_setting
The plaintext output
A trace file generated with cryptrace
Transmission of Nov. 15, 2007, 16:00 UTC
This transmission was received much stronger, and as far as I understand it was also the first that was received in usable quality by the Colossus team in Great Britain.
I mailed my original result to Tony Sale and HNF at 16:40 UTC on Nov 15, 2007.
Input files:
MP3 of 16:00 UTC recorded audio. Cipher text is transmitted with "modern" two-tone signal. Shift was reversed relative to plaintext header. The cipher text transmission starts at 03:24 min after the beginning of the recording.
Shell script tailored to 16:00 UTC transmission.
Used to supply wheel patterns to quick_setting
Output files:
Cipher text file with some diagnostic data (first line omitted on input to quick_setting)
Output of quick_setting, the actual counterpart to the work of Colossus
New key file with settings found by quick_setting
The plaintext output
A trace file generated with cryptrace
Transmission of Nov. 15, 2007, 17:00 UTC
Again a strong signal. There was apparently some problem at DL0HNF and the cipher text started with some delay. According to the transmission schedule, this was the first Challenge 3 (no wheel settings given) that I could receive. However, I always let the computer search for all 12 wheel settings, so this was no difference to me.
I mailed my original result to Tony Sale and HNF at 17:51 UTC on Nov 15, 2007.
Input files:
MP3 of 17:00 UTC recorded audio. Cipher text is transmitted with "modern" two-tone signal. Shift was reversed relative to plaintext header. The cipher text transmission starts at 12:56 min after the beginning of the recording.
Shell script tailored to 17:00 UTC transmission.
Used to supply wheel patterns to quick_setting
Output files:
Cipher text file with some diagnostic data (first line omitted on input to quick_setting)
Output of quick_setting, the actual counterpart to the work of Colossus
New key file with settings found by quick_setting
The plaintext output
A trace file generated with cryptrace
Source code:
Sound recording from PC soundcard
Record raw audio from soundcard. Works under NetBSD.
Demodulating and decoding (not decrypting) of audio data into Baudot symbol stream
Package specification related to Baudot code.
Package body related to Baudot code.
Demodulation of raw audio: Arguments are sample rate, a window width, and a list of tone frequencies. Output is a list of amplitude values for each tone.
Combines tone amplitudes by performing a weighted sum
Decodes demodulated signal according to Baudot code. The patterns of the Baudot symbols are convoluted with the signal. For each time step, the best matching symbol and its amplitude are listed.
Extract the symbols and construct the stream of Baudot symbols by evaluating the peaks in the amplitudes listed by the decoder. For solid clock recovery of noisy signals, a phase locked loop (PLL) is used.
Same as above, but without the PLL. Can be used for analysis if PLL locks on wrong symbol rate or phase.
Some R scripts that help in the analysis of the digitised audio signals. In particular, the audio tones can be found through Fourier analysis. A utility function for cutting a section out of a large audio file is also provided.
SZ42 crypto and code breaking routines
Package specification for SZ42 crypto routines
Package body, implements the algorithms of the SZ42 machine
Package specification for textual I/O of SZ42 data
Package body for textual I/O of SZ42 data
Package specification for handling of score charts (like top ten listings)
Package body for the above
Package specification for various cryptographic attacks on the SZ42
Package body for the above. Core of the crypto attacks.
Progeam that uses the crypto libraries for wheel setting (all wheels). Breaks Chi setting by successive optimisation of random trial settings, then does a brute force search for the best Motor settings, and then breaks Psi settings with random trials again. It does this by calling the procedures provided by package Break_Lib above.
Program that decrypts (or encrypts) text like the SZ42
Program that lists the inner state of the SZ42 for all steps of a decryption
Tarballs for download:
Ada sources for complete toolchain:
SZ42_src.tgz
Input files for the three example runs shown above (A.mp3, process.sh, and
key0.txt for each):
1200UTC.tar
1600UTC.tar
1700UTC.tar
Joachim Schüth (or Schueth if you have no umlauts)