later the Australians, the Canadians, the Dutch and even the ‘neutral’ Swedes. It is justly famous for revealing some of the ‘giants’ of Russian espionage, including Klaus Fuchs and Donald Maclean, but the vast pool of messages that remain unsolved is also significant. Even now, it points unambiguously to many other cases yet to be resolved.
Anxiety about the compromise of sigint secrets was always central to the code-breaking profession. Back in 1927, Prime Minister Stanley Baldwin’s infamous exposure of the reading of Soviet high-grade systems in the House of Commons had taught a whole generation of interwar code-breakers the price of careless talk. Thereafter, anxiety about the Ultra secret persuaded more than ten thousand people to keep their wartime vow of silence for decades. However, Venona introduced an even greater level of paranoia, since it hinted at the possibility of hundreds of Soviet agents active inside the governments of the West, some in high positions. For this reason it is unlikely that Venona was ever made known to President Roosevelt, and it was three years before his successor, Harry Truman, was let into the secret. Clement Attlee, Britain’s first post-war leader, was not told until a major security case made it unavoidable in late 1947.[3] Nevertheless, the Venona project was compromised by several Soviet agents within five years of its initiation. This did not entirely negate its value, since the Soviets could not prevent the West from continuing work on the immense volume of KGB messages that had already been collected during the 1940s, patiently revealing the names of important agents. In the late 1950s, for example, GCHQ suddenly began to have success with Soviet Naval Intelligence messages, having used a new analytic technique.[4]
The extreme secrecy of the Venona project was its Achilles heel. Although the material often pointed to the identity of Soviet spies in the West, for security reasons it could not be shown to those arrested to persuade them to confess; still less could it be produced in court. Any sensible defence lawyer would seek to probe the nature of Venona, not only exposing its fragmentary nature, but also revealing sensitive secrets about sigint. Therefore, once spies had been identified by Venona, they had to be either caught red-handed meeting with their KGB controller, or successfully interrogated and broken. The result was a game of cat and mouse in which the mouse sometimes got away. In 1951, Donald Maclean, Guy Burgess and John Cairncross were among those who escaped by a whisker. Remarkably, Theodore Hall, an American Communist spy within the atomic programme, also brassed it out, despite close interrogation, escaping what would almost certainly have been death in the electric chair.[5] By contrast, in 1950 the atom spy Klaus Fuchs succumbed to repeated and patient questioning by MI5 after his arrest. He told his interrogators that he ‘supposed he would be shot’, and was pleasantly surprised when he wasn’t.[6]
Venona revealed the security-minded nature of the Soviets. Much of their traffic was encrypted using a one-time pad system. This was time-consuming and slow, but they were willing to put in vast effort to protect their communications. This required huge volumes of tear-off pads with sheet after sheet of random numbers. The difficulty of generating thousands of sheets of truly random numbers should not be underestimated, and no one is clear how the Soviets made them. One individual has recalled a room full of women simply shouting out any number that came into their heads, but this seems improbable. Others have described devices not unlike lottery machines, with numbered balls. Whatever system was used, the logistical difficulties of generating many thousands of one-time pads and distributing them proved too much for wartime Russia.[7] Some time in early 1942, with Moscow on the verge of evacuation and much of Soviet industry badly dislocated, operators began to run out of pads. The KGB department that printed them committed the fatal error of reprinting twenty-five thousand pages. This made a small proportion of the messages, which should have been unbreakable, vulnerable to cryptanalysis. Far worse, they were sent to KGB units as well as to military and diplomatic users.[8]
The Venona project that exploited this mistake began in Washington. The Americans had collected Soviet messages during the war, but they lacked time to work on them. On 1 February 1943 the US Army’s code-breaking service, called the Signals Intelligence Service, began a modest effort to see if it could exploit Soviet diplomatic communications. The telegrams had been collected at Arlington Hall, in Virginia, a former girls’ school which was commandeered by the Army as its main code-breaking centre. Interest increased dramatically when it was discovered that some of the streams of traffic related to espionage. In October 1943 a young code-breaker, Lieutenant Richard Hallock, a Signal Corps reserve officer who had been a peacetime archaeologist at the University of Chicago, was looking at Soviet commercial traffic when he realised that the Soviets had committed a terrible error and were reusing their pads. This was an astonishing discovery, and thereafter Venona slowly began to unravel some of the KGB’s most precious secrets.[9]
The US Army’s head of signals intelligence, Carter W. Clarke, was the main enthusiast for Venona. Clarke was a tough, impatient, hard-drinking individual who many regarded as uncouth, but he was also a lateral thinker. Like many military intelligence chiefs in both Britain and the United States, he nurtured a deep-seated distrust of the Soviets, asserting bluntly: ‘They’re your friends today and they’re your enemies tomorrow, and when they’re on your side find out as much as you can about them because you can’t when they become your enemy.’[10] The US Navy code-breakers also began work on Soviet traffic in the summer of 1943. The fact that by the autumn of 1944 the two rival armed services were both referring to all Soviet radio intercepts by the same code name of ‘Rattan’ suggests a directive from a high level. The following year the code name was changed to ‘Bourbon’.[11]
By 1944, another talented young American code-breaker, Meredith Gardner, was busy making the first breaks into KGB traffic and even some from Soviet military intelligence (GRU). Other code-breakers were now drafted in to help. One of them was Cecil Phillips, a chemistry student who was sent to Arlington Hall in June 1943, initially to work on Japanese naval messages. In May 1944 he was switched to Soviet diplomatic traffic. He quickly realised the scale of duplication, and made a number of progressions that led to wider breaks in the cypher system used by the KGB.
However, substantial activity had to await the end of the war with Japan, when larger numbers of staff could be transferred to work on ‘the Russian problem’.[12] Some of the Soviet messages were double-encrypted, and so represented a fantastic level of difficulty. Nevertheless, on 20 December 1946 Gardner decrypted a KGB message listing the names of scientists who had been working on the wartime development of the atomic bomb at Los Alamos, known as the ‘Manhattan Project’. In spring 1947 he decyphered a message that showed that the Soviets were being given highly classified material from inside the US War Department.[13] KGB agents were rarely referred to by their real names in the messages. The British spy Donald Maclean, for example, was ‘Homer’ or ‘Gomer’. Accordingly, their identities had to be figured out from their activities and from what material they were providing to the Soviets.[14]
Early accounts of Venona suggested that the first breaks were achieved as a result of the recovery of a partly burned Soviet codebook found in Finland and sold to America’s wartime intelligence agency, the Office of Strategic Services. Stories have long circulated about how American diplomats insisted that protocol required that it be returned to the Soviets. In fact, up until 1952, the progress made on Venona was probably driven by the pure sweat of mathematics, and represented a remarkable intellectual achievement. A little help was gained by intercepting Japanese traffic that contained Soviet material purchased from the Finns in 1944. The Finns had not been reading high-grade traffic, but had learned enough to be able to sort messages into homogeneous groups, the first stage of a cryptanalytical attack.[15] It was only in 1953 that the American team realised that one of the KGB systems it was working on related to a Soviet codebook that had been in their possession since 1945. At the end of the war TICOM Team 6, led by Lieutenant Colonel Paul Neff, had seized a copy of a partially burned Soviet codebook while exploring the German sigint centre at Burgscheidungen. The Germans had themselves seized the codebook from the Soviet Consulate in Petsamo in Finland during June 1941.[16]
The big shock was revelations about espionage within the Manhattan Project. This immediately raised the question of how the material might be employed for counter-espionage. Liaison was established with Robert Lamphere of the FBI’s Intelligence Division, which had responsibility for maintaining physical surveillance on Soviet espionage activities. Venona was of immense help to the FBI, but it was not a one-way