of generating thousands of sheets of truly random numbers should not be underestimated, and no one is clear how the Soviets made them. One individual has recalled a room full of women simply shouting out any number that came into their heads, but this seems improbable. Others have described devices not unlike lottery machines, with numbered balls. Whatever system was used, the logistical difficulties of generating many thousands of one-time pads and distributing them proved too much for wartime Russia.7 Some time in early 1942, with Moscow on the verge of evacuation and much of Soviet industry badly dislocated, operators began to run out of pads. The KGB department that printed them committed the fatal error of reprinting twenty-five thousand pages. This made a small proportion of the messages, which should have been unbreakable, vulnerable to cryptanalysis. Far worse, they were sent to KGB units as well as to military and diplomatic users.8
The Venona project that exploited this mistake began in Washington. The Americans had collected Soviet messages during the war, but they lacked time to work on them. On 1 February 1943 the US Army’s code-breaking service, called the Signals Intelligence Service, began a modest effort to see if it could exploit Soviet diplomatic communications. The telegrams had been collected at Arlington Hall, in Virginia, a former girls’ school which was commandeered by the Army as its main code-breaking centre. Interest increased dramatically when it was discovered that some of the streams of traffic related to espionage. In October 1943 a young code-breaker, Lieutenant Richard Hallock, a Signal Corps reserve officer who had been a peacetime archaeologist at the University of Chicago, was looking at Soviet commercial traffic when he realised that the Soviets had committed a terrible error and were reusing their pads. This was an astonishing discovery, and thereafter Venona slowly began to unravel some of the KGB’s most precious secrets.9
The US Army’s head of signals intelligence, Carter W. Clarke, was the main enthusiast for Venona. Clarke was a tough, impatient, hard-drinking individual who many regarded as uncouth, but he was also a lateral thinker. Like many military intelligence chiefs in both Britain and the United States, he nurtured a deep-seated distrust of the Soviets, asserting bluntly: ‘They’re your friends today and they’re your enemies tomorrow, and when they’re on your side find out as much as you can about them because you can’t when they become your enemy.’10 The US Navy code-breakers also began work on Soviet traffic in the summer of 1943. The fact that by the autumn of 1944 the two rival armed services were both referring to all Soviet radio intercepts by the same code name of ‘Rattan’ suggests a directive from a high level. The following year the code name was changed to ‘Bourbon’.11
By 1944, another talented young American code-breaker, Meredith Gardner, was busy making the first breaks into KGB traffic and even some from Soviet military intelligence (GRU). Other code-breakers were now drafted in to help. One of them was Cecil Phillips, a chemistry student who was sent to Arlington Hall in June 1943, initially to work on Japanese naval messages. In May 1944 he was switched to Soviet diplomatic traffic. He quickly realised the scale of duplication, and made a number of progressions that led to wider breaks in the cypher system used by the KGB.
However, substantial activity had to await the end of the war with Japan, when larger numbers of staff could be transferred to work on ‘the Russian problem’.12 Some of the Soviet messages were double-encrypted, and so represented a fantastic level of difficulty. Nevertheless, on 20 December 1946 Gardner decrypted a KGB message listing the names of scientists who had been working on the wartime development of the atomic bomb at Los Alamos, known as the ‘Manhattan Project’. In spring 1947 he decyphered a message that showed that the Soviets were being given highly classified material from inside the US War Department.13 KGB agents were rarely referred to by their real names in the messages. The British spy Donald Maclean, for example, was ‘Homer’ or ‘Gomer’. Accordingly, their identities had to be figured out from their activities and from what material they were providing to the Soviets.14
Early accounts of Venona suggested that the first breaks were achieved as a result of the recovery of a partly burned Soviet codebook found in Finland and sold to America’s wartime intelligence agency, the Office of Strategic Services. Stories have long circulated about how American diplomats insisted that protocol required that it be returned to the Soviets. In fact, up until 1952, the progress made on Venona was probably driven by the pure sweat of mathematics, and represented a remarkable intellectual achievement. A little help was gained by intercepting Japanese traffic that contained Soviet material purchased from the Finns in 1944. The Finns had not been reading high-grade traffic, but had learned enough to be able to sort messages into homogeneous groups, the first stage of a cryptanalytical attack.15 It was only in 1953 that the American team realised that one of the KGB systems it was working on related to a Soviet codebook that had been in their possession since 1945. At the end of the war TICOM Team 6, led by Lieutenant Colonel Paul Neff, had seized a copy of a partially burned Soviet codebook while exploring the German sigint centre at Burgscheidungen. The Germans had themselves seized the codebook from the Soviet Consulate in Petsamo in Finland during June 1941.16
The big shock was revelations about espionage within the Manhattan Project. This immediately raised the question of how the material might be employed for counter-espionage. Liaison was established with Robert Lamphere of the FBI’s Intelligence Division, which had responsibility for maintaining physical surveillance on Soviet espionage activities. Venona was of immense help to the FBI, but it was not a one-way street. Occasionally the Bureau undertook burglaries of Soviet premises and photographed Soviet documents. Over the next decade, attempts were made to match material from these ‘black bag jobs’ with Venona material, but sadly there were few connections. Nevertheless, Lamphere ensured a coordinated exploitation system with the code-breakers.17
Meredith Gardner recalls that tight security for Venona only crept in slowly. In the beginning, everyone in the branch where it was being worked on was potentially privy to it, and ‘no special treatment was given’. This was partly because cryptanalysts had to support each other by discussing problems, since systems were often related to each other. There were people who genuinely needed to know, and there were also ‘mere busy-bodies who perhaps considered themselves consultants at large for all’. The Army intelligence liaison man, Howard Barkley, heard that ‘there was something interesting going on’ and came for a look, even though he had not been formally indoctrinated. Knowledge of Venona ‘might have been picked up almost anywhere’ in the branch at Arlington.18
Yet Venona was ‘so sensational’ that eventually something unusual had to be done on the security front. The focus was less on restricting the knowledge that it existed than on tightly controlling the contents of the messages. However, counter-intelligence is a messy business. What the US Army code-breakers needed in order to identify the spies was background material from other government departments – so they were forced to work closely with a gradually expanding circle of people scattered across Washington. Typically, seven copies of one Venona message, issued on 30 August 1947 and entitled ‘Cover Names in Diplomatic Traffic’, were circulated. One went to GCHQ through its liaison, Colonel Patrick Marr-Johnson. The US Army code-breakers noted that the British surrounded the material with ‘rigid safeguards’. Two copies went to the heads of Army and Navy code-breaking. Four went to mainstream Army Intelligence, Naval Intelligence and FBI. The State Department was also an important collaborator. Given that informal secondary briefing must have taken place, this means that perhaps as many as thirty people may have been given information from one circulated Venona message.19 By contrast, an understanding on Venona was only reached with the CIA in September 1948,