that never ceases day and night, throughout India.
RUDYARD KIPLING, Kim1
Governments have conducted espionage and intelligence-gathering efforts for centuries. Indeed, intelligence-gathering – often said to be the world’s second oldest profession – is as old as governments themselves. In Britain, there was a ‘secret service’ operating at least since the reign of Queen Elizabeth I in the sixteenth century, which under Sir Francis Walsingham was tasked to gather intelligence on the Spanish Armada and to uncover various Catholic intrigues and plots. However, it was not until the nineteenth century, and more importantly the early twentieth century, that the British government began to devote significant resources to intelligence, and turn it into a professional, bureaucratic enterprise. Despite Britain’s long history of clandestine espionage work, in fact it was not in the ‘domestic’ realm that its intelligence-gathering was to develop most rapidly. Instead, it was in the British empire, which in the course of the nineteenth and twentieth centuries grew to become the greatest empire in world history, that intelligence found a particularly important role.2
From the earliest days of the British intelligence community, which was established in the early twentieth century, there was a close connection between intelligence-gathering and empire. It is not an exaggeration to say that in its early years British intelligence was British imperial intelligence.This is not surprising when it is considered that intelligence played an essential role in the administration of the empire, which by the 1920s had grown to encompass one-quarter of the world’s territory and population. After 1918, as one geographer proudly commented, the empire reached its widest extent, covering ‘one continent, a hundred peninsulas, five hundred promontories, a thousand lakes, two thousand rivers, ten thousand islands’. The empire had four kinds of dependent territories: colonies, protectorates, protected states and trust territories. At one end of the spectrum, colonies were those territories, like Kenya, where the monarch of the United Kingdom had absolute sovereignty, while trust territories, at the other end of the spectrum, were those assigned to Great Britain for administration under a special mandate, like Palestine. There was often little practical difference between colonies and protectorates. The Colonial Office usually referred to territories under ‘traditional’ rulers, with a British resident, as ‘protected states’. The typology of these dependent states was incredibly confusing (sometimes even to the Colonial Office itself).
One reason for the importance of intelligence in the empire was the lack of sheer manpower required to cover such enormous territories. Even at its height, British rule in India was maintained through an incredibly small number of administrative officials, with the renowned Indian Civil Service in the Raj boasting a total of just 1,200 posts, at a time when the population of India was probably around 280 million. Before 1939 the Indian army of 200,000 men, together with a British garrison of 60,000, was responsible for keeping the peace on land from Egypt to Hong Kong – British territories ‘East of Suez’, to use the phrase from the time. With such meagre resources at its disposal, British rule in India required up-to-date and reliable information on its enemies, both imagined and real. This was acquired through networks of informants and agents, and from intercepted communications. It is little wonder that, as one study has termed it, the British empire in the nineteenth century was an ‘empire of information’.3
Intelligence-gathering also came to the forefront in Britain’s imperial military campaigns in the late nineteenth and early twentieth centuries. One of the most exhilarating theatres for intelligence operations, or spying, lay in India’s North-West Frontier – now the tribal borderlands of Pakistan – where Victorian Britain fought the ‘Great Game’ with Russia, a conflict memorably portrayed by Rudyard Kipling in Kim, arguably one of the greatest espionage novels of all time. In Kim, Kipling described the ‘Great Game’ as essentially an intelligence conflict, which ‘never ceases day or night’, with both Britain and Russia running spies and informants to discover the other’s intentions. However, the reality was that it was often not difficult for Russia to spot British imperial intelligence agents: they were often extremely amateurish and deployed flimsy covers, variously posing as butterfly collectors, archaeologists and ethnographers. That said, it was in the ‘Great Game’ that some distinctly more professional forms of intelligence-gathering were born, particularly in a process that would later become known as signals intelligence (SIGINT), the interception and decryption of communications, or ‘signals’. In 1844 the Indian army pioneered one of the first permanent code-breaking bureaus in the world, which gained notable successes in reading Russian communications long before any similar European SIGINT agency had done so. The British military also made innovative use of intelligence during its campaigns in Egypt in the 1880s, successfully deploying a series of agents and scouts to reconnoitre the location of Egyptian forces in the desert.4
The very process of Britain’s colonial expansion in the Victorian period, especially during the so-called ‘scramble for Africa’ beginning in the 1880s, necessitated new forms of systematic intelligence-gathering, such as mapping and census-taking. In undertaking such activities, Britain was not acting differently from its imperial rivals at the time – France, Russia, Germany and Italy. Before any colonial power could dominate, control and exploit colonial populations, in Africa or elsewhere, it first had to map them. In practice, however, the process of mapping an empire often ignored its realities. Maps imposed European geometrical patterns on amorphous landscapes, drawing frontiers that cut through tribal communities as well as ethnic and linguistic groups. To this day, it is not difficult to spot the borders of those countries, particularly in Africa, which were drawn by European cartographers: many are arranged at right angles and slice through geographical features and ethnographic groupings. Sometimes European powers displaced and resettled colonial populations in order to make them reflect the ethnographic colonial maps. In the ‘white man’s burden’ of colonial rule, subtle realities did not matter.5
Given all that, it is no coincidence that Britain’s first Directorate of Military Intelligence, established in 1887, grew from the Topographical and Statistical Department in the War Office, which was responsible for mapping much of the British empire. Moreover, it was a violent colonial ‘small war’ in an outpost of the British empire, the Second Anglo-Boer War in southern Africa, waged between 1899 and 1902, which first alerted the British government to the need for establishing a permanent intelligence service. The so-called Boer War exposed to Britain’s military leaders, the Chiefs of Staff in London, how fragile the nation’s colonial holdings were. It took the British military much longer than expected, three years, and also the deployment of some 45,000 troops, to defeat a group of rebellious Dutch Boer farmers in the Cape Colony (now South Africa) who harried the British Army through guerrilla warfare. In fighting the insurgency there, it has to be noted that the British military developed some ominous strategies, not least the establishment of ‘concentration camps’, or detention camps, where suspected insurgents were ‘concentrated’. This type of warfare, in which the distinction between combatants and non-combatant civilians was blurred, was to have horrific echoes in the twentieth century. As far as intelligence was concerned, the kind of irregular warfare that Britain faced in the Boer War, like that experienced by other European powers in their own colonial ‘small wars’ – literally guerrilla in Spanish – revealed the paramount need for effective intelligence-gathering. In fact, it was during the Boer War that a British officer, Lt. Col. David Henderson, wrote an influential paper for the War Office in London, ‘Field Intelligence: its principles and practice’, which became the basis of a manual, ‘Regulations for intelligence duties in the field’, published by the War Office in 1904. This manual became the inspiration for the British Army’s intelligence corps, founded ten years later, on the outbreak of the First World War.6
Despite Britain’s long history of intelligence-gathering, a watershed occurred in the early twentieth century. Partly in response to fears of Britain’s colonial frailty, as revealed by the Boer War, but more specifically as a result of fears about the growing threat posed by the German empire, in October 1909 the British government took the momentous decision to establish a permanent, peacetime intelligence department. This decision was taken by the Committee of Imperial Defence – significantly, it was imperial defence that led to the setting up of Britain’s spook agencies. The department, known