recommendations, micro-targeted ads, search results, risk predictions, etc. – controls and influences citizens, workers and consumers. Many critical scholars have argued that the widespread delegation of human choices to opaque algorithms results in a limitation of human freedom and agency (e.g. Pasquale 2015; Mackenzie 2006; Ananny 2016; Beer 2013a, 2017; Ziewitz 2016; Just and Latzer 2017). Building on the work of Lash (2007) and Thrift (2005), the sociologist David Beer (2009) suggested that online algorithms not only mediate but also ‘constitute’ reality, becoming a sort of ‘technological unconscious’, an invisible force orienting Internet users’ everyday lives. Other contributions have similarly portrayed algorithms as powerful ‘engines of order’ (Rieder 2020), such as Taina Bucher’s research on how Facebook ‘programmes’ social life (2012a, 2018). Scholars have examined the effects of algorithmic ‘governance’ (Ziewitz 2016) in a number of research contexts, by investigating computational forms of racial discrimination (Noble 2018; Benjamin 2019), policy algorithms and predictive risk models (Eubanks 2018; Christin 2020), as well as ‘filter bubbles’ on social media (Pariser 2011; see also Bruns 2019). The political, ethical and legal implications of algorithmic power have been discussed from multiple disciplinary angles, and with a varying degree of techno-pessimism (see for instance Beer 2017; Floridi et al. 2018; Ananny 2016; Crawford et al. 2019; Campolo and Crawford 2020).
Given the broad critical literature on algorithms, AI and their applications – which goes well beyond the references mentioned above (see Gillespie and Seaver 2016) – one might ask why an all-encompassing sociological framework for researching intelligent machines and their social implications should be needed. My answer builds on a couple of questions which remain open, and on the understudied feedback loops lying behind them.
Open questions and feedback loops
The notion of ‘feedback loop’ is widely used in biology, engineering and, increasingly, in popular culture: if the outputs of a technical system are routed back as inputs, the system ‘feeds back’ into itself. Norbert Wiener – the founder of cybernetics – defines feedback as ‘the property of being able to adjust future conduct by past performance’ (1989: 33). According to Wiener, feedback mechanisms based on the measurement of performance make learning possible, both in the animal world and in the technical world of machines – even when these are as simple as an elevator (1989: 24). This intuition turned out to be crucial for the subsequent development of machine learning research. However, how feedback processes work in socio-cultural contexts is less clear, especially when these involve both humans and autonomous machines. While mid-twentieth-century cyberneticians like Wiener saw the feedback loop essentially as a mechanism of control producing stability within complex systems, they ‘did not quite foresee its capacity to generate emergent behaviours’ (Amoore 2019: 11). In the words of the literary theorist Katherine Hayles: ‘recursivity could become a spiral rather than a circle’ (2005: 241, cited in Amoore 2019).
Consider as an example a simplified portrait of product recommendations on e-commerce platform Amazon. Input data about platform users’ purchasing behaviour are fed in real time into an algorithmic model, which considers two products as ‘related’ if they are frequently bought together (Smith and Linden 2017; Hardesty 2019). By learning from customers’ datafied behaviour, the system generates as output a personalized list of items related to the browsed product. On the other side of the screen, millions of Amazon customers navigate recommended products, and decide whether to purchase some of them, or not. It is estimated that automated recommendations alone account for most of Amazon’s revenues (Celma 2010: 3). Since users largely rely on the algorithm to decide what to purchase next, and the algorithm analyses users’ purchasing patterns to decide what to recommend, a feedback loop is established: the model attempts to capture user preferences without accounting for the effect of its recommendations and, as a result, input data are ‘confounded’ by output results (Chaney, Stewart and Engelhardt 2018; Salganik 2018). This techno-social process has implications that go well beyond the engineering aspects of the system. Feedback loops in recommender algorithms are believed to lead to the path-dependent amplification of patterns in the data, eventually encouraging the formation of filter bubbles and echo chambers (Jiang et al. 2019). The idea here is that the very same digital environment from which the algorithm learns is significantly affected by it. Or, in the words of STS scholars MacKenzie and Wajcman (1999), the social shaping of technology and the technological shaping of society go hand in hand. Which leads to our two main sociological questions.
The first one is about the social shaping of algorithmic systems, or the culture in the code (Chapter 2). Platform algorithms like the one in the example above can autonomously ‘learn’ from users’ datafied discourses and behaviours, which carry traces of the cultures and social contexts they originated from. For instance, in 2017, Amazon’s recommender algorithm proposed as ‘related items’ the ingredients for making an artisanal bomb (Kennedy 2017). The recommendation system suggested to customers a deadly combination of products, most likely following the scary shopping habits of a bunch of (wannabe?) terrorists. That was one of the (many) cultures inscribed in the platform data, then picked up by the algorithm as a supposedly innocent set of correlational patterns. Far from being an isolated case, this incident is only one in a long list of algorithmic scandals covered by the press. Microsoft’s infamous chatbot ‘Tay’, which eventually started to generate racist tweets in response to interactions with social media users (Desole 2020), or the ‘sexist’ algorithm behind Apple’s credit card – allegedly offering higher spending limits to male customers (Telford 2019) – are other examples of how machine learning can go wrong.
The main way in which the critical literature surveyed above has dealt with these cases is through the notion of bias. Originated in psychology, this notion indicates a flawed, distorted and ‘unfair’ form of reasoning, implicitly opposed to an ideal ‘neutral’ and ‘fair’ one (Friedman and Nissenbaum 1996). Researchers have rushed to find practical recipes for ‘unbiasing’ machine learning systems and datasets, aiming to address instances of algorithmic discrimination. Still, these attempts are often ex post interventions that ignore the cultural roots of bias in AI (Mullainathan 2019), and risk paradoxically giving rise to new forms of algorithmic censorship. As Završnik puts it: ‘algorithms are “fed with” data that is not “clean” of social, cultural and economic circumstances […]. However, cleaning data of such historical and cultural baggage and dispositions may not be either possible or even desirable’ (2019: 11). While the normative idea of bias has in many cases served to fix real-life cases of algorithmic discrimination and advance data policies and regulations, it hardly fits the sociological study of machine learning systems as social agents. In fact, from a cultural and anthropological perspective, the worldviews of any social group – from a national community to a music subculture – are necessarily biased in some way, since the socially constructed criteria for ultimately evaluating and valuating the world vary from culture to culture (Barth 1981; Latour and Woolgar 1986; Bourdieu 1977). Hence, the abstract idea of a ‘bias-free’ machine learning algorithm is logically at odds with this fundamental premise. ‘Intelligent’ machines inductively learn from culturally shaped human-generated data (Mühlhoff 2020). As humans undergo a cultural learning process to become competent social agents – a process also known as ‘socialization’5 – it can be argued that machine learning systems do so too, and that this bears sociological relevance (Fourcade and Johns 2020). Here I propose to see Amazon’s controversial recommendations and Tay’s problematic tweets as the consequences of a data-driven machine socialization. Since user-generated data bear the cultural imprint of specific social contexts, a first open question is: how are algorithms socialized?
A second question raised by techno-social feedback mechanisms is about the so-called ‘humans in the loop’, and how they respond to algorithmic actions. This concerns the technological shaping of society (MacKenzie and Wajcman 1999), or what in this book I call code in the culture (Chapter 3). The outputs of recommender systems, search engines, chatbots, digital assistants, information-filtering algorithms and similar ‘calculative devices’ powerfully orient the