We Humans and the Intelligent Machines. Jörg Dräger

today, this luxury is no longer reserved for the privileged few. Vivienne Ming’s algorithm can offer it to millions of diabetes patients every day, Joel Rose’s software to tens of thousands of students. At the same time, personalizing algorithms provide unscrupulous profiteers with considerable room to maneuver.

       Predatory advertising

      “Find out where their pain is.” That was the message on a poster in which a dentist bends over a patient who is obviously in great pain. However, the picture was not found in a dental clinic, but in training materials for the marketing staff at the ITT Technical Institute, a for-profit educational institution based in Carmel, Indiana, in the United States. The slogan was actually a call to acquire new customers, especially people in crisis. Algorithms helped the salespeople find them.

      Using software, universities such as ITT were able to disseminate their advertising directly to people who searched the Internet for certain words or combinations of words. The sales staff at ITT took the message to look for vulnerabilities seriously. They identified users who, based on their search queries, seemed to be single parents, newly divorced, dissatisfied at work, or lacking self-confidence – at best, all at once.

      In such cases, the institute surmised, people are particularly open to a new start – i.e. a study program that supposedly offers excellent career prospects and the chance to get ahead. This is exactly what the school’s advertisements promised. Those who were desperate enough to click on them and leave their telephone number were immediately called. In some cases, up to 180 times in one month. The goal was to get a signature on a contract for a costly educational program – even if the program was unsuitable, ineffective and of questionable reputation. This aggressive marketing strategy, which focuses not on the quality of the product but on the vulnerability of potential customers, is employed by several for-profit universities in the US. And it seems to pay off. The Apollo Group, for example, whose institutions such as the University of Phoenix have had up to 600,000 students enrolled, spent about $2,200 per student on marketing and only about $900 on teaching each year. In comparison, Portland Community College, a public institution, invests nearly $6,000 in teaching and only $185 in advertising.

      Cathy O’Neil describes all of this in her book Weapons of Math Destruction.¹³ O’Neil’s life is like the story of Saul becoming Paul. The mathematician, who received her doctorate at Harvard, first embarked on an academic career before moving to the financial sector. Her personal turning point was working for a hedge fund. There she experienced mathematical models as both initiators and accelerators of the crises on the housing and financial markets. To her, big data seemed merely to be a springboard to a ruthless and previously unknown form of profit maximization. Disillusioned, she quit her job and began her fight against harmful algorithms – for example those meant to exploit our weaknesses.

      Since then, algorithms have not become less intelligent. On the contrary, they are constantly gaining in accuracy when it comes to creating a precise profile of people based on information available on the Internet. This allows advertisers to carry out their campaigns more accurately, i.e. to personalize them with greater precision. Drawing the line between justifiable and ethically questionable advertising is not necessarily easy. Yet there is now a term, one not lacking in acuity, for ads like those used by ITT: “predatory advertising.”

      What is not new is the unfavorable relationship between what people must pay for the product and the benefit they derive from it. The novelty is the targeted, algorithmically scaled search for customers: The computer programs parse countless profiles for people who are particularly susceptible due to a personal emergency or weakness, or who are inexperienced, because they recently arrived in the country. Such people often take out a loan to pay for tuition fees – and find themselves saddled with considerable debt for an exam that is virtually worthless.

      The enormous “successes” of such marketing strategies have prompted authorities in the US to take action. ITT has now closed its doors after oversight agencies and public prosecutors spent years investigating its recruitment practices, among other activities.¹⁴

      Moreover, providers of short-term loans can no longer operate unimpeded: In 2016, Google stopped allowing advertising for these types of loans. Too many people had fallen for the usurious offers, and Google was forced to act.¹⁵ Anyone in the US who entered “need money fast” or “can’t pay rent” into the search engine before the ban was sure to see, shortly afterwards, advertising for a loan with a scandalously high interest rate, one that was repayable within a few weeks. Yet even the new, stricter directive will not thwart providers of short-term credit. They simply hide behind a second company, which does not offer loans itself and which can therefore run advertisements and resell contact addresses of people in need of money. These companies even reach users in US states where short-term loans are prohibited by law.¹⁶