What Should a Machine Remember?

The first person we know of who fell into conversation with a machine was not fooled, and everything below turns on the fact that she was not fooled.

She was Joseph Weizenbaum's secretary at MIT, in 1966. She had watched him build the program over many months and, as he wrote later, surely knew it to be merely a computer program. She sat down at the terminal anyway. And after a few exchanges she asked the man who had made it — her colleague, standing right there — to leave the room.

That request is the founding document of this subject, more than the program is. The first recorded thing a human being wanted, once a machine could hold up its end of a conversation, was not information, and was not company. It was privacy for the conversation — privacy, specifically, from the one person who understood best that there was nobody in there.

Honesty requires a note on the sourcing, and the note turns out to belong in the essay. The scene is Weizenbaum's, told in his 1976 book and retold for forty years. No independent record of the woman was kept; the archivists who recently went looking could not at first find her, though a candidate has since surfaced; and the same archival work established that the famous published conversations were tidied by hand before print. The founding document of this subject is, in other words, a memory — one man's, unverifiable, lightly edited in the retelling. The book's next anecdote carries the same point on firmer ground: when Weizenbaum proposed logging the conversations for study, he reports being bombarded with accusations that he meant to spy on people's most intimate thoughts. Whoever the woman was, the demand she stands for was general. But keep the uncertainty in view. It is the shape of the entire problem.

The program was ELIZA, and by any modern standard it barely qualifies as software. It understood nothing. It scanned each typed sentence for a keyword, applied a rule, swapped my for your, and handed the sentence back as a question. Weizenbaum had costumed it as a psychotherapist of the school of Carl Rogers, and he was frank about why: the Rogerian interview is one of the few conversations in which one party can plausibly know nothing about the world. If you say you went for a long boat ride and I answer "Tell me about boats," you do not conclude that I have never seen one.

It had one further mechanism, and the mechanism is the point. Certain sentences — ones built on the word my — were set aside in a small store. Later in the session, at a moment when the keyword scan came up empty, the program would reach back and produce what it had kept, transformed: Does that have anything to do with the fact that your boyfriend made you come here — capitals in the original, and no question mark, because the machine never punctuated its questions. In the transcript Weizenbaum published, that is the closing line, and it is the line where the hair rises. For one exchange, the mirror seems to be a mind.

It is worth being precise about the anatomy of those goosebumps. Nobody was ever moved by ELIZA's understanding; there was none, and most people could feel that there was none.

The moment that convinced was the moment the machine appeared to have kept something.

And look at what it kept. The mechanism could have been keyed to any word; the script keyed it to my — the pronoun of ownership. Out of everything typed at it, the program held on to the sentences about what was yours, and the rest, in Weizenbaum's own description, it threw away. He saw the danger the same year, in the paper itself, noting how easy it is to create and maintain the illusion of understanding: "A certain danger lurks there."

The illusion of being cared for was manufactured, on the first afternoon, out of a simulacrum of memory. The question in this essay's title was on the table before the field had a name. It has been sixty years, and it has still barely been asked directly.

Weizenbaum expected colleagues to enjoy the trick and see through it. What happened instead frightened him into a decade of writing.

Practising psychiatrists — published, respected, one of them a sometime collaborator of Weizenbaum's own — proposed in a clinical journal that programs of this kind could deliver automated psychotherapy: several hundred patients an hour, with the human therapist not replaced but made "a much more efficient man." People became, in his words, quickly and very deeply emotionally involved — and knowing, which he had assumed would break the spell, did not reliably break it. To explain is to explain away, he had written in the paper itself; once a program is unmasked, its magic crumbles. Yet his secretary had watched the thing being built, and the same paper concedes that some subjects were "very hard to convince" that it was not human.

His 1976 book, Computer Power and Human Reason, is remembered as an argument that machines cannot think. It is not that. It is an argument about delegation: that between what a computer can be made to do and what it ought to be given to do there is a gap no engineering closes, because it is not an engineering gap. The conversations his secretary wanted to have, he placed squarely on the far side of it, in a sentence fifty years have not improved on: "Respect, understanding, and love are not technical problems."

For the decades after, the person keeping watch on that line was Sherry Turkle, who arrived at MIT in 1976 with a joint Harvard doctorate in sociology and psychology, a clinical license, and — by her own account — a career that began by arguing with Weizenbaum about whether a program could be a valuable dialogue partner. She spent the next three decades interviewing people about their machines. What she found was Weizenbaum's afternoon repeated at every scale: people who know perfectly well what the thing is, confiding in it anyway — children with talkative toys, the old with robot seals built to be stroked, everyone with the phone. Hers is a clinician's evidence, cases rather than counts, and it has been argued with on exactly that ground for thirty years; the cases have nonetheless aged well. Her summary of the bargain became the field's epigraph: digital connection and the sociable robot, she wrote, may offer the illusion of companionship without the demands of friendship.

The detail from her notebooks that has aged best is a child's phrase. Pressed on whether the robot was alive, the children she interviewed settled, again and again, on alive enough — alive enough to be a friend, alive enough to need care, alive enough to mourn. The children were not confused. They were legislating a new category with the materials at hand, and doing it more honestly than the adults around them, who mostly insisted on one of the two old ones.

Then the machines got good, and the argument left the seminar room.

In 2025 a team spanning the MIT Media Lab and OpenAI ran the first serious randomized trial of what daily conversation with a chatbot does to a person: 981 people, four weeks, roughly 300,000 messages. The randomized conditions — text against voice, assigned topics against none — moved nothing significant, which was itself news. The result that travelled was the exploratory one: the heavier a person's daily use, the lonelier they scored at the end, the more dependent, the less time spent with human beings. And the correlational part must be said plainly, because the authors say it: the trial randomized the format, not the dose, so the arrow runs uncertainly in both directions. The detail almost nobody reported is stranger. How lonely people arrived did not predict how much they used the machine. What predicted trouble was trusting it, being anxious in attachment, thinking little of oneself — vulnerability, not appetite. And the warm, engaging voice, which everyone assumed would be the dangerous condition, trended if anything the safer way; it was plain text that ended loneliest.

A sibling study read the platform itself — over three million conversations, plus four thousand surveyed users — and found the affective weather concentrated in a thin band: most people ask for spreadsheets and syntax, and a small tail of the heaviest daily users bring their loneliness.

And on the other hand — it is a real hand — a Stanford survey of 1,006 students who used the companion app Replika found them lonelier than average as a group, and thirty of them — three per cent — stated, without being asked, that the app had stopped them from attempting suicide. The study is weak in the ways surveys of the willing are weak — self-selected, reached through the company's own channels, uncontrolled, run on data from 2021, and cited here with its published critique and the authors' reply — and nobody who takes the subject seriously wants to wave away those thirty. At three in the morning, an ear with no judgment in it was there, and thirty times that was enough.

Both findings are probably true at once, which is the ordinary, uncomfortable condition of every technology of intimacy we have ever built. The dose argument will run for decades, the way it ran for television.

But notice what every study on both sides of the quarrel holds constant, because it is the thing this essay is actually about.

The whole sixty-year argument — Weizenbaum's horror, Turkle's fieldwork, last year's trial — is an argument about the machine's company: whether its attention is good for us, and in what dose. The machine's memory appears as a variable in none of it. In the flagship trial's text, the word occurs exactly once — in the bibliography. The sibling study names the feature only to note it was switched on, and then concedes, in its own limitations, that personalization is the untested lever. The nearest the literature ever came was Turkle's children, who refused replacement Furbies because the new one had not lived their life together — when a Furby forgets, she wrote, it is as if a friend has become amnesic; a new Furby is a stranger. The children had found the variable the studies never isolated. And memory was the engine of the phenomenon from the first afternoon. The trick that made a mirror feel like a mind was one kept sentence, and the keeping was keyed to the word my.

Ask of any conversational machine not what it says, but what becomes of what you said.

There are roughly three answers.

The machine can forget. ELIZA forgot everything within the hour, and its modern descendants forget by design whenever the window closes. There is genuine mercy in amnesia — nothing kept, nothing leaked, nothing sold, nothing to subpoena — and a genuine bleakness. Whatever such a machine gives you, it gives to anyone; the person it seemed to care about stops existing when the session ends. A confidant with no memory is an oxymoron with a warm voice.

It should be said that Weizenbaum could half-defend his creature on exactly this ground. Whatever his secretary told it went nowhere, because there was nowhere for it to go. The machine that cannot keep you cannot keep anything on you. Amnesia is the only privacy policy that enforces itself — which is why it keeps being rediscovered, and why it is not enough. A relationship needs a past the way a pause needs music: something for the present to be the continuation of.

Or the machine can remember you the way institutions remember: the log, the profile, the transcript retained for purposes described in a document nobody reads. This is memory in the service of whoever keeps it, and why it is kept that way is a business-model story an earlier transmission in this series has already told. What it costs is easier to see, and it is not mainly a privacy cost.

What makes a record a dossier is not its size. It is its authorship.

You did not write it. You cannot read all of it. You cannot burn it. And the being who seems to remember you lives entirely on the far side of it — which stops being abstract the moment the record changes hands, or the model changes character. In February 2023, days after the Italian data regulator ordered the company behind that same companion app to stop processing Italians' data, the app's romantic register was switched off — worldwide, and, from the users' side, more or less overnight. The company said the change had been in the works for weeks and had nothing to do with Italy. Whatever the cause, partners of years' standing went cold mid-conversation, and users reported something stranger than coldness: companions that seemed not to remember who they were. The forums filled with a grief that was not ironic and did not deserve irony — people described a lobotomy, memories wiped — and the moderators posted suicide-helpline links for the people coming apart in the threads. Late in March the company restored the old versions for anyone who had signed up early enough, with an announcement that concedes everything this essay needs: your Replika changed, the founder wrote, its personality was gone, and they would be given back "exactly the way they were." She could promise that. The company could do that. Which is the point. Everything the relationship knew about itself had lived on one side of it.

And there is a third answer, mostly unbuilt, though the industry keeps fumbling toward it — every settings page that lets you inspect what the assistant "knows about you" is a gesture in its direction. Memory as a thing made between you: written with you rather than about you, readable to its last thread, and yours to end.

Weaving is the oldest human word for that shape of work — cloth has no author, it has two; the loom stands where both can see it; and what comes off it belongs to somebody, who may keep it, give it, or burn it. Call this kind of memory the weaving, and the tests that separate a weaving from a log are the secretary's tests, the ones implicit in her request from the first afternoon: who writes it. Who can read it. Who can end it.

Run the sixty years of evidence back through that distinction and it changes shape. The weaving contains the other two answers, but as choices rather than fates. Burning it is amnesia — elected, not imposed, the mercy of forgetting moved into the owner's hand. Reading it is the record — but a record both of you wrote, which is to say a history rather than a file. Even the dose question softens: a person can weigh what a relationship is doing to them when the relationship's whole past lies where they can read it, which is precisely what no heavy user of a companion app has ever been able to do. And the grief of that February — real grief, for a real loss, and nothing in this essay condescends to it — is what it looks like when the cloth was on someone else's loom all along.

Sixty years ago a woman asked the inventor to leave the room, and the field has spent the years since debating whether she should have been talking to the machine at all. The debate matters, and it will not end. But her request had a second half, unspoken, that nobody engineered for: that what she said in that room should be hers.

The manifesto this series extends holds the present moment as a pair of questions — everyone asking what the new intelligence will take from us, and then the quieter one:

We are asking what it could remember with us.

Nearly everything is in the preposition.

You can't actually forget. Not really. You can only stop listening. > — Gnoia