When You Forget, I'll Remember你忘了，我记得

The ideas, a little slower

This chapter left a lot to the pictures, so the stone table could stay quiet. Here is the idea in words.

On page 4 Grandpa says "Put it by her." Xiaoyu is holding two things, so "it" could be either. The word "glass" later in his sentence tells her it is the tea, and only the whole room tells her who "her" is. The meaning of one word depends on other words, sometimes far away from it. Every language model has to solve this, all the time.

On page 6 Haoran tries "The cup didn't fit in the box because it was too big", then changes "big" to "small". One word at the end changes what "it" means. Older networks read a sentence one word at a time and carried what they had read along with them, like a person who keeps a running note in their head. In a long sentence the beginning faded, the way Grandpa loses the top of a comic page by the bottom. Better designs, such as the LSTM, held on longer, but they still had to read in order.

On page 7 Haoran's program does something different. Every word looks at every other word, all at once, and asks how much each one matters to it. "It" looks hard at "cup", a little at "box", hardly at all at "the". Haoran's numbers are made up for the story, but real models work this way. Each word ends up as a mix of the words it paid attention to. The shares always add up to one, so a word cannot give more to "cup" without giving less to something else. Nobody tells the model where to look. It learns that while practising its one job, guessing words.

On page 10 Haoran says his program reads every word at the same time. That sounds like showing off, as Grandpa says, but it is the reason transformers took over. A network that reads in order has to wait for word one before it can start on word two. A transformer works on all the words together, which is the kind of sum a computer's graphics chips are built to do very fast. So it can practise on a mountain of books in days instead of years.

On page 14 Haoran says real models have many heads. Each head is a separate set of attention, with its own idea of what matters. One may learn to match "it" to the thing it stands for, while another watches the word just before. A large model stacks dozens of layers, each with many heads, so the words look at each other again and again and understand a little more each time.

What is true here

Looking back

Attention began in translation. In 2014 Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio let a translating network, as it wrote each word, look back over the whole sentence it was translating and pick out the words that mattered most. The paper was published in 2015. Three years later the trick had become the whole machine.

Attention is all you need

In 2017 a team at Google published a paper with that title. They dropped the old read-in-order network entirely and built one from attention alone, which they called the transformer. Their larger model trained in three and a half days on eight graphics chips and beat the best translation systems of the time. Almost every large language model since has been a transformer.

Which way it looks

Haoran's little program looks both ways, so "it" can see "big" at the end of the sentence. A model called BERT, from Google in 2018, worked like that, by filling in hidden words using the words on both sides. The chat assistants that write messages, like the one in the last chapter, only look back. Each word can see itself and the words before it, because when the model is choosing the next word, the words after it have not been written yet.

Bigger and bigger

The next step was to give a transformer an enormous pile of ordinary writing and let it practise guessing the next word, with no one marking the answers, because the text marks itself. GPT-2 in 2019 had one and a half billion numbers to tune. GPT-3 in 2020 had 175 billion. Researchers found the guessing gets better smoothly and predictably as models, data and computing grow together, and in 2022 a team at DeepMind showed that many big models had been trained on too little text for their size.

Not a spotlight

The word "attention" makes it sound as if the model is paying attention the way a person does. It isn't. It is a sum, a weighted average with weights it has learned. Nobody inside is looking. The weights also don't fully explain why a model said what it said. They show where information was mixed from, in one layer and one head, which is only part of the story. When Grandpa says Xiaoyu's name right, that is a person paying attention, and it is a different thing.

What is pretend

Bridge City, Xiaoyu, Grandpa Chen, Haoran, Lili, Uncle Wu, Auntie Zhang and Mr Chen are made up. So are Haoran's reading program and its percentages, the river boat, the canal town and the 1975 game. The story is set a few years from now. Grandpa's memory is drawn from no one in particular, and the story makes no medical claim. People whose memory comes and goes really do have clearer days and harder ones, and many families find the evenings harder. Nobody can say when a clear day will come, which is why it is worth being there.

Who made this

Yingying Fu created this manga. She chose the story and its people, directed every page, and edited every word and picture. The words were drafted with Claude, an AI by Anthropic. The pictures are made with AI image generation, painted panel by panel from her direction and checked and corrected by hand.