When You Forget, I'll Remember, Chapter 17

An early March afternoon, soft grey light, a little warmer than the morning, an overcast pale grey sky, cold early March, no snow, bare trees except the willow's tiny green tips. At the stone table in the park under the bare willow, whose hanging tips carry the first tiny green buds, Grandpa's round stone stool at the LEFT has a folded white cloth lying on it. Xiaoyu sits on Uncle Wu's old stool at the RIGHT in her plain black wool coat and a grey scarf, small in the picture, both hands round the closed red notebook (dark red cloth cover, black elastic) on the table beside the carved board, which has no pieces set out. She looks at the empty stool. One face, small and calm. The big pale sky and the hanging willow above are the quiet area for the title. No writing anywhere.

Chapter 17第十七章Looking Inside往里看

Dark night, 3 in the morning, the lab lit only by the soft glow of the big wall screen, snow on the roofs outside the tall window at the LEFT, the glass meeting-room wall at the RIGHT, a dark winter night sky with snow falling softly. From behind the desk we see Xiaoyu alone at the middle of the long pale desk in her moss-green cardigan, the lab's big black headset over both ears, her back and the edge of her face in three-quarter lit by the wall screen, which shows a dark field of thousands of tiny soft lights. Her small dark-green suitcase stands packed on the floor beside her chair at the RIGHT. Her sky-blue phone (cracked corner at the top RIGHT) lies face up on the desk by her RIGHT hand, its screen dark. The glass wall at the RIGHT is dark. One face, small, from behind; the dim wall above is the quiet area for the caption.

Zurich, February 2039. At 3 a.m. I was looking inside Swan, a program that tells night nurses which rooms to visit first.二〇三九年二月，苏黎世。凌晨三点，我正在细看一个叫天鹅的程序。夜班护士先去哪几间病房，都由它来排。

Swan is still skipping the same rooms, Professor.教授，天鹅还在跳过那几间病房。

3.02 in the morning, a few minutes after panel 1.1 (not the same minute), dark. Close from directly above on the phone screen alone, filling the panel: Xiaoyu's sky-blue phone (cracked corner at the top RIGHT) lying face up on the pale desk, its screen lit with an incoming video call, painted blank (no words, no name, no picture; the screen text is lettered in afterwards). No hand, no face. The dark desk around it is the quiet area.

Mama, video call妈妈视频通话

My phone lit up beside me. I didn't see it.手机就在手边亮了，我没看见。

The same night, 3 in the morning, from outside. The snowy old city of Zurich in the dark, snow falling softly from a low dark winter sky, steep roofs under snow, and in one plain modern building a single lit window high up, small and warm. No people, nothing moving. The big quiet dark sky above is the area for the caption.

In Bridge City, where Grandpa lived, it was ten in the morning. He was awake and asking for me.外公住在桥城，那边已经是上午十点。他醒着，一直在问我。

Four hours earlier, 11.05 at night, the desk lamp on, snow falling past the window. In her small rented room under the sloping ceiling Xiaoyu sits on the narrow bed in her moss-green cardigan with her laptop on her knees. We see only the BACK of the laptop lid and its glow on her face. Her dark-green suitcase stands open on the floor at the LEFT. One face, stunned and moving fast; the sloping ceiling above is the quiet area.

Four hours before, Mama had called and said, "Come home." I looked for a plane at once.四个钟头前，妈妈来电话说："回来吧。"我立刻上网找机票。

Same minute, lamp-lit night. Close on Xiaoyu lit from below by the laptop, whose back edge shows at the bottom of the panel, jaw set, eyes moving over something off panel. One face in the middle; the dark wall behind is the quiet area.

No planes leave Zurich at night. The first plane home was at seven.苏黎世夜里没有航班。最早一班回家的飞机，要等到早上七点。

Same night. Close on the phone screen alone, filling the panel: Xiaoyu's sky-blue phone (cracked corner at the top RIGHT) lying on a dark blanket, its screen lit with her message to Mama, painted BLANK and lettered in afterwards. No hand, no face. The dark blanket around it is the quiet area.

I'm on the 7 a.m. plane. Call me any time, Mama.妈，我坐早上七点的飞机。随时打给我。

Half past midnight, the desk lamp the only light, snow at the window. In her room Xiaoyu sits on the edge of the narrow bed with her navy coat on over her cardigan and the grey scarf round her neck, her small dark-green suitcase now closed and standing upright beside the bed at the RIGHT. She is turned to the viewer's LEFT, her sky-blue phone at her LEFT ear in her LEFT hand, her RIGHT hand resting on her knee, listening hard. One face, in the middle; the quiet sloping ceiling above is the area for the words.

At half past midnight my lab head, Professor Brunner, called about Swan, which tells night nurses which rooms to visit.十二点半，实验室主任布伦纳教授打来电话，为的是天鹅。天鹅替夜班护士安排先查哪些病房。

Xiaoyu, I'm sorry. Swan is skipping rooms in four hospitals tonight, and you know Swan best.小雨，对不起。今晚天鹅在四家医院都漏了病房，天鹅你最熟。

One in the morning, the lab dark except the desk lamps coming on one by one, the wall screen at the front dark, the tall window at the LEFT black with snow on its ledge. Professor Brunner at the LEFT, tall and lean in his dark grey wool coat with snow melting on its shoulders, a dark green jumper under it, switches on a desk lamp with his RIGHT hand. Xiaoyu at the RIGHT has just come in, in her navy coat and grey scarf, pulling her small dark-green suitcase with her LEFT (near) hand, the suitcase on the near side of her. Both turned three-quarter towards each other and the room. Two faces in the middle band; the dark wall screen above is the quiet area.

Swan was the program that told the night nurses in those hospitals which rooms to visit first.那几家医院的夜班护士先去哪间病房，都听天鹅的。

Same minute, lamp light. Close on Professor Brunner turned to the viewer's RIGHT towards Xiaoyu off panel, grave and plain, steel-rimmed glasses, short grey beard. His face in the middle of the panel; the pale wall behind is the quiet area.

Tonight Swan sent the nurses…今晚天鹅一直让护士……

…past some old people's rooms.……跳过几位老人的病房。

Same minute. Close on Professor Brunner, still turned to the viewer's RIGHT, his eyes lowered, speaking low and tired. His face in the middle of the panel; the pale wall behind is the quiet area.

An old man had a fever for hours before anyone came.有位老先生发高烧，烧了好几个钟头才有人来。

Same night, 1 in the morning, the desk lamps on. Xiaoyu at the RIGHT sits at the long pale desk, her navy coat already hanging over her chair back (no taking it off), her moss-green cardigan showing; her dark-green suitcase stands by the chair. Professor Brunner stands at the LEFT in his green jumper, arms folded, turned to the viewer's RIGHT towards her. Two faces in the middle band; the dark wall screen above is the quiet area.

Tonight the nurses are checking rooms by hand. But there are never enough nurses for every room.今晚护士们只能自己一间一间查。可护士从来都不够，不可能每间都照顾到。

Same night, the desk lamps on. From behind the desk, Xiaoyu in her moss-green cardigan sits small at the middle of the long pale desk, the lab's black headset lying beside her keyboard, her dark-green suitcase on the floor beside her chair at the RIGHT. The big wall screen above her glows faintly with a dense dark-blue field of thousands of tiny dim lights, all alike; a wide quiet band at the top of the panel is where the box sits. One face, three-quarter from behind.

My plane was at seven. Until then, other people's grandfathers were waiting for a nurse.我的飞机七点起飞。在那之前，别人家的爷爷还在等护士。

WHY NOBODY KNOWS. A program like Swan is made of billions of numbers, set by training, not written one by one. Its makers can see every number. They still can't say in plain words what it has learned, or why it gave an answer. Finding out is called interpretability.为什么没人说得清。像天鹅这样的程序，由几十亿个数字组成。这些数字是训练出来的，不是人一个个写进去的。造它的人每个数字都看得到，却没法用大白话说清它学会了什么、为什么给出这个答案。弄清这些的研究，叫“可解释性”。

Same night, about 1.30, the desk lamps on. From behind the desk, Xiaoyu sits at the middle of the long pale desk typing with both hands on the keyboard; Professor Brunner stands at the LEFT a step behind her chair, arms folded, watching the screen. The big wall screen at the front glows with the dark field of thousands of tiny soft lights, a pale band at its top left left blank. Two figures in three-quarter from behind; the dark ceiling above is the quiet area.

First we asked Swan, the nurses' program, to explain why it left those rooms for last.我们先让天鹅自己说说，为什么把那几间病房排在最后。

Same minute. Close on the wall screen alone, filling the panel: the dark blue field with a few soft lights and a plain pale band along the top, painted BLANK and lettered in afterwards with Swan's answer. No people. The dark field is the quiet area.

Rooms 31 and 32 were visited last because those patients were stable. Their readings were normal at the last check.31、32号病房排在最后，因为患者病情稳定，上次检查各项读数正常。

Same minute, lamp light and the glow of the wall screen. Close on Professor Brunner turned to the viewer's RIGHT, looking at the wall screen off panel, eyebrows up, half convinced. His face in the middle of the panel; the pale wall behind is the quiet area.

That sounds sensible.听着挺有道理。

Same minute. Close on Xiaoyu turned to the viewer's LEFT towards Professor Brunner off panel, shaking her head slowly, tired and sure, lit by the desk lamp and the cool glow of the wall screen on her face, in her moss-green cardigan. Her face in the middle of the panel; the dim wall behind is the quiet area.

The last check was at six. By eight, the old man in room 31 had a fever.上次检查是六点钟。到八点，31号那位老先生已经发烧了。

Same night. Wide and quiet from behind the desk: Xiaoyu small at the middle of the long pale desk in her cardigan, the big wall screen above her lit softly with a pale blank band where Swan's answer was, no words. A wide quiet band of dark wall below the screen is the area for the box. One figure, three-quarter from behind.

Swan's answer sounded like a reason. It wasn't the real one.天鹅这话听着像个理由，可真正的原因不是这个。

ASKING IT WHY. A program can explain itself. But the explanation is produced like any other text, so it may not match what really happened inside. In 2023 researchers found that a program's step-by-step reasons sometimes left out the real cause of its answer.问它为什么。程序能解释自己。可这些解释和它写的其他文字一样，都是生成出来的，未必符合它内部真正发生的事。2023年，研究者发现，程序一步步列出的理由，有时恰恰漏掉了它这样回答的真正原因。

Half past two in the morning, the desk lamps on. From behind the long pale desk, Xiaoyu in her cardigan pulls the big black headset onto her head with both hands; her sky-blue phone lies face up on the desk by her RIGHT hand. Through the glass wall at the RIGHT, in the small meeting room, Professor Brunner stands at a table with a flat speakerphone, solid and clear. The people are solid and opaque; reflections only on empty glass. Two figures; the dark wall above is the quiet area.

At half past two, Professor Brunner went next door to talk to the four hospitals. I listened on the lab headset.两点半，布伦纳教授到隔壁去，跟四家医院通电话。我戴着实验室的耳机，在这边听。

Same night. Close on the wall screen alone, filling the panel: the dark blue field of thousands of tiny soft round lights scattered like fireflies, a few brighter than the rest, a plain pale band at the top left left blank. No people, no letters, no numbers. The dark field is the quiet area.

Swan was made of millions of tiny parts. On the screen, each part was a light.天鹅里面有几百万个小零件。屏幕上，每个零件是一个小光点。

Same night. Close on the wall screen alone: one small light ringed in pale blue, three thin lines running from it to three soft simple shapes at the edges of the panel, a door shape, a lift shape (a box with two sliding doors) and a snowflake shape. Shapes only, no letters, no numbers, no people. The dark field is the quiet area.

One part lit up for room 31, for a broken lift, and for snowy roads.有个零件，碰上31号病房会亮，碰上电梯坏了会亮，碰上路面积雪也会亮。

Same night, about 3 in the morning. From behind the desk, Xiaoyu in the headset (pushed back off one ear) rubs her eyes with her LEFT hand, her RIGHT hand off the mouse; above her the wall screen shows one bright light ringed in pale blue with a faint door shape, a lift shape and a snowflake shape drifting round it (one part, many things), a wide quiet band of dark wall at the bottom is the area for the box. One figure, three-quarter from behind.

I couldn't read Swan one part at a time.这样一个零件一个零件看下去，我根本读不懂天鹅。

ONE PART, MANY THINGS. One unit, or part, of a network can light up for things with nothing in common, like cat faces and car fronts. A network has more ideas to hold than units, so ideas overlap, the way many radio stations share the same airwaves.一个零件，管好几样。网络里的一个单元（也就是一个零件），可能被几样毫不相干的东西点亮，比如猫脸和汽车车头。网络要装的概念比单元多，只好叠在一起放，就像许多电台共用同一片电波。

Same night, about 3.15, the desk lamps on. From behind the desk, Xiaoyu in her moss-green cardigan and the black headset leans in towards the wall screen, which now shows the same lights gathered into many small separate clusters on the dark blue field, glowing a cool blue-white across the desk. A wide band of quiet dark wall above the screen is the area for the box. One figure, three-quarter from behind.

So we ran a second, simpler program that sorted Swan's lights into features, each one lit for one thing.于是我们又跑了一个更简单的程序，把天鹅的光点分成一个个“特征”，每个特征只为一样东西亮。

SORTING THE LIGHTS. A second, simpler program can split a network's activity into "features", only a few lit at a time. In 2024 a team at Anthropic found up to 34 million features in one model, one of them for the Golden Gate Bridge. When they turned that feature up, the model talked as if it were the bridge.把光点分开。可以再用一个更简单的程序，把网络的活动拆成一个个“特征”，每次只有少数几个亮着。2024年，Anthropic的一个团队在一个模型里找到了多达3400万个特征，其中一个对应金门大桥。把这个特征调强，模型说起话来就像自己是那座桥。

A memory, with soft grey edges and a faint paler wash, a winter afternoon in the flat, a flat grey winter sky and bare trees at the balcony glass. Grandpa sits in the olive armchair at the LEFT in his dark navy padded coat worn open over the grey cardigan and a grey scarf, waving his RIGHT hand as if to say

Grandpa never once pressed a call button for help. He hated to be any trouble.外公从来没按过一次呼叫铃。他最怕麻烦别人。

Same night. Close on the wall screen alone, filling the panel: one cluster of lights lit bright on the dark blue field, the rest dimmed, a thin plain arrow running from it to a small simple door shape with a clock face (no numbers), and a plain pale band along the top, painted BLANK and lettered in afterwards. No people. The dark field is the quiet area.

Feature 4117. Asked for nothing tonight. Pushes Swan to visit later.特征4117。今晚什么都没要过。让天鹅推迟查房。

Same night, about 3.30. Close on Xiaoyu turned to the viewer's RIGHT towards the wall screen off panel, lit from the side by the bright cluster, her eyes widening, understanding. Her face in the middle of the panel; the dim wall behind is the quiet area.

Swan learned that patients who never ring need nothing. But some old people never ring.天鹅学到的是：不按铃的病人就没事。可有的老人从来不按铃。

3.38 in the morning, the desk lamps on. From behind the desk, Xiaoyu in her cardigan sits back in her chair, both hands off the keyboard, the black headset on, the wall screen showing the bright cluster now dimmed. Through the glass wall at the RIGHT, Professor Brunner stands in the small meeting room by the speakerphone with his RIGHT thumb up, solid and clear. The people are solid and opaque; reflections only on empty glass. Two figures; the dark wall above is the quiet area.

At 3.38 I turned that feature down in a test copy of Swan.凌晨3.38，我在天鹅的测试副本里调弱这个特征，验证猜测。

In the practice copy, Swan now visits the quiet patients first. The hospitals will use it that way tonight.练习副本里，天鹅现在先查安静的病人了。今晚医院就按这个来。

3.40 in the morning, the desk lamps on. Xiaoyu lifts the black headset off her head with her LEFT hand and picks up her sky-blue phone in her RIGHT hand; we see the BACK of the phone and its glow on her face. One face, the first second of seeing, in the middle band; the dim wall above is the quiet area.

At 3.40 I took the headset off. My phone had been on silent for work, so Mama's calls only lit it up.凌晨3.40，我摘下耳机。干活时手机调了静音，妈妈打来的电话，只让屏幕亮了亮。

Same minute. Close on the phone screen alone, filling the panel: Xiaoyu's sky-blue phone (cracked corner at the top RIGHT) lying on the pale desk, its screen lit with two lines of text, painted BLANK and lettered in afterwards. No hand, no face. The dark desk around it is the quiet area.

Missed video call, Mama, 3.02 a.m. Missed video call, Mama, 3.09 a.m.未接视频通话妈妈凌晨3.02 未接视频通话妈妈凌晨3.09

Same minute. Close on the phone screen alone, filling the panel: the sky-blue phone lying on the pale desk, its screen lit with Mama's short message, painted BLANK and lettered in afterwards. No hand, no face. The dark desk around it is the quiet area.

Grandpa woke up clear-headed and asked for you. He says to wear a hat in Zurich. He's asleep again.外公醒了，人很清楚，问你呢。他让你在苏黎世记得戴帽子。现在又睡了。

Mama held the phone to his ear while I talked. He didn't wake.我说话的时候，妈妈把手机贴在外公耳边。他没醒。

‹Grandpa, it's Xiaoyu. I'm on my way. Wait for me.›外公，我是小雨。我在路上了，你等等我。

Inside a small plane at seven in the morning, blue dawn light. A small oval window at the LEFT shows blue dawn over snowy mountains, a clear cold winter sky. Xiaoyu sits in the grey window seat in her navy coat and grey scarf, turned to the window, her sky-blue phone face down on her knee under her LEFT hand. One face, small and still; the pale grey cabin wall above is the quiet area. No other passengers in view.

At seven, my plane took off.早上七点，我的飞机起飞了。

Nine twenty in the morning, grey overcast late-winter daylight through the high glass wall, a pale cloudy sky and bare trees beyond, no snow. At the LEFT, the arrivals sliding doors; Xiaoyu has just come through in her navy coat and grey scarf, pulling her small dark-green suitcase with her RIGHT hand, turned to the viewer's RIGHT, seeing him. Haoran stands alone at the RIGHT in his long dark grey coat, hands at his sides, no sign, not waving, turned to the viewer's LEFT towards her so we see his LEFT ear with the white earbud. A few far, faint travellers only. Two faces in the middle band; the pale ceiling above is the quiet area. The sliding doors have just closed behind her; she is on the hall side.

I changed planes once and landed in Bridge City on Saturday morning. Haoran waited past the arrivals doors and didn't wave.中间转了一次机，星期六上午我落地桥城。浩然等在到达口外面，没有挥手。

Same minute. Close on Haoran turned to the viewer's LEFT towards Xiaoyu off panel, so we see his LEFT ear with the white earbud, sad and steady, square black glasses. His face in the middle of the panel; the pale hall behind is the quiet area.

Grandpa died this morning, before it got light.外公今天早上走的，天还没亮。

Same minute. Close on Xiaoyu turned to the viewer's RIGHT towards Haoran off panel, very still, her mouth a little open, her RIGHT hand slipping off the suitcase handle at the bottom edge. Her face in the middle of the panel; the pale hall behind is the quiet area.

Was someone with Grandpa?有人陪着外公吗？

Same minute. Side-on, Haoran at the RIGHT holds Xiaoyu, who is at the LEFT, with both arms round her back, turned to the viewer's LEFT so we see his LEFT ear with the white earbud, his eyes closed. Xiaoyu's face is pressed against his dark grey coat, hidden, her arms closed round his back, her hands flat on his coat (four arms in all). Her small dark-green suitcase stands upright and alone at the far LEFT edge. Two figures in the upper middle; the pale floor below is the quiet area.

Your mama and I were both with Grandpa.你妈妈和我都在他身边。

On Friday morning Mama phoned Haoran in San Francisco. He flew home and reached Grandpa near midnight that night.桥城时间星期五早上，妈妈给远在旧金山的浩然打了电话。他坐最早的航班赶回来，当天快半夜时到了外公身边。

About eleven in the morning, grey light at the balcony glass, no lamp on, a pale overcast sky and bare plane trees beyond the glass, cold late February. At the open dark green gate in the LEFT wall stands Mama in a dark grey jumper and black trousers, her eyes red, both hands reaching for Xiaoyu. Xiaoyu stands in the doorway in her navy coat and grey scarf, her hands coming up to meet Mama's. Haoran stands behind her on the landing in his long dark grey coat, holding her small dark-green suitcase in his LEFT (near) hand, turned to the viewer's LEFT so we see his LEFT ear with the white earbud. Beyond them the olive armchair stands empty. Three faces in the middle band; the pale wall above is the quiet area. The door and gate are both OPEN against the wall; Mama stands just inside the flat, Xiaoyu in the doorway itself, Haoran out on the landing behind her, nobody behind the bars.

‹Your grandpa went very peacefully, just before it got light.›你外公走得很安详，就在天快亮的时候。

Same morning, grey light, a pale overcast sky and bare trees beyond the balcony glass. Close on the empty olive armchair: the grey cardigan folded on the seat, the corner of the dark red notebook showing from its RIGHT pocket. On the wooden side table beside the chair lies Willow's cream wristband, its orange light dark (no glow, no halo). No people. The plain wall behind is the quiet area.

Grandpa's grey cardigan was folded on his armchair, with his notebook still in the pocket.外公的灰开衫叠好了，放在他的扶手椅上，本子还揣在口袋里。

Same morning. Side-on close: Mama holds Xiaoyu's face in both her hands (two hands), their foreheads almost touching, Xiaoyu's hands holding Mama's wrists (two hands), her navy coat and Mama's dark grey jumper. Two faces in the middle of the panel; the plain wall behind is the quiet area. No words. They stand just inside the flat by the OPEN gate, Haoran out of frame on the landing.

A memory, with soft grey edges, the dim lamp before dawn. On the dark red and cream checked quilt lies Grandpa's LEFT hand, a smooth manga hand, Willow's cream band on its wrist with a dim orange dot, held in both of Mama's hands, her thin silver watch on her LEFT wrist. Only hands and the quilt, no faces. The plain quilt is the quiet area.

Mama had held Grandpa's left hand.妈妈握着外公的左手。

A memory, with soft grey edges, the same dim lamp before dawn. On the same dark red and cream checked quilt lies Grandpa's RIGHT hand, a smooth manga hand, held in both of Haoran's hands (Saturday about 4.30 a.m.; his sleeve the cuff of his charcoal jacket over a white t-shirt cuff, coat off). Only hands and the quilt, no faces. The plain quilt is the quiet area.

Haoran had held Grandpa's right hand.浩然握着外公的右手。

Three in the afternoon, grey light at the balcony glass, one lamp on, a pale overcast sky and bare trees at the balcony glass. At the round folding table Mama sits at the LEFT in her dark grey jumper, her RIGHT hand resting flat on her own phone, which lies face down on the table. Xiaoyu sits at the RIGHT, now in a plain black jumper and dark trousers, leaning towards her. Two faces in the middle band; the pale wall above is the quiet area.

‹Mama, what did Grandpa say on his last day?›妈，外公最后那天，都说了些什么？

Same afternoon. Close on Mama, tired and kind, her RIGHT hand holding her phone out across the table to Xiaoyu off panel to the RIGHT (the moment after, the phone's back to us). Her face in the middle of the panel; the pale wall behind is the quiet area.

‹He slept most of the day. Willow, his care program, kept a record. Look.›他大半天都在睡。照看他的小柳有记录，你看。

Same afternoon. Close on the phone screen alone, filling the panel: Mama's phone lying on the round table, its screen lit with Willow's day record in a few plain lines, painted BLANK and lettered in afterwards. No hand, no face. The pale tabletop is the quiet area.

Friday 25 February. Asleep, 15 hours. Water, half a glass. Pain, none. Words spoken, 37. Words not saved, by family choice.2月25日星期五睡眠15小时饮水半杯无疼痛说话37个词按家属要求未保存说话内容

Same afternoon. Close on Xiaoyu in her black jumper holding Mama's phone lowered in both hands, its back to us, staring past it at nothing. One face in the middle of the panel; the grey balcony window behind her, a flat grey winter sky and bare trees in it, is the quiet area.

Willow knew how many words Grandpa said on his last day.外公最后一天说了多少个词，小柳清清楚楚。

It didn't know a single one.说了什么，它一个字也不知道。

Same evening, dusk, the lamp on, a dim grey winter sky and dark bare trees at the balcony glass. Mama sits in the olive armchair at the LEFT with Grandpa's grey cardigan across her knees, smoothing it with both hands (two hands), the corner of the dark red notebook showing from its RIGHT pocket. One face in the upper middle of the picture; the lamp-lit floor below is the quiet area.

On Monday evening, Grandpa's clothes would go to the funeral home.星期一傍晚，外公的衣服就要送去殡仪馆了。

‹He'll wear this cardigan, with his notebook in the pocket, just as he always did.›就让他穿这件开衫，本子揣兜里，跟平时一样。

Dusk, cold, an overcast late-winter sky, the lake grey beyond the rail with bare willows and plane trees along the shore, no snow, no leaves, a few lit windows across the water. Haoran stands at the LEFT leaning on the rail in a black jumper over a white collar, looking out at the lake, turned to the viewer's LEFT so we see his LEFT ear with the white earbud. Xiaoyu stands at the RIGHT in her black jumper, a tea cup in each hand (one in each), holding one out to him with her LEFT hand. Two faces in the middle band; the dim sky above is the quiet area.

I had to know whether Grandpa minded that I wasn't there. Haoran had been with him.我得知道，我没在，外公心里怪不怪我。浩然一直在他身边。

Same overcast late-winter dusk. Close on Xiaoyu turned to the viewer's LEFT towards Haoran off panel, her cup held in both hands, too intent, her eyes bright and fixed. Her face in the middle of the panel; the grey dusk behind is the quiet area.

What did Grandpa say, Haoran? Tell me every word.浩然，外公都说了什么？一个字一个字告诉我。

Same overcast late-winter dusk. Close on Haoran turned to the viewer's RIGHT towards Xiaoyu off panel, so the ear we see is his bare RIGHT ear, his cup in his RIGHT hand, looking down into it, square black glasses. His face in the middle of the panel; the grey dusk behind is the quiet area.

Grandpa thought I was you. He held my hand and called me Xiaoyu.外公把我当成你了。他拉着我的手，叫我小雨。

‹I'm here, Grandpa.›外公，我在呢。

Same overcast late-winter dusk, a little darker. Xiaoyu at the RIGHT has turned away from him, her cup set down on the rail beside her, her shoulders tight. Haoran at the LEFT turned to the viewer's RIGHT towards her (the ear we see is his bare RIGHT ear), his RIGHT hand reaching out towards her shoulder and stopping in the air, his LEFT hand at his side. Two faces in the middle band; the dim sky above is the quiet area.

You let Grandpa think you were me.你就让外公把你当成我了。

He was holding my hand. I couldn't tell him you were still on the plane.他攥着我的手呢。我怎么开得了口，说你还在飞机上？

About ten at night, one park lamp, frost on the stone, a cold clear night sky, no snow, late winter, the bare willow overhead, the lake dark beyond. At the stone table Grandpa's round stone stool at the LEFT is empty. Xiaoyu sits on Uncle Wu's old stool at the RIGHT in her plain black wool coat and a grey scarf, facing the empty stool across the carved board, which has no pieces set out. One face, small. The big dark quiet sky and the willow above are the area for the caption.

That night I went to the stone table in the park and sat on Uncle Wu's stool, across from Grandpa's.那天夜里，我去了公园的石桌，坐在吴爷爷的凳子上，对着外公的凳子。

Same cold late-winter night, the park lamp, bare branches against a clear dark sky. Close on Xiaoyu in her black coat turned to the viewer's LEFT towards the empty stool off panel, both hands in her coat pockets, tears on her face, quiet. Her face in the middle of the panel; the dark frosted ground behind is the quiet area.

Grandpa asked for me, and I was looking inside a machine.外公找我的时候，我正埋头往一台机器里面看。

A memory, with soft grey edges. Midnight, the desk lamp the only light. Xiaoyu at eighteen sits cross-legged on her bed with her hair loose, in her navy t-shirt, the open red-and-gold tin beside her and about twenty thin exercise books in rubber bands spread on the bedspread, reading the oldest in both hands so we see its brown cover. One face in the middle band; the dark wall above is the quiet area.

When Uncle Wu died, I read every one of his notebooks in a single night.吴爷爷走后，我一个晚上把他的本子全读完了。

Same cold late-winter night, the park lamp, a clear dark sky, no snow. Xiaoyu stands up from Uncle Wu's stool at the RIGHT, turned to the viewer's RIGHT towards the way home, her face clear and sure, her black coat and grey scarf. One face, in the middle band; the lamp, the hanging willow and the dark sky above are the quiet area.

I looked inside machines every day. I had never read Grandpa's whole notebook.我天天往机器里看，外公的本子，却从来没有从头读到尾。

Sunday, eight in the morning, grey overcast late-winter light at the small kitchen window, bare trees beyond. Mama stands at the stove at the LEFT in her plain black cardigan over a white blouse, her RIGHT hand lifting the kettle by its handle and pouring into a tall clear tea glass standing on the counter, her LEFT hand on the counter edge. Xiaoyu stands in the kitchen doorway at the RIGHT in her charcoal-grey knitted cardigan over a black t-shirt, speaking to her. Two faces in the middle band; the pale wall above is the quiet area.

‹Mama, Grandpa once said I should read only the pages I need. Now I need every page.›妈，外公以前说过，本子要用得着才翻。现在，我每一页都用得着。

Same morning. Close on Mama turned to the viewer's RIGHT towards Xiaoyu off panel, the tall clear tea glass in her RIGHT hand, firm and tired. Her face in the middle of the panel; the pale kitchen wall behind is the quiet area.

‹It's his notebook. He meant only what you need, not everything.›那是他的本子。他说的是用得着的才看，没说全看。

Same morning. Close on Xiaoyu turned to the viewer's LEFT towards Mama off panel, gentle, seeing something in Mama's face, her head tilted a little. Her face in the middle of the panel; the pale kitchen wall behind is the quiet area.

‹Mama, what are you afraid the notebook says?›妈，你是怕本子里写着什么？

Same morning. Mama sits at the small square kitchen table, both hands round the tall clear tea glass (two hands), looking down into it. One face in the upper middle of the picture; the grey table below is the quiet area.

‹He grew old and ill while I lived in Scotland. I'm afraid of what he wrote about me.›他老了，病了，我一直在苏格兰。他写了我什么，我怕看。

A few minutes later, grey overcast morning light at the balcony glass, bare trees beyond. In the living room Mama stands at the LEFT in her plain black cardigan over a white blouse beside the olive armchair, Grandpa's grey cardigan folded on the chair's arm; she has just taken the dark red notebook from the cardigan's RIGHT pocket and holds it out in both hands. Xiaoyu stands at the RIGHT in her charcoal-grey cardigan over a black t-shirt, both hands coming up to take it, the moment before they touch. Two faces in the middle band; the pale wall above is the quiet area.

‹You have one night. In the morning, the notebook goes back in his pocket.›就一晚上。明天早上，本子放回他兜里。

Ten at night, the desk lamp the only light, a cold dark late-winter night beyond the window. Xiaoyu sits at her small desk under the window in her charcoal-grey cardigan over a black t-shirt, Grandpa's dark red notebook open before her, five small pads of coloured paper slips in a row beside it (yellow, green, blue, brown, white), a pen in her RIGHT hand. Through her open door at the RIGHT, a glimpse of the living room: Papa, tall, asleep on the sofa in his navy jumper under a blanket, his face turned into the cushion, one big sock-footed foot sticking out. One face, hers, in the middle band; the dark window above is the quiet area.

Papa had flown in that evening and fallen asleep on the sofa in his socks.爸爸傍晚刚下飞机，穿着袜子就在沙发上睡着了。

I read Grandpa's notebook the way my lab reads a program, with a coloured paper slip for each person.我像实验室里读程序那样读外公的本子，每个人用一种颜色的小纸条。

Same night, lamplight. Close from directly above on the first page of the notebook lying flat, cream paper with faint blue lines, the ink faded, the page painted BLANK and its words lettered in afterwards; a white paper slip lies at its edge. No hands, no faces. The desk around is quiet.

The first page, from the summer I was eleven. White was me.第一页，写在我十一岁那年夏天。白色是我。

周小雨。外孙女。十一岁。喜欢打牌。在学中文。很聪明。周小雨。外孙女。十一岁。喜欢打牌。在学中文。很聪明。

Same night, lamplight. Close from directly above on one notebook page lying flat, cream paper, a few soft unreadable pencil marks at the top and one short entry below, painted BLANK and lettered in afterwards; a yellow slip at its edge. No hands, no faces. The desk around is quiet.

Yellow was Uncle Wu. "Old Wu took back a move, then said he hadn't."黄色是吴爷爷。

老吴悔棋，还说没悔。老吴悔棋，还说没悔。

Same night, lamplight. Close from directly above on one notebook page lying flat, soft unreadable pencil marks at the top and one short entry below, painted BLANK and lettered in afterwards; a green slip and a white slip at its edge. No hands, no faces. The desk around is quiet.

Green was Grandma Meilan. "Poured Meilan a cup of tea. She doesn't drink it, so I do."绿色是外婆美兰。

给美兰倒了茶。她不喝，我喝。给美兰倒了茶。她不喝，我喝。

Brown was Papa, whose name is Callum. "Callum came. Lost at cards again. A good man."棕色是爸爸。

卡伦来了。打牌又输了。是个好人。卡伦来了。打牌又输了。是个好人。

Same night, lamplight. Close from directly above on one notebook page lying flat, soft unreadable pencil marks at the top and three short entries below, painted BLANK and lettered in afterwards; a yellow slip at its edge. No hands, no faces. The desk around is quiet.

"Old Wu has died." He wrote it three times that summer. Each time, he had just found out again.那年夏天他写了三回。每一回，他都像头一回听说。

二〇三三年七月十九日　老吴走了。七月二十四日　老吴走了。八月三日　老吴走了。二〇三三年七月十九日　老吴走了。七月二十四日　老吴走了。八月三日　老吴走了。

Most pages needed two colours or three.大多数页要贴两三种颜色。

One page could be about several people at once.一页上，可以同时写着好几个人。

Much later that night, lamplight. From above, the closed dark red notebook lies on the desk, fat with dozens of coloured slips sticking out of its page edges like feathers (yellow, green, blue, brown), and a white slip showing on every page. No people. The plain desk around it is quiet.

There was a white slip on every page.每一页都有一张白纸条。

Same night, lamplight. Close from directly above on one notebook page lying flat, cream paper, soft unreadable pencil marks at the top and one short entry below, painted BLANK and lettered in afterwards; a white slip at its edge. No hands, no faces. The desk around is quiet.

"July 2027. Xiaoyu has gone back to Scotland. She'll come back."那年夏天，我回了苏格兰。

二〇二七年七月　小雨回苏格兰了。她还回来。二〇二七年七月　小雨回苏格兰了。她还回来。

Same night, lamplight. Close from directly above on one notebook page lying flat, cream paper, soft unreadable pencil marks at the top and one entry below, painted BLANK and lettered in afterwards; a white slip and a yellow slip at its edge. No hands, no faces. The desk around is quiet.

The summer I was sixteen, on a day when his mind was clear. "Waited for Xiaoyu to play chess. She didn't come. Never mind. She'll come back."我十六岁那年夏天，他难得清醒的那一天。

二〇三一年七月十九日　等小雨下棋。她没来。没关系，她还回来。二〇三一年七月十九日　等小雨下棋。她没来。没关系，她还回来。

About two in the morning, lamplight, a cold dark late-winter night at the window. Close on Xiaoyu turned to the viewer's LEFT, her LEFT hand over her mouth, her black watch on that wrist, her eyes wet. Her face in the middle of the panel; the dark window behind is the quiet area.

Grandpa never minded. I still did.外公从来没怪过我。是我自己放不下。

About three in the morning, lamplight. Close from above on the notebook's last pages fanned under one thumb (Xiaoyu's thumb, the only hand in the panel, entering from the bottom edge): the last written page, then plain empty pages with a faint pale grid. No face, no readable writing. The dark desk around is quiet.

After the last page, the rest of the notebook was empty.最后一页后面，本子全是空白。

About three in the morning, lamplight. Close from directly above on the last written page of the notebook lying flat, cream paper, the hand bigger and slower, painted BLANK and lettered in afterwards; a white slip at its edge. No hands, no faces. The desk around is quiet.

The last page, written the Sunday before he died. "Xiaoyu. Granddaughter. Twenty-three. Learning in Zurich. Very clever. She'll come back. No hurry."最后一页，写在他走前的那个星期天。

二〇三九年二月二十日　小雨。外孙女。二十三岁。在苏黎世学本事。很聪明。她还回来。不急。二〇三九年二月二十日　小雨。外孙女。二十三岁。在苏黎世学本事。很聪明。她还回来。不急。

Almost four in the morning, lamplight. Close from directly above on a worn sheet of squared practice paper (white with a pale green grid of squares), unfolded flat from small squares and creased soft, with big uneven child's characters on it, one character crossed out and written again beside it, painted as a blank grid and lettered in afterwards; the dark red notebook's open back cover lies at its edge. No hands, no faces. The desk around is quiet.

Inside the back cover, folded small, was the toast I wrote for his seventieth birthday when I was eleven.封底里夹着一张叠得小小的纸，是我十一岁那年给外公七十大寿写的祝酒词。

外公，你教全公园的人下棋。你教我打牌。你忘了，我记得。外公，你教全公园的人下棋。你教我打牌。你忘了，我记得。

Half past six in the morning, first grey light at the balcony glass, a pale overcast late-winter dawn with bare trees beyond. Xiaoyu sits at the round folding table in the middle in her charcoal-grey cardigan over a black t-shirt, the notebook bristling with coloured slips before her. Mama steps out into the living room in a grey cardigan over pale nightclothes, her silver watch on her LEFT wrist. Papa lies asleep on the sofa against the RIGHT wall under a blanket in his navy jumper, one arm hanging off the edge. Two faces awake in the middle band, Papa's asleep and turned away; the pale wall above is the quiet area. Grandpa's bedroom door at the back RIGHT is OPEN behind Mama as she steps out into the living room.

At first light, Mama came for the notebook.天刚亮，妈妈来拿本子了。

Same morning. Close on Xiaoyu, the notebook open in both hands before her chest (its pages towards her, so we see the back of its cover), looking up at Mama off panel to the RIGHT, turned to the viewer's RIGHT. Her face in the middle of the panel; the pale wall behind is the quiet area.

‹Mama, let me read you two pages first. They're about you.›妈，先听我念两页，写的是你。

Same morning, grey light. Close from directly above on one notebook page lying flat, soft unreadable pencil marks at the top and one short entry below, painted BLANK and lettered in afterwards; a blue slip at its edge. No hands, no faces. The tabletop around is quiet.

Blue was Mama, called Huihui. "Huihui is going back to Scotland again. Let her go. She has her own home."蓝色是妈妈，外公叫她慧慧。

慧慧又要回苏格兰了。走吧。她有她的家。慧慧又要回苏格兰了。走吧。她有她的家。

"December 2038. Huihui has come home. She didn't need to, the silly girl."去年十二月，妈妈回来照顾他。

二〇三八年十二月　慧慧回来了。不用回来的，傻闺女。二〇三八年十二月　慧慧回来了。不用回来的，傻闺女。

Same morning. Mama, in her grey cardigan over pale nightclothes, sits down hard on a chair at the RIGHT of Xiaoyu, facing us, laughing and crying at once, her RIGHT hand over her mouth; Xiaoyu, in her charcoal-grey cardigan, is at the LEFT, turned to the viewer's RIGHT, and puts her RIGHT (near) hand on Mama's back. Two faces in the upper middle of the picture; the grey tabletop below is the quiet area.

‹Your grandpa never blamed me at all.›你外公一点都没怪我。

A little later, the same grey light, Mama still in her grey cardigan over pale nightclothes, Xiaoyu in her charcoal-grey cardigan. Side-on, Xiaoyu at the LEFT holds the notebook out to Mama in both hands; Mama at the RIGHT folds both her hands over Xiaoyu's and presses the notebook back towards her (four hands in all). Mama's thin silver watch on her LEFT wrist. Two faces in the middle band; the pale wall above is the quiet area.

‹He wrote us all down so he wouldn't lose us. Keep the notebook, Xiaoyu.›他把咱们一个个都写下来，就怕把咱们弄丢了。小雨，本子你留着吧。

I didn't write in Grandpa's notebook. I wasn't ready to. I wrote him a page of his own.我没有在外公的本子上写字，还下不了笔。我给他单独写了一页。

Monday late morning, daylight. Close from directly above on Xiaoyu's finished sheet of squared practice paper lying flat, crisp, six short lines in a neat grown hand, painted as a blank grid and lettered in afterwards. No hands, no faces. The desk around is quiet.

"Grandpa. Eighty-two. Taught the whole park chess. Taught me cards. Very clever. Xiaoyu remembers you."我照着他的写法写的。

外公。八十二岁。教全公园的人下棋。教我打牌。很聪明。小雨记着你。外公。八十二岁。教全公园的人下棋。教我打牌。很聪明。小雨记着你。

Five in the afternoon on Monday, the lamp on, dusk at the balcony glass, a dim grey sky and dark bare trees. At the round folding table Mama stands at the LEFT in her dark navy knitted dress folding Grandpa's grey cardigan. Xiaoyu in her dark grey roll-neck jumper, in the middle, slides her page, folded in four, into the cardigan's RIGHT pocket with her RIGHT hand (the notebook is not there; it lies on her desk). Papa at the RIGHT, tall in a plain black jacket over his navy jumper, holds a plain dark cloth bag open with both hands. Three faces in the upper middle band; the lamp-lit tabletop below is the quiet area.

That evening Grandpa's cardigan went to the funeral home, with my page in the pocket.那天傍晚，外公的开衫送去了殡仪馆，兜里装着我写的那一页。

Grandpa would have liked your page in his pocket, Xiaoyu.小雨，你这页放在他兜里，他会喜欢的。

Eleven at night, the stairwell's one landing light. Haoran stands at the RIGHT in the open doorway of his parents' flat, in his dark grey jumper over a white collar, turned to the viewer's LEFT towards her so we see his LEFT ear with the white earbud. Xiaoyu stands at the LEFT on the landing before Grandpa's dark green metal gate, in her dark grey roll-neck jumper, holding the dark red notebook against her chest with both hands. Two faces in the middle band; the plain wall above is the quiet area. His parents' door is OPEN behind him with warm light spilling out; he stands in the doorway on the threshold, Xiaoyu out on the landing.

You stayed with Grandpa when I couldn't. Thank you.我赶不回来，是你陪着外公。谢谢你。

May I read his notebook tonight? I want to know him too.外公的本子，今晚能借我看看吗？我也想认识他。

Tuesday, a quarter to nine in the morning, grey light. In front of the plain Chinese funeral hall of grey brick, with a grey tiled roof with gently upturned eaves, two red columns by the glass door at the back, grey stone steps, and a bare tree at the RIGHT; an overcast pale grey late-winter sky, cold, no snow; no sign, plaque, banner or writing. Haoran stands at the LEFT in a plain black suit and white shirt with a small white paper flower pinned on his chest, his hands empty at his sides, as if he has not slept, turned to the viewer's RIGHT towards her (the ear we see is his bare RIGHT ear, the earbud is on the far side). Xiaoyu stands at the RIGHT in her plain black wool coat with a small white paper flower on her LEFT breast, holding the dark red notebook in her RIGHT hand. Two faces in the middle band; the pale sky and the bare branches above are the quiet area. The glass door at the back is SHUT, with nobody in it.

I had lent Grandpa's notebook to Haoran that night. He gave it back at the door of the hall where we said goodbye.那天晚上，我把外公的本子借给了浩然。第二天在告别厅门口，他还给了我。

Nine in the morning, soft white daylight from the high windows under an overcast grey late-winter sky. From behind the front row of the plain farewell hall: the backs of Mama, Xiaoyu and Papa (left to right) in black coats, a small white paper flower on each, seated. Ahead, Grandpa's large black-framed portrait on its easel among banks of white chrysanthemums. No coffin, no body, no one else. No faces seen. The high cream wall above is the quiet area.

Grandpa's portrait showed him at the stone table, laughing at a chess move.遗像上，外公坐在石桌边，正为一步棋笑着。

Same morning, soft white daylight, an overcast grey sky beyond the door. At the back of the farewell hall by the door, Teo (a plain black jacket, dark trousers, his small clear earbud in his RIGHT ear, turned to the viewer's RIGHT so the ear we see is his RIGHT ear with the earbud) and Lili (a plain black coat, her tight high ponytail with the red tie) stand side by side, quiet, hands folded. Two faces, small; the pale wall above is the quiet area. No one else. The hall door beside them is SHUT; they stand inside the hall.

Teo, from my old lab in Bridge City, and my friend Lili stood at the back. They stayed to the end.桥城老实验室的特奥，还有我的朋友丽丽，站在最后面，一直到结束才走。

Same morning. From behind and above as in 19.2, laps in the front row in black: the dark red notebook on Xiaoyu's knee under her LEFT hand; Mama at the LEFT edge of the panel, her RIGHT hand laid over Xiaoyu's (a plain black cuff only); Papa at the RIGHT edge, his big hand on Xiaoyu's RIGHT shoulder. No faces, no words. The black cloth is the quiet area.

Tuesday about three in the afternoon, soft grey light, a little warmer, an overcast pale grey sky, cold early March, no snow, bare trees except the willow's tiny green tips. At the stone table under the willow, whose hanging tips carry the first tiny green buds, a folded white cloth lies on Grandpa's round stool at the LEFT. Xiaoyu sits on Uncle Wu's old stool at the RIGHT in her plain black wool coat, the dark red notebook on the carved board under her LEFT hand. Haoran stands a little way off at the far RIGHT by the willow trunk in his plain black suit, jacket open, a small white paper flower on his chest, a plain cloth bag over his shoulder. Two faces, small; the big willow and pale sky above are the quiet area.

That afternoon the old men of the park had laid a white cloth on Grandpa's stool, as they did for Uncle Wu.那天下午，公园里的老人们在外公的凳子上铺了一块白布，就像当年给吴爷爷铺的那样。

Same overcast early-March afternoon, a pale grey sky. Close on Xiaoyu turned to the viewer's LEFT towards the empty stool off panel, a small wet smile, her black coat collar up. Her face in the middle of the panel; the pale willow and sky behind are the quiet area.

‹I came back, Grandpa.›外公，我回来了。

Same afternoon. Close on one hanging willow tip with its first tiny green buds against a plain grey sky, cold early March, no snow. No people, no words. The pale sky around it is the quiet area.

Grandpa's in here, in this speaker.外公在这里面，在这个音箱里。

Same overcast early-March afternoon, a pale grey sky. Xiaoyu at the LEFT sits on Uncle Wu's stool, her LEFT hand still on the dark red notebook, her RIGHT hand not yet reaching, looking from the small grey speaker to Haoran's face. The speaker stands on the stone table between them, unlit. Haoran at the RIGHT in his plain black suit, turned to the viewer's LEFT so we see his LEFT ear with the earbud. Two faces; the quiet willow above is the area for the caption. End on her face.

Next: Chapter 18. Echo下一章：第十八章回声

The ideas, a little slower

The lab at night, the notebook and the stone table are in the pictures. Here are the ideas in words.

On page 3 Professor Brunner describes what Swan did, and the first box says why nobody can easily tell why. A neural network is made of many simple units joined by numbers called weights, and training sets billions of them. After training we know every number, yet we cannot say in plain words what the network has learned or why it gave a particular answer. The field that tries to find out is called interpretability. It matters for every chapter so far. If we can see that a model looks at snow rather than at the wolf, or at a hospital's stamp rather than at the lungs, we can fix it before it does harm. Chapter 13 told both stories (Ribeiro and colleagues 2016, Zech and colleagues 2018).

On page 4 Swan explains itself, and the explanation sounds sensible. But a model's explanation of itself is generated in the same way as any other text, so it may not match what happened inside. Turpin and colleagues (2023) found that step-by-step explanations sometimes left out the real reason for an answer, such as a hint planted in the question. Baker and colleagues (2025, met in Chapter 16) found that written reasoning can be a useful window, but it can become less honest if a model is trained against it. Swan's "stable at the last check" is the story's version, a reason that was true at six o'clock and false by eight.

On page 5 Xiaoyu cannot read Swan one part at a time. Olah and colleagues (2020) proposed studying networks almost like biology, piece by piece, and found circuits, such as curve detectors built from simpler edge detectors. They also found polysemantic units, single units that respond to unrelated things such as cat faces and car fronts. Elhage and colleagues (2022) offered an explanation. A network has to hold more ideas than it has units, so the ideas are stored overlapping, in what they called superposition, a bit like many radio stations sharing the same airwaves.

On page 6 a second, simpler program sorts Swan's lights into features. This is called dictionary learning, done with a sparse autoencoder, a second network trained to rewrite the first one's activity as a much larger set of features, only a few active at a time. Bricken and colleagues (2023) and, independently, Cunningham and colleagues (2023) showed that the features are often far easier to understand than single units. Templeton and colleagues (2024) scaled the method up to a large model, Claude 3 Sonnet, and found up to 34 million features, including one for the Golden Gate Bridge, one for computer code with security flaws and one for flattering praise. Turning a feature up changed what the model did. With the bridge feature turned up, the model began to describe itself as the bridge, and Anthropic briefly let the public try this as "Golden Gate Claude". Gao and colleagues (2024) at OpenAI scaled similar methods to GPT-4. Real features are patterns of numbers. In the story, as in real labs, a feature's name is only a guess until it is tested. That is why Xiaoyu does not trust her guess about "asked for nothing tonight" until she has tried it in a practice copy of Swan.

On pages 14 to 16 Xiaoyu gives each person in the notebook a colour and finds that most pages need several, the way one part of a network takes part in several ideas. This is a picture, not a method. Her colours are guesses too, and she checks them page by page.

One last sentence for families. People build AI by training it, not by writing down every rule, so even its makers do not fully know how it works inside. Scientists are learning to look inside, and that helps them find and fix mistakes. In places where it matters, like hospitals, people keep checking by hand and decide whether to trust a program.

What is true here

Following a thought step by step

In 2025 researchers at Anthropic went further and traced how features connect into step-by-step "attribution graphs" inside a production model, Claude 3.5 Haiku (Lindsey and colleagues 2025). Asked for the capital of the state containing Dallas, the model seemed to represent "Texas" internally before it answered "Austin". Writing a rhyming verse, it seemed to pick the rhyming word before it wrote the line leading to it. The authors present these as case studies, not a complete account.

Why it is still hard

The first reason is scale. Large models have hundreds of billions of weights, and even millions of features capture only part of what they do. The second is faithfulness. An explanation can look convincing and still be wrong. Adebayo and colleagues (2018) found that some popular "saliency maps", which highlight the pixels that supposedly mattered, looked much the same even for networks with randomised weights, so the maps partly showed the picture and not the network's reasoning. The third is that researchers still debate what it would mean to have "understood" a network. As of 2026 looking inside gives real, specific insights and some practical tools, but nobody can yet read a large model's mind in general.

Where this began

The idea of asking what a part of a network responds to is not new. Chapter 7 told how Zeiler and Fergus (2014) and Olah, Mordvintsev and Schubert (2017) pictured what units in an image network see. Chapter 13's wolves and hospital stamp are a case of shortcut learning, which Geirhos and colleagues (2020) describe, and looking inside is one way to catch such shortcuts.

SourcesTurpin, Michael, Perez and Bowman (2023), Language models don't always say what they think, unfaithful explanations in chain-of-thought prompting.Olah, Cammarata, Schubert, Goh, Petrov and Carter (2020), Zoom in, an introduction to circuits.Elhage and colleagues (2022), Toy models of superposition.Bricken and colleagues (2023), Towards monosemanticity, decomposing language models with dictionary learning.Cunningham, Ewart, Riggs, Huben and Sharkey (2023), Sparse autoencoders find highly interpretable features in language models.Templeton and colleagues (2024), Scaling monosemanticity, extracting interpretable features from Claude 3 Sonnet.Gao and colleagues (2024), Scaling and evaluating sparse autoencoders.Lindsey, Gurnee, Ameisen and colleagues (2025), On the biology of a large language model.Adebayo and colleagues (2018), Sanity checks for saliency maps.Baker and colleagues (2025), Monitoring reasoning models for misbehavior and the risks of promoting obfuscation (met in Chapter 16).Zeiler and Fergus (2014), Visualizing and understanding convolutional networks; Olah, Mordvintsev and Schubert (2017), Feature visualization (met in Chapter 7).Ribeiro, Singh and Guestrin (2016), Why should I trust you, explaining the predictions of any classifier; Zech and colleagues (2018), Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs; Geirhos and colleagues (2020), Shortcut learning in deep neural networks (met in Chapter 13).

What is pretend

Bridge City and everyone in it are made up. So is the whole world of 2039, with its garden towers, solar boats and self-driving taxis. The Zurich lab, Professor Brunner and the four hospitals are made up. So are Swan, its answer, its parts, the feature "asked for nothing tonight", feature 4117 and the practice copy. Willow and its record are made up. Grandpa, his notebook and every page of it are made up, and so are the park's custom of a white cloth on a stool and the small grey speaker. The research has no case of a hospital program that leaves quiet patients till last. Looking inside a network for a hidden shortcut is real work, and Swan's night is the story's way of showing it. No real hospital system is described, and in real hospitals nurses and doctors decide who is seen first. Grandpa's illness is not named, and the story makes no medical claim. Many airports limit night flights, but the flight times in the story are made up.

Who made this

Yingying Fu created this manga. She chose the story and its people, directed every page, and edited every word and picture. The words were drafted with Claude, an AI by Anthropic. The pictures are made with AI image generation, painted panel by panel from her direction and checked and corrected by hand.

再慢慢讲一遍

夜里的实验室、外公的本子、公园的石桌，画里都有了。下面用文字把里面的道理讲一讲。

第三页，布伦纳教授讲了天鹅做了什么，第一个方框讲的是，为什么谁都很难说清它为什么这样做。神经网络由许多简单的单元组成，单元之间用叫作“权重”的数字相连，训练要定下几十亿个这样的数字。训练好以后，每个数字我们都知道，却没法用大白话说清网络学到了什么，为什么给出某个答案。想弄清这些的研究领域，叫“可解释性”。前面每一章都跟它有关。要是能看出模型盯着的是雪而不是狼，是医院的印章而不是肺，我们就能在它闯祸之前把它改好。这两个故事第十三章都讲过（里贝罗等，2016年；泽克等，2018年）。

第四页，天鹅解释了自己，听上去很有道理。可模型对自己的解释，和它写别的文字一样，是生成出来的，所以不一定符合它内部真正发生的事。图尔平等人（2023年）发现，模型一步步写出的解释，有时会漏掉答案的真正原因，比如问题里事先埋下的一个提示。贝克等人（2025年，第十六章提到过）发现，写出来的推理过程可以是一扇有用的窗口，可要是专门照着它去训练模型，推理就可能变得不那么老实。天鹅说的“病情稳定，上次检查正常”，就是故事里的例子，这个理由六点时是真的，到八点就不是了。

第五页，小雨没法一个零件一个零件地读懂天鹅。奥拉等人（2020年）提出，可以几乎像研究生物那样，一块一块地研究网络。他们找到了“回路”，比如由更简单的边缘检测器搭起来的曲线检测器。他们还发现了“多义单元”，也就是一个单元会对毫不相干的东西起反应，比如猫脸和汽车车头。埃尔哈奇等人（2022年）给出了一种解释。网络要装的概念比单元多，只好让概念叠在一起存放，他们把这叫作“叠加”，有点像许多电台共用同一片电波。

第六页，一个更简单的程序把天鹅的光点分成了一个个特征。这种做法叫“字典学习”，用的工具是“稀疏自编码器”，就是另训练一个网络，把原网络的活动改写成数量多得多的特征，每次只有少数几个亮着。布里肯等人（2023年）和独立开展研究的坎宁安等人（2023年）都发现，这样得到的特征往往比单个单元好懂得多。坦普尔顿等人（2024年）把这个方法用到大模型Claude 3 Sonnet上，找到了多达3400万个特征，有对应金门大桥的，有对应带安全漏洞的代码的，还有对应奉承话的。把一个特征调强，模型的行为就跟着变。金门大桥特征调强以后，模型开始把自己说成那座桥，Anthropic还短暂地开放给公众试用，叫“金门大桥Claude”。OpenAI的高等人（2024年）把类似的方法用到了GPT-4上。真实的特征是一组组数字的规律。故事里和真实的实验室里一样，一个特征的名字没验证之前，都只是猜测。所以小雨对“今晚什么都没要过”这个猜测，要先在天鹅的练习副本里试过，才肯相信。

第十四页到第十六页，小雨给本子里的每个人一种颜色，发现大多数页要用好几种，就像网络里的一个零件会参与好几个概念。这只是个比方，不是一种方法。她的颜色也是猜测，所以她一页一页地核对。

最后，给大人和孩子说几句。AI是训练出来的，不是人把每条规则都写下来，所以连造它的人，也不完全清楚它里面是怎么运转的。科学家正在学着往里看，这能帮他们发现错误、改正错误。在医院这样要紧的地方，人还是会亲手检查，由人来决定要不要相信一个程序。

哪些是真的

一步一步跟着它想

2025年，Anthropic的研究者更进一步，在一个已经正式使用的模型Claude 3.5 Haiku里，追踪特征之间怎样一步步连起来，画成“归因图”（林赛等，2025年）。问它“达拉斯所在的州，首府是哪里”，模型在回答“奥斯汀”之前，内部似乎先出现了“得克萨斯”。写押韵的诗句时，它似乎先选好了押韵的词，再写通向这个词的那一行。作者把这些当作个案研究，并不是完整的解释。

为什么还是很难

第一个原因是规模。大模型有几千亿个权重，就算找出几百万个特征，也只能说明它所做的一部分。第二个是忠实性。一个解释看起来很有说服力，却可能是错的。阿德巴约等人（2018年）发现，有些流行的“显著图”（用来标出据说起了作用的像素），换成权重随机的网络，画出来也差不多。可见这些图有一部分显示的是图片本身，而不是网络的推理。第三，研究者至今还在争论，怎样才算“弄懂”了一个网络。到2026年，往里看已经带来了真实、具体的发现和一些实用的工具，但还没有人能普遍地读懂一个大模型的“心思”。

这件事从哪里开始

问“网络的某个部分对什么有反应”，不是新想法。第七章讲过，泽勒和弗格斯（2014年）以及奥拉、莫尔德温采夫和舒伯特（2017年）怎样把图像网络里的单元“看到”的东西画出来。第十三章的狼和医院印章，是“捷径学习”的例子，盖尔霍斯等人（2020年）专门讲过这个问题。往里看，是抓住这类捷径的办法之一。

出处Turpin, Michael, Perez and Bowman (2023), Language models don't always say what they think, unfaithful explanations in chain-of-thought prompting.Olah, Cammarata, Schubert, Goh, Petrov and Carter (2020), Zoom in, an introduction to circuits.Elhage and colleagues (2022), Toy models of superposition.Bricken and colleagues (2023), Towards monosemanticity, decomposing language models with dictionary learning.Cunningham, Ewart, Riggs, Huben and Sharkey (2023), Sparse autoencoders find highly interpretable features in language models.Templeton and colleagues (2024), Scaling monosemanticity, extracting interpretable features from Claude 3 Sonnet.Gao and colleagues (2024), Scaling and evaluating sparse autoencoders.Lindsey, Gurnee, Ameisen and colleagues (2025), On the biology of a large language model.Adebayo and colleagues (2018), Sanity checks for saliency maps.Baker and colleagues (2025), Monitoring reasoning models for misbehavior and the risks of promoting obfuscation (第十六章提到过).Zeiler and Fergus (2014), Visualizing and understanding convolutional networks; Olah, Mordvintsev and Schubert (2017), Feature visualization (第七章提到过).Ribeiro, Singh and Guestrin (2016), Why should I trust you, explaining the predictions of any classifier; Zech and colleagues (2018), Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs; Geirhos and colleagues (2020), Shortcut learning in deep neural networks (第十三章提到过).

哪些是虚构的

桥城和城里所有的人都是编的，2039年的整个世界也是，包括种满花草的高楼、太阳能小船和无人驾驶出租车。苏黎世的实验室、布伦纳教授和那四家医院是编的。天鹅、它的回答、它的零件、“今晚什么都没要过”这个特征、特征4117和那个练习副本，也都是编的。小柳和它的记录是编的。外公、他的本子和本子里的每一页是编的，公园里在凳子上铺白布的老规矩和那只灰色的小音箱也是。研究文献里，并没有哪个医院程序把安静的病人排到最后。往网络里面找藏着的捷径，是真实的研究工作，天鹅这一夜，是故事用来展示这种工作的办法。故事没有描写任何真实的医院系统，在真实的医院里，先看谁、后看谁，由护士和医生决定。外公得的是什么病，故事没有说，也不做任何医学上的断言。很多机场夜里限制航班起降，不过故事里的航班时间是编的。

谁做了这本漫画

这本漫画由 Yingying Fu 创作。故事和人物由她决定，每一页由她导演，每一句话、每一幅画由她修改。文字是和 Anthropic 的人工智能 Claude 一起起草的。画面由人工智能生成，按她的要求一格一格地画，再逐张检查、修正。

When You Forget, I'll Remember你忘了，我记得