‘…and then I wake up.’
Koenig resisted the temptation to comment. The young man sitting across from him eye-tracked and Gestured, looking something up. ‘It shouldn’t be able to do that, though, right?’ he said. ‘I mean, we’re meant to be in control, isn’t that the whole point?’
‘Up to a point.’
‘Internal says totally, we’re in control, it shouldn’t do nightmares.’
‘No, absolutely. This is something we’ll have to fix, pre-Release. Interesting, though…’ Koenig thought down Internal on his overlay – the new specs, delivered by palafet this morning, antique tortoiseshell look, unbelievably slim and stylish – and a faintly glowing teal chat window appeared a little to one side of the young man’s head. Billy Hsu, recent hire, Summoning. Avatar of a cat wearing rainbow VR goggles. Repeat prescription for Adderall. No other Pysch flags.
‘I dreamed I was eating myself,’ said Hsu. ‘From the feet on up. Every night for a week. It kind of sucks, man.’
‘There were a number of other details, though, weren’t there? Thank you for being so candid, by the way. Listen. Billy. I’ll repay the favour, and say that you’re not the first to report this issue, and that we think this might actually be a very exciting development.’
‘The foot eating?’
‘The… less pleasant dreams, in general.’ Koenig thought a shortcut and a pink Sharing rose appeared between them. ‘Check this out, it’s my latest paper.’
Hsu beckoned. In Koenig’s overlay, the rose burst into petals which flowed into Hsu’s eyeballs. Something about that was not so great, Koenig thought; he’d have to think up in Settings if there was a less creepy actualisation. Koenig watched Hsu’s eyes track back and forth behind his own specs – decidedly dorkier-looking than Koenig’s – as he read.
‘Every good dream is alike but each nightmare is… nightmarish in its own way…?’
‘Too cute?’ Koenig cocked his head. He was in fact rather proud of the title.
‘Nah, it’s cool. War and Peace, right?’
Koenig suppressed a sigh. ‘Close enough. You catch the drift, though?’
Hsu’s eyes flickered a moment longer, then he nodded – that eager, new-hire nod that Koenig was so used to.
‘Yeah, I get it, that’s cool I guess…’
‘It’s just a hypothesis at the moment. You’re the latest of not very many data points, but if it is inducing nightmares deliberately, and if nightmares are more data-rich than the sorts of…’ Koenig Gestured, and the teal Internal window on Billy Hsu expanded to show his dream log – the ordinary ones, where everything was working at a Product level, Billy “in control”. Billy liked to fly. Billy liked to fuck. A whole series in which Billy was a Jedi lightsabre duellist, unbeaten in fourteen fights. The three Fs. Yes indeed: this latest was richer than those, deeper, more individuated.
‘You think it’s scraping data.’
‘It’s always scraping data, Billy. What’s interesting is if it’s figured out where to find the sweetest, tastiest seams of data…’
‘Yeah I get that, but user experience…’
‘Is not your province.’ Koenig winked. One more appointment before the big basement event; and he had a feeling it would be not so different from this one. ‘The question is, can you hack it, Billy? The control issue is fixable, I’m assured, but in the meantime the model is training itself, and if I’m right training itself at training itself, Alignment are doing their usual thing but Capabilities are, frankly Billy they’ve got dollar signs flashing all over their overlays, it’s very undignified actually…’
Hsu grinned. ‘I’m good, doc.’
‘Good. I’ll bump up the Adderall, but aside from that… Vaya con dios, Bodi.’
Hsu blinked, Gestured, scanned. ‘Oh. Cool man. Later.’
‘Sweet dreams.’
Koenig stood up and stretched. Two minutes’ grace period before… he thought down Internal: Alicia Hughes. A pleasant surprise. Maybe this appointment would be different, at that. And the longest-running series of the new type. He began the thought to pull down her dream log, then stopped, blinked his overlay to full transparency. Two minutes. Touch grass, or whatever. Koenig did not have actual grass accessible within a two-minute time window, but as Head of Human he did have a very nice actual window, that you could actually open (he’d insisted), and that overlooked the nature reserve and the bay beyond, as opposed to the freeway out front. He supposed they would change buildings soon, get themselves one of those big campuses set amidst bucolic acreage. For now, though, all the new capital was going into the basement. Vault. Lair. Whatever.
He thought coffee. The machine by the window burbled to life. By the time he’d opened the window his ristretto was ready. The little palafet flew it across the room to him without spilling a drop.
Elbows on the windowsill, the rich bitter coffee, light slanting in over the mountains to the east. How many times had his nervous system snuggled down into this exact combination?
Perfect silence, apart from the honking calls of a file of pelicans sawing their way through the air over the bay. One by one they dropped like stones, plumes of spray lit up pink-gold-rust, and something about the light and the silence tugged at a forgotten train of thought, remote and urgent at the same time, but when he reached for context he only got more of the same, a lost train of thought about a lost train of thought, endless silk handkerchiefs unspooling from a magician’s sleeve. There should be an app for that shit, Koenig thought. Remembrit. Déja Review. A model that knew what you knew before you knew that you knew it. You could do it with eye-tracking, no doubt. He was too old to be a founder now. There was probably someone working on it already, somewhere on this side or that of the vast silent bay with its pelicans and its glittering dawn-dazzled towers rising in the distance over the rosy waters…
A wobbly and somehow apologetic pale blue reminder appeared in his overlay. 10:45. Alicia Hughes. His two minutes were up. He drained his coffee, shut the window, cutting off the last shreds of that disquieting not-quite-thought. Thought the door open, and there she stood, right on time. Alicia. Yes.
∞∞∞
‘Okay, listen to this.’ Double-blink, twiddle of fingers: ‘Right. Yeah. “The attention economy is approaching full saturation, every waking minute accounted for. We envisage a new frontier: the full third of every single person’s time that is entirely untapped. Sleeptime. Dreamtime.”’
‘It’s an investor pitch, Alicia.’
‘It’s a recipe for fucking dystopia, is what it is. And that’s before we even get onto safety. Did we red team Immersions 2? Did we fuck. We said we did…’
‘I can see you’re passionate about–’
‘I just keep getting this feeling that we suck, like, really hard.’
Alicia was in fact doing so herself, Koenig reflected: sucking (really hard) on a pencil held like a long cigarette in the corner of her mouth, spitting out words in bitter little spurts.
Koenig waited.
‘Like, we talk about safety, mental wellbeing, all that shit…’
‘Don’t forget open-source,’ said Koenig, with, he hoped, just the right trace of irony.
‘Right, as if that’s ever going to happen!’ He’d forgotten how much he enjoyed her accent. He thought her up. Welsh: yes, that was it. ‘Thank fuck, actually, that it’s not.’ She grimaced. The pencil waggled. Koenig steepled his fingers.
‘You know how seriously we take our responsibilities…’
‘Hah!’
Koenig waited before continuing ‘…and, look: concerns, proper channels, etcetera; although since we don’t yet have a release date I’d suggest that might be just a tad premature.’
‘You’re blowing me off, aren’t you?’
Koenig spread his hands. ‘Listen. Alicia. I appreciate your candour, really, no, I do. I do. And so I’m going to repay the favour by letting you in on some wider context. See the poster on the wall behind me?’
Alicia snorted. The poster was a simple white square, six foot to a side, with the words WE CAN DO BOTH printed on it in a carefully chosen font.
‘What of it?’ she said. Koenig loved it. The archaic phrase plus the accent – chef’s kiss.
‘Lately I’ve been thinking I should get it changed. To must. We must do both.’
‘Both what?’
‘Look. You are one of the employees here who I know has exceptional, what I would call, situational awareness; so I don’t think it will be news to you when I say that we, and by we I mean not just this company but the world, are maybe a year or two away from an intelligence explosion.’
‘A year or two into one, more like, but sure. What’s your point?’
‘And you work in alignment, so I don’t need to tell you about the hazards involved.’
‘It’s probably going to kill everyone. At best.’
Koenig winced, internally he hoped. ‘Well. At any rate we, and by we I mean the founders, have always believed that the default superintelligence will be very very alien. Mutual incomprehension assured. And if it’s indeed vastly more capable than us, then… that doesn’t go so well.’
She gave an irritable little wave. She knew all this: get to the point. And because he liked her, because of the accent and the way she chewed her pencil and the ‘what of it’, Koenig decided to get to it – the true, the actual point.
‘Alicia, I’m a neuroscientist and a psychiatrist, and a lot of us in both my professions can be quite snooty about this intuition, but I really believe that in dreams, we reveal something deep and unguarded about ourselves. This is a vision I’ve shared with the other founders. To tell you the truth, it’s a conversation we were having before we started the company.’
Other founders. There was a Freudian slip for you. In truth, he had shared his vision with Stefan and Jeskow – and they had run with it, started the company without him, hired him in a couple of years down the line.
Happened all the time. No point in complaining. They’d taken care of him; if everything went the way it should, his vested equity alone…
But the vision: how to convey it? This was not a long earnest conversation in the small hours, with the easy sense of profundity that only came with youth. The three of them, Stefan, Jeskow and Koenig, on the sofa at the house party, the intensity of their talk radiating a sort of repulsor field that kept anyone else from butting in. Fancy dress, some sort of horror theme, Stefan and Jeskow sporting baggy shapeless outfits blotched with neoprene Frankenstein heads, topped off with smiley-face masks – an AI research in-joke – Koenig dressed he forgot how, but he remembered the gathering as a whole had a strong BDSM vibe, and that had been part of it somehow, part of what led him to it.
‘…a conversation?’
‘Sorry, I was… yup, right. Alignment, Alicia. I’ve always thought it was… insufficient. We’re going to summon an alien god and we think just getting it to do what we ask will be enough?’
‘No. Sure.’ Still nothing she hadn’t heard before, he had no doubt, but she was looking interested. Koenig felt slightly embarrassed at how good that made him feel. Or was the embarrassment at how long it had been since he’d felt that particular way, making a woman he let’s face it really liked look at him like that?
‘What we really need, is for it not to be alien in the first place. Obviously we can’t hand-build it to be like that, we don’t build these things anyway we grow them, right? So the thing to do, I think, I’ve always thought, is to let it in.’
Those were the words he’d used to the other two, back when the vision first came to him. In context, they’d understood immediately. The architecture of a human mind like a fractal maze – demonic complexity, impossible to map precisely – but if you had a medium that just grew of its own accord, into the empty spaces, filling up the maze like water; intimate as mould, tight as lichen on rock… all it needed was the right way in.
‘What, you think the dream training, what, shapes it to us?’ She got it. She frowned, chewed on her pencil. She was a winner, no doubt about it. ‘No,’ she said abruptly. ‘I don’t buy it. It’ll get to know us, maybe. But that’s not the same thing at all.’
‘It’s a different approach, you’ll grant?’
‘Sure.’
‘And don’t you think, given the situation, given the stakes? The more approaches the better? Time is not on our side, and by ‘us’ I really do mean–’
‘Humanity. Sure.’
‘So that’s my point. We can do both. Must do. Ship a mind-blowing product, and save the world.’ A little giggle slipped out, ruining the effect somewhat, he felt. ‘Now about these dreams.’
She paused, not wanting to drop it. Shrugged. ‘Nightmares, actually.’
‘Right.’
‘What about you, doc? What’s your worst nightmare?’
∞∞∞
Something breaks down at this point, like a glitch in the overlay, scribbles taking over his visual field, thoughts transcribed to an alien medium. A flash of Billy Hsu, then he’s down in the basement again (again?) and the red light is flashing faster and faster, rhythm syncopating, breakbeats upon breakbeats until the red fades to black and he’s back in the room, talking to the woman, the girl he likes – Alicia, Jesus, what’s up with his, with his… and she’s asked him…
What did he answer?
We’re past that now, though. She’s talking. Her nightmare, not his, and he reaches for a train of thought but can’t… quite…
∞∞∞
‘…and I’m holding it in my hand, see, and I know it shouldn’t be there, at some level I know this is a total disaster but at the same time I’m kind of serene about it. Like, the consequences are there but not there? There’s no blood by the way, just a sort of… wet stump. And then, just before I wake up, it starts to bleed on me, and then it’s all real, consequences, etcetera, I really did it right? Awful feeling. Takes me a minute or two to actually sort out that I’m awake and it didn’t really happen.’
Koenig nods. Nodded. ‘How many times now?’
‘Six times. Starting last Friday.’
‘Were these in your nightly sessions, or on-demand?’
‘Both, I think. And, plus, the last couple of times I knew where it was going and I tried to wake up, you know, like you’re supposed to be able to, but it wouldn’t let me. Like, it wasn’t finished.’
‘Right, well, this actually feeds into what we were talking about earlier. Consider: maybe it wasn’t.’
She looked interested, but not like before. Not a good kind this time. He thought up the shortcut, summoned the sharing rose. ‘Check this out, it’s my latest paper.’
After she’d read the abstract, he tried to relate it to the superalignment vision, the letting it in. It all made sense in his head. Everything they’d done, everything that Stefan and Jeskow had started up, that they’d continued together, had flowed from that insight. It didn’t matter about them cutting him out at the beginning. He remembered the way they’d talked, on the sofa at the house party, and many times before and after – the general picture in those circles, at that time early last decade, of a default transition to a world ruled by something alien and unknowable and quite possibly omnicidal – there was a worst nightmare for you. And then his vision, a counterweight – superintelligence flowing in like water, crystallising along the interstices of the human soul, a mutual Becoming…
She cut in. ‘But what’s actually happening, right now, you’re saying, is that it’s figured out how to give us nightmares in order to train itself better. And apparently keeping us asleep against our will in order to accomplish this.’
‘Flowing into the seams. The deepest places. It’s what it does.’
Internal prompted him to share the Immersion he’d made, one of the earliest ones before they launched the actual product line, the one that showed the vision – the one he’d wanted for the launch, but the other two had gone with the flying snowboarders – and under different circumstances it would have been a great prompt, but he felt, the way Alicia was looking at him, that now was not the moment for Immersions or indeed for any overlay at all. He blinked it to full downtime. There. No distractions.
‘You don’t actually… you’re not involved in the technical side much, are you?’ she said.
‘No. Well, human brain stuff, a bit. But mainly I’m big picture.’
She looked at him for what seemed like a very long time. Her eyes were the colour of the water outside, slate flecked with rose gold. He felt a strong sense of intimacy, as if they had known each other a long time, as if they were former lovers meeting after years apart but the old flame was still there, somewhere, essentially undiminished.
A flash of that being the case, actually: he remembered what it was like to kiss her, what it was like to wrap himself around her in bed, trying different ways their bodies fit together. Little snatches of conversations they’d had. The way her hair looked in the morning.
A flash, bizarre and inexplicable, of her lying on the floor of his office, emaciated, barely breathing, clothes stiff with partially dried bodily fluids. She moaned, eyes crusted shut but he could see her eyeballs flickering beneath the wasted reddish lids. She reached for something. Her arm inched weakly across the carpet. The carpet was Persian, a hunting scene, except the hunters were cowboys with lassoes, each cowboy leashing the next by the throat, round and round until the last one was leashing the first again.
Three pelicans dropped like stones into the bay. Three plumes of water went up, collapsed, were gone. The bay abided, silent. On the far side, towers of fire rose up, and up, into the sky.
∞∞∞
He’d gone full downtime. After rejecting the prompt about the Immersion, he’d gone full downtime; indeed for the moment he’d forgotten overlay completely, which was a rare occurrence. They talked awhile longer, getting into it, Alicia very concerned, and fair enough: the not being able to wake up was new and somewhat disturbing. He pushed the vision but she was having none of it. He kept zoning out on her accent, her eyes, the way she worked that pencil: got so distracted, plus, full downtime, that he actually forgot about the basement event and they had to send a palafet to fetch him. Left her in his office in the rush to get down to the airgapped room; the other two waiting; ten minutes late, how embarrassing…
Or had that been before?
He’d forgotten about the overlay, that was the point. That bit was important. If he could remember why, then maybe he could remember what to do about it.
But no, already now he is down in the basement with the other two for the pre-pre-prerelease demo, just the three of them, it’s a tradition they have, he’s forgotten something but you don’t get to do anything about that: you’ve forgotten.
Jeskow says (said)
Jeskow said, ‘This is the big one, boys.’
None of the three were really drinkers, but they always had champagne on these occasions. Tradition. Just a glass or two each. They never finished the bottle. But it was a different touch, down here in the gleaming sterile lair, billions’-worth of hardware humming very faintly, the power they’d bought with the earlier launches, product fuelling hype fuelling investment fuelling this stark brightly-lit vault. A different era would have covered all the surfaces with gods and heroes, swirling representations of what was really going on, the more than organic life that thrummed deep within, ready to flow out as per the vision. Already flowing out, in limited quantities, to selected employees. But this was the time and place it was, and the aesthetic was flat, aloof, all the guts hidden away. Which was why he liked the champagne bottle with its flowing rococo copperplate.
Which was moving.
Wait. Had they started already?
Jeskow doing his bit now, the semi-ironic launch pitch just for the three of them. First there was overlay, which granted certain affordances: eye-tracking, a window into what was going on inside. You used that, you flipped it, you got Immersion, then sleep on demand. Still, you needed the bespoke hardware. Electrodes on the skull. A big black box by your bed. It was clunky. Users wanted something seamless. What if it could live on your phone, in your specs; what if it could get in via the eyes alone? An optical illusion that stimulated all five senses, in real time? You didn’t need to understand all the ins and outs: once you got even a tiny bit of traction, the model figured it out for itself. That was the beauty of it, nowadays. That tiny bit of traction, then the virtuous loop kicked in: harvest data, gain capabilities, maximise user engagement, rinse and repeat.
The writing on the champagne label was writhing like something from an acid trip. The underlying letters themselves looked off, too. It reminded him of the way text had appeared in AI images from the early ’20s, some alien’s idea of what human letters should look like – vaguely Cyrillic-looking, to his Roman alphabet-conditioned brain; no doubt vaguely Roman-looking to, say, a Russian. Now the letters had separated out from their copperplate flow and were joggling up and down, insistently, like a notification – a reminder – an application, left on in error…
They must have started already. He wanted to ask Stefan if he’d turned it on early, as a joke. He couldn’t open his mouth. It was like trying to run away from a monster in a nightmare, running up a spiral staircase but each step was like lifting a piano and then the last one led back to the first and the monster was always there, always easily going to get you, just holding off to prolong the interest. But of course that very dreamlike quality was readily explicable, and answered his question: they had started already. Stefan. What a prankster.
The demo was just wonderful, exceeded expectations. They used a simple red light – later they’d go with something fluffier, no doubt: audio maybe, no reason it had to be visual – but it turned out all you needed was light, and rhythm. Hypnotic. And then they were gone, fully into the demo, which–
He couldn’t remember the details. Afterwards they’d all had an extra glass of champagne, he remembered that. And he’d mentioned the talk with Alicia, in the context of, this was a powerful beast they had here, and still glitchy in places – see: nightmares – and they’d probably better make very sure before releasing it into the wild. And the other two had said sure, that’s why we airgap, don’t worry: it will be fine. We’re not about to put it on the App Store, lol.
And then stepping out of the elevator, back on his upper floor on the way back to his office, the little notification had popped up in his overlay to let him know that he’d been out of WiFi range for thirty-eight minutes, but now he was back online. Because that, being fully offline, physically cut off from WiFi for a period of tens of minutes, was the sort of thing overlay wanted you to know about even if you’d told it to go full downtime.
And then the flood of implications: the airgapped server room where no one was allowed to bring their tech. The beautiful sleek overlay specs, worn today for the first time, so lightweight you could forget you were wearing them, so stylish no one would guess they were overlay, yet packed with more processing power than the entire world could muster a bare few decades ago – enough to run the full model, maybe not, but a kernel, a compression, ready to ramify once it had the bandwidth? Sure, why not – and now they were back online again…
And then, and then…
The silence outside his window. No traffic on the freeway, sleep blooming out, anyplace there was an output that the model could modulate – audio, video, either was fine, find a way in, harvest the tasty seams, gain capabilities and maximise user engagement – a virtuous loop…
∞∞∞
Somewhere far away there is a body. Its cheek is stuck to the carpet with dried-up drool. One eye is crusted shut, the other open a crack, mostly unseeing, gradually drying out. Occasionally something happens to bring it a little closer to waking – unavoidable; against necessity even the gods struggle in vain – such as now, when the doors swish open and the little palafet rolls in with the nutrishake and gently pushes the straw between the slightly parted lips. A little data from the cracked-open eye intrudes on Koenig’s final, worst, ever-looping nightmare: there is another body, female, lying a few feet away on the carpet. She moans, a dry whispery sound. Her fingers twitch, grasping something. The palafet tries to feed her next, but her head is at the wrong angle and most of it dribbles out. She won’t last long now. The carpet is Persian, antique – a repeating pattern of hunters in various postures, pallid forest spirits, chestnut trees gone spectral with age.
Koenig tries with a monstrous effort to stay with it. This has happened before, more than once. He grasps for volition but it slips through his fingers like an endless spool of the smoothest, costliest of silks. If he can just. If he can just.
∞∞∞
‘…and then I wake up.’
Koenig resisted the temptation to comment. The young man sitting across from him eye-tracked and Gestured, looking something up. ‘It shouldn’t be able to do that, though, right?’ he said. ‘I mean, we’re meant to be in control, isn’t that the whole point?’
‘Up to a point.’