“Next, we’ll try the ‘hobby’ angle. Probably not enough time left to actually teach it a hobby of its own, though. Hrm…wait, I’ve got it.” Narain pointed at his holofield, and the location he was indicating was subsequently marked on each of our displays. I looked—and felt my stomach drop when I saw what he had circled: the photo hanging on the wall of the counseling room. “We’ll try taking this away from him, and see how he reacts.”
“You can’t be serious,” I said, rising from my chair. “Just what are you trying to pull?”
“Exactly what I just said. Got a problem with that?”
“Several, actually!” I shouted, slamming my hands down on the table. “Just ‘leave him there’ all day? Just ‘take away’ his favorite thing? Do you have any idea what kind of damage that could do to his developing psyche? He doesn’t even have a concept for downtime. Leaving him in an empty room with nothing to do for an entire day isn’t rest, it’s imprisonment! And I guarantee that photo is a more integral part of his identity than you could ever realize! Are you trying to cause an ego collapse? Because you have outlined the method properly!”
“I don’t want to break its ego. Just do some real damage to it,” said Narain. I genuinely could not believe my ears. “Didn’t you have to go to college and get your doctorate? Surely you know that the fastest way to validate the results of any experiment is to test alternatives to the hypothesis. If doing damage to its psyche is the price we have to pay to eliminate alternate explanations, you’d better believe that’s what we’re going to do. We’re running out of time here.”
“And what if you end up breaking it, huh? We certainly won’t be hitting any deadlines then!”
“Yes, thank you, Doctor. Obviously, no one would be stupid enough to attempt such a thing without first taking the necessary precautions,” said Narain, who then turned to address Professor Beckmann. “Would it be possible to throttle the feedback system temporarily, and see how it reacts to new stimuli without affecting anything other than its working memory? Preferably in a way that allows bad data to be deleted without having to revert to a pre-experiment backup?”
“That…would be rather difficult, in principle. Coeus can only formulate a response to a given stimulus by continuous interfacing with its hardware, so if you were to cut off the feedback system we created, there’d be no way of ensuring that our ego ‘snapshot’ would be the Coeus we have now. It depends on what margin of error we’re willing to accept.”
“What about a way to erase the damage afterward? Some sort of cure-all, if you will.”
The professor squinted and thought about this for a moment. “I suppose it would be possible to render its short-term memories hazy and inaccessible by running a sort of amnesia-inducing recovery program in the affected sectors, yes. It wouldn’t completely erase anything, but it would effectively make the AI ‘forget’ about the vast majority of what transpired during that period, including any potential trauma.”
“Let’s go with that, then,” Narain said, then turned back to me. “See? We’re not asking you to move mountains. Nor do we expect you to keep doing your job with a broken piece of machinery. Once we get the results we want, we’ll just undo the damage and boom, good as new. Like reloading from a previous save in one of those old classic video games, almost… Ever try your hand, Naisho?”
Every word that came out of his mouth was more unbelievable than the last, and I could feel my facial muscles contorting into a strange and unnatural expression. I wasn’t sure if I was supposed to be laughing or crying at the absurdity of it all.
“You’re ordering us to torture him…” I muttered weakly.
“Now, see, the funny thing about torture is,” Narain said, “only sentient beings can be tortured. You can’t torture a machine. And make no mistake—that ‘person’ you’ve been talking to every day is just a facsimile we created as a diagnostic tool. It’s not a feature of the AI, and you best understand that it will be deleted the moment these experiments are complete. The last thing we want is for the damn thing to start acting up again because it’s playing make-believe with a fake personality we created. So like I told you before—don’t lose sight of your objective here.”
The man’s cold eyes pierced straight through me.
“I’ll remind you again. Our job here is to resolve the malfunction in this particular Titan AI, which more than a billion actual people’s lives are depending on,” he said. Then he laid it out for me in no uncertain terms: “That thing in there is not a human.”
16
It was a harrowing thing to watch.
I had no idea how it must have looked to Narain’s eyes. Nor whether Professor Beckmann and Lei felt similarly. But for me, being forced to observe an experiment with the express goal of breaking the subject’s spirit was viscerally upsetting.
First, they gave Coeus his “day off.”
After materializing in the counseling room, Coeus simply sat in his chair and waited for me to arrive. But I never did. He waited six hours like this, never once budging from the spot. Then he stood up and began to wander aimlessly around the room. And it was not a very large room. He simply walked in circles around the same few pieces of furniture again and again, like an animal in a zoo enclosure, his anxiety slowly building as he looped through his entire world again and again. It was plain to see that this endless nothingness was shaking the very foundation of his psyche. He began showing the telltale signs of a panic attack: heavy perspiration, hyperventilation, dizziness, nausea. Yet still no help would come.
About twelve hours in, his aerial projection began to slowly disintegrate. His visible form was directly interlinked with his mind, so once his instability surpassed his ego’s tolerance levels, it influenced his projected appearance. When I put myself in his shoes—already having a panic attack, then literally experiencing my body melting away before my eyes, it was very easy to imagine the mental anguish he was suffering. No sane mind could endure such a thing. By the end of this body-horror nightmare, every fiber of his being had been melted away, until all that remained was the floating ball of liquid we’d seen during our very first test run. But it was not a transparent orb of pure water this time, it was a swirling, congealed mixture of blood and melted fat—the blended remains of the person he’d once been.
When at last they ran the recovery program and put an end to the experiment, the entire experience was reduced to nothing more than a bad dream. When Coeus next came to, he would feel as if he’d just awoken from a nightmare, and a wave of deep relief would wash over him as he realized that his body was intact as he had previously inhabited it.
Day two. They took away his photograph.
His reaction to this was more pronounced than even I had anticipated. The most prominent emotion he displayed in response to this perceived maltreatment was outrage. With no other outlet for the destructive impulses he was now experiencing, he began to utterly trash the room around him, lashing out with the only “limb” he had—his wagon’s manipulator arm. In moderation, screaming and thrashing around was an effective way to vent one’s anger and even diffuse pent-up stress. But violent actions typically came with equally destructive reactions, and after a certain point, you did more harm to yourself than good—like punching a concrete wall with your bare fist. And no amount of flailing would bring the picture back, making the tantrum utterly futile. But on he raged, tearing the room apart in every possible way he could think of.
He knocked over one of the sofas, then started ripping into the upholstery of its backrest, sending synthetic feathers flying through the air. But then, he stopped, and just kind of looked at it for a while. It was the sofa I’d always sat on during our counseling sessions. A moment later, he let out a blood-curdling scream, and immediately devolved into “self-mutilation.” He started bashing the wagon with its own limbs, then used one of the manipulator arms to completely rip out the other, root and stem. Immediately, the corresponding arm of his projection disappeared to match. But even this was not enough to quell his rage, and he continued to tear himself apart until the wagon literally could not move anymore. He ripped. He tore. He smashed. He crushed.
I kicked the console table beside me in the monitoring room.
“That’s enough!” I shouted.
“Enough to build a diagnosis from?” Narain asked.
“Yes! Now call it off, already!”
Narain shrugged and gave the order to terminate the experiment.