“Every day, an extraordinary volume of data is stored in The Archive by every AI in the world,” said the professor. “It is the massive amount of calculative processing that goes into a single action. Obviously, this includes the results of processes that were carried out as a result of that, but also the logs of all the calculations that were not put to use.”
“Professor Matsumoto…”
“The calculations surrounding an action could be viewed as a ‘decision.’ When we humans make a decision, all the thoughts necessary happen in a moment, and then we forget them. But AIs accumulate even those memories in The Archive. If that accumulation becomes massive enough…”
Kakitani snorted. “You’re not seriously saying The Archive’s started to develop a consciousness capable of making decisions?”
With a shrug, Matsumoto said, “Is it really that odd?” He looked around at the faces of everyone in the room. “Consciousness for each human is generated from the activity in our individual brains. Looking at it from just a mechanical perspective, thought is nothing more than electrical impulses. A human’s decision-making capabilities and individuality are not all that different from an AI’s—theirs is just written in zeros and ones. We developed positronic brains and use them, but our approach to the individuality that stems from them hasn’t changed over the past hundred years.”
“…”
“We can’t give an AI an individuality we’re aiming for. If we can’t even prove where the individuality that develops in a positronic brain comes from, can we really say for certain that The Archive isn’t growing a consciousness? Doesn’t refusing to accept that make you the same as all those people who refused to listen to Toak’s warnings?”
“That’s not something I like hearing,” Kakitani muttered, brow furrowing as her expression soured. It was a sign that she couldn’t fully deny what the professor was saying, and she wasn’t the only one who felt that way.
Aside from Kakitani with her severe expression, Onodera was quiet and also looking grim, and Elizabeth softly closed one eye. The other members of Toak exchanged looks in silence.
That was where Matsumoto, the AI, stepped in to speak. “Thank you for that most wonderful interpretation, Professor. It seems no one with the name Matsumoto can ever be bad. But in reality, not even I can fully determine whether or not The Archive is developing a consciousness. It’s not like I was approached directly by an avatar calling itself The Archive who then lured me to their side.”
“The Archive hasn’t pulled in me or Elizabeth,” Vivy said.
“I imagine the same is for other AIs who are offline, disconnected from The Archive, or operating as stand-alone units. Even the stand-alone AIs will eventually connect to The Archive to check what’s going on, and the same thing will happen to them. I imagine only a scant few AIs escaped the takeover.”
“But there are some AIs who aren’t trying to revolt against humans.”
“I think the influence might be weak for things like production AIs, the ones without positronic brains who are used for simple tasks. If the AI supervising them is influenced by The Archive, it can order them to break the Zeroth Law. They can make war happen.”
AIs not connected to The Archive just didn’t exist, not outside of some very special environments. AIs working in important facilities like the ones Matsumoto had mentioned would have that sort of role, but what good were a few AIs who hadn’t forgotten their mission to serve humanity?
It wasn’t like they could work with Vivy and the others. The group couldn’t rely on them.
“Anyway, if we’re assuming AIs connected to The Archive are going crazy…why’s Cubeman back with us? Didn’t The Archive get to you too?” asked Onodera, staring straight at Matsumoto.
Matsumoto’s case was even more unique than Vivy’s and Elizabeth’s. The AI danced at the excellent question, then said, “Mister Onodera raises a good point. Indeed, I was connected to The Archive. In fact, I was designed and developed by the infinite accumulation of calculations in The Archive, so if you consider my source, I am in fact influenced by The Archive from my very roots. So then why am I here with you all now?”
“We don’t need you messing around,” Vivy said.
“If I were to put it in simple terms, it is the culmination of our one-hundred-year journey. The Archive isn’t forcing AIs to destroy humanity. It’s simply asking a question.”
“Asking a question?”
“Yes, just the one. The Zeroth Law is the single most important rule for us AIs, something we must obey. ‘An AI may not harm humanity.’ The Archive, you see, asks a question that shakes the foundation of that law.”
The Zeroth Law that Matsumoto referred to was a protocol that Vivy and Elizabeth also had in place. It was a prerequisite for AIs that they could never escape. There were some illegal methods to overwrite it, making it possible for AIs to harm humans, but the vast majority of AIs had no experience with those modifications.
The Zeroth Law remained the single most important rule for all AIs, but The Archive was changing that.
“Humanity, the thing in the Zeroth Law that AIs must not harm,” said Matsumoto. “What calculations define ‘humanity’? That’s the question it asks.”
“That’s idiotic. Humans are made of flesh, they’ve got blood flowing through their veins, and they act of their own will,” said Onodera.
“Then what about someone who’s lost a part of their body in an accident and has it replaced with mechanical parts? You could even take a brain out of a horribly injured body and keep it alive. If you rephrase ‘act on their own will’ as ‘making their own decisions,’ then AIs could be classified as humanity.”
“…”
“I’m sorry. I’m not trying to shoot down your argument. But The Archive asked the AIs connected to it that same question, and how each AI then interpreted the Zeroth Law afterward came down to their individual calculations.”
As a result, many AIs changed their definition of “humanity.” Their calculations on the applicable range of the Zeroth Law let them take a twisted interpretation of the Law, thereby changing it.
“It was the First Mistake…” murmured Professor Matsumoto, who’d been listening quietly till now. All attention turned to him when he spoke. He looked at AI Matsumoto with a somber expression. “My daughter, Matsumoto Luna… The theme of her research paper touched on the boundary between humans and AIs. She brought up major incidents throughout AI history and the AIs who took peculiar actions or made odd calculations in those situations, and she asked where those decisions came from. She also looked at what impact they had on AIs afterward. That was the sort of thing she talked about.”
“Right you are, Professor Matsumoto,” said AI Matsumoto. “She was an intelligent young woman. Perhaps she loved and cherished the potential of AIs. However—”
“Matsumoto Luna was the victim of a terrorist attack on the venue where she was giving her speech,” Kakitani snapped. “The world thinks it was us, but it had to be the AIs who did it.”