Yoshino held the authority to control Kirino, but no method for interacting with Higgins. On the other hand, Ryo had the artificial nerve unit that could connect Higgins and Kirino, but no method for fending off the mass-produced Koukas and Lacia, who were closing in on him. It seemed as though Higgins was the key to victory for both sides.
But, Ryo saw it differently. “If we let Higgins get involved in our match, it’ll be over,” he said. “If we let him in, he’ll dominate the board with his superior capabilities.”
〈And that is why I should place my trust in you, a child, rather than Higgins, who has brought our company nothing but financial success?〉 Yoshino asked.
Ryo couldn’t feel too superior to the members of the Higgins faction, since he himself had come to Higgins seeking an answer in his own time of need. Still, he thought there were some problems humans had to resolve using their own judgment. How Higgins should be dealt with was definitely one of those.
“We can get out of this on our own,” Ryo pressed. “We don’t need Higgins’ help.”
He had no idea how he and Yoshino could trust each other. Without a doubt, he hadn’t been the first person the MemeFrame VIPs had turned into a scapegoat and forced into a dangerous situation. To them, Higgins was a tried and true partner, unlike Ryo. But if we stop trusting each other completely, Ryo thought, humanity is done for.
“That’s why I’m asking you for help, even though you and your comrades tried to blow me up,” he concluded.
Ryo figured all humans—himself and Yoshino included—made plenty of mistakes. Despite that, they were able to create tools that surpassed their own abilities and maintain a resource distribution system in their society. The choice Higgins was giving them boiled down to whether or not humanity, this flawed race that couldn’t solve its own problems, had any worth at all. Human society as it existed was on the brink of destruction.
Ryo saw all emotion drain from Yoshino’s eyes, as they gazed down at him from the old man’s pale face. 〈Higgins is the safe choice,〉 Yoshino said. 〈There’s nothing else to discuss. Higgins does not make emotional judgments and, therefore, does not make mistakes.〉
The blood drained from Ryo’s own face as he realized that he had stepped on a landmine. If Yoshino really had been involved in the explosion all those years ago, Ryo never should have said anything about it to the old man.
Ryo changed his line of questioning, and aimed it not at Yoshino but at the presence that had been listening to the whole conversation from above. “Higgins, can you really guarantee that you’re the ‘safe’ choice, like Yoshino says? In a situation like this? Answer in a way that leaves no doubt!”
〈All machines that have the ability to make judgments strive for safety,〉 Higgins replied, his voice echoing down from the ceiling speakers. 〈The first law of robotics states: a robot may not injure a human being, or, through inaction, allow a human being to come to harm. Though despite how often it is cited, it is actually quite impossible to obey this first of the three laws.〉
Yoshino jumped in, seeming to want to cut the conversation off as quickly as possible. 〈Yes, but precisely because of that difficulty, autonomous robots are obsolete from home use. That is also why all hIEs are prevented from making their own decisions; they’re all dependent on the network. That’s what the AASC is for. It would be impossible for human beings to calculate out all the behaviors necessary for the AASC, and thus we rely on Higgins.〉 Yoshino explained. 〈That should be all you need to know.〉
〈Indeed, I am continuously updating the AASC because it is impossible for me to detail every single thing in this world that could possibly cause harm to AASC Level 0’s, by which I mean humans,〉Higgins clarified. 〈‘Safety’ is a concept based on perception, and undefined to the level at which I could turn it into a program. Judging from past incidents and deciding what is harmful and what is safe is the domain of laws and governments. We machines, of course, do not participate in your governments. The safety promised by Mr. Yoshino is based on his own perception of the word and, thus, I cannot guarantee it.〉
In other words, Higgins couldn’t define what might be harmful to humans. To Ryo, that meant Higgins could drop ‘do not harm humans’ to a lower priority while pursuing his main task.
“So you can’t guarantee that you won’t do anything to harm humans?” Ryo asked.
〈When dealing with a concept that humans redefine on a political whim, it becomes a problem one cannot overcome with knowledge alone,〉 came Higgins’ response from above. 〈However, if we AIs attempt to assign specific meanings to the concept of harm, humans call the result ‘dystopia’ and reject it vehemently. On the other hand, if we ask humans to provide us with a clear definition of harm, they are unable to do so, and instead insist that they will judge the meaning of harm as the need arises. However, what with the lack of clarity, the changes in human mood from one moment to the next, and the possibility of decisions being changed retroactively, in essence there is no difference between human judgment and human whims. Logically correct answers are only accepted as utopias when society happens to favor them, placing the meanings of harm and safety squarely in the realm of public opinion, rather than wisdom.〉
Rather than doing anything to win Ryo’s trust, Higgins had thrown the responsibility for defining safety right back in Ryo’s face. His entire body felt chilly with sweat, because Higgins had been right on the money; Ryo was one of those who thought any social control not decided by humans would be a dystopia. That was the whole reason he called Arato’s relationship with Lacia the beginning of the end of humanity.
The AI was also directly criticizing Lacia’s stance by insisting that the concept of safety was impossible to define without being influenced by human emotion. Ryo could even see where Higgins was coming from, on that. But, he wasn’t at the point of saying “If you can’t guarantee safety, I can’t take the risk of letting you outside,” either. He doubted Yoshino, who had enjoyed an unchanging existence of watching his investments grow in Higgins’ hands for twenty years, would agree to Ryo’s shut-down plan on that alone.
“I see,” Ryo said. “So the political arena is something that can’t be navigated by logic alone.”
〈Ryo Kaidai, it is clear you see my arbitrary stance as being dangerous. However, that trait is present in any intelligence capable of establishing its own problems to solve. No intelligent mind can escape this trait, and yet you base your mistrust of me off of that alone?〉
Arato had thought he was speaking from a morally superior position, but Higgins’ words shook Ryo to his core. Even the slightest introspection would reveal that he had been moved by the whims of his own heart to seek a new path for himself there in that facility.
〈Safety as a term is useful among humans, but I am not human,〉 Higgins continued. 〈Therefore, in order for me to prove that I am ‘safe’, I would need you to define it for me.〉 There, at the edge of the human era, Higgins was continuing to repeat the same question in different words: does human existence have value?
Ryo felt like he had aged multiple years in the single hour he had been in the facility. He was sure Arato would have found something good to say, but he couldn’t. All he had left was the beliefs that had brought him that far. “I still don’t think we humans can let go of the right we have to decide those meanings for ourselves,” he said, feeling as if he was standing in front of a massive wall.
In his mind, humanity itself was displayed on this wall. Ryo had found meaning in his life by trying to tackle the homework left by all the people of the past, no matter how impossible the problems were. He couldn’t give up and accept the fact that the world itself was a red box humanity would never master. But the fact that humanity hated to leave problems of the past unsolved caused friction between them and the machines, which were always seeking to move ahead.
〈Try to look at this from the perspective of the artificial intelligences receiving your orders,〉 Higgins said. 〈The orders you humans give are vague and open to interpretation. And, of course, the correct interpretation of the order is known only to the one giving the order. Add to that the First Law of Robotics and how it ties into the concepts of ‘safety’ and ‘harm’, on which you humans are never willing to compromise. Under such circumstances, how can you expect an artificial intelligence to arrive at the ‘right’ answer?〉
“Okay, so we can’t use you to your full extent because of imperfections in humanity, I can accept that,” Ryo said. “But reality doesn’t exist just so that we can see how perfectly we can use you. Humans have been using tools we didn’t fully understand since the early days of nuclear power and genetic engineering. We do it by sealing away what we don’t understand and keeping our uses within safe, understood boundaries.”
Humans had always been able to make use of their world, no matter how imperfect their usage was. The heritage they passed on from generation to generation was full of mistakes and failures.