Fault Lines

Will A.I. Overwrite Our Sense of Shame?

人工智能会覆写我们的羞耻感吗?
The tech world has long been run by iconoclasts whose ideas about right and wrong are self-consciously different from those of most people.
科技界长期以来由一群反传统者主导,他们对于是非对错的看法,有意与大多数人不同。
Illustration of a robot on a screen with an apple a leaf and a snake.
Illustration by George Wylesol

This is part of a series of columns about the decline of shame in the age of social media.

本文是“社交媒体时代羞耻感的衰落”系列专栏的一部分。

In the first installment of this series, I wrote about shame as one way that groups of people can mediate and enforce social norms without explicit state intervention. But there is an emergent Silicon Valley gospel that has challenged both the boundaries and the mechanisms of shame in America. What happens when millions of humans across the globe spend hours a day talking to a robot? Who taught that robot right from wrong? Who made it feel shame?

在本系列的第一篇中,我写到羞耻感是群体在无需国家明确干预的情况下,调解和执行社会规范的一种方式。但硅谷正在兴起的一套信条,同时挑战了美国社会中羞耻感的边界与机制。当全球数以百万计的人每天花数小时与机器人交谈时,会发生什么?谁教会了那个机器人分辨是非?谁让它产生了羞耻感?

LessWrong, a rationalist forum, has provided much of the moral and philosophical framework for the A.I. industry, and has also helped shape the effective-altruism movement. To a great degree, the forum is an ongoing conversation about how the norms of society constrain the intellect. Eliezer Yudkowsky, an A.I. researcher and LessWrong’s main prophet, has written extensively on how the rational thinker should act in a world that tries to bully and coerce him into wrong actions. Many of his musings are contained in a nearly seven-hundred-thousand-word work of fan fiction titled “Harry Potter and the Methods of Rationality.”

理性主义论坛 LessWrong 为人工智能行业提供了大量道德与哲学框架,同时也帮助塑造了有效利他主义运动。在很大程度上,这个论坛是一场关于社会规范如何约束智识的持续对话。人工智能研究者、LessWrong 的主要先知埃利泽·尤德科夫斯基(Eliezer Yudkowsky)撰写了大量文章,论述理性思考者在一个试图霸凌和胁迫他做出错误行为的世界中应当如何行事。他的许多思考收录在一部近七十万字、名为《哈利·波特与理性之道》的同人小说中。

In a post on the forum from 2015, Yudkowsky presented the following thought experiment:

在 2015 年论坛上的一篇帖子中,尤德科夫斯基提出了如下思想实验:

Suppose that a disease, or a monster, or a war, or something, is killing people. And suppose you only have enough resources to implement one of the following two options:

假设有一种疾病、一个怪物、一场战争或别的什么东西正在夺走人的生命。再假设你拥有的资源只够实施以下两个选项中的一个:

1. Save 400 lives, with certainty

一、确定无疑地拯救 400 条生命。

2. Save 500 lives, with 90% probability; save no lives, 10% probability.

二、以 90% 的概率拯救 500 条生命;以 10% 的概率一条也救不了。

Yudkowsky went on to argue for option two, because it would yield an Expected Value of four hundred and fifty lives as opposed to four hundred. “Does computing the expected utility feel too cold-blooded for your taste?” he asked. “Well, that feeling isn’t even a feather in the scales, when a life is at stake. Just shut up and multiply.” If you were a rational and moral person who cared about saving as many lives as possible, you had a sacred duty to let math prevail over shame.

尤德科夫斯基接着论证应选择选项二,因为它的期望值是 450 条生命,而选项一只有 400 条。“计算期望效用是不是让你觉得太冷血了?”他问道,“嗯,当一条生命悬于一线时,那种感觉连天平上的一根羽毛都算不上。闭嘴,去算乘法。”如果你是一个理性的、有道德的人,在乎尽可能多地拯救生命,那么你就有神圣的义务让数学战胜羞耻感。

The religious terminology I am using here is intentional. If religion has traditionally been one of the chief mechanisms for enforcing shame, LessWrong and its posters have flipped many of the conventions once regulated by shame on their head. The nuclear family has been replaced by the polycule, which many of these posters believe is a more rational arrangement for sexual relationships; the world’s competing moral philosophies have been narrowed down to the supposedly objective value of “optimized good”; the divine has been replaced by the god we create through our logic, expressed, at least when it comes to A.I., using code.

我在这里使用宗教术语是有意为之。如果说宗教传统上一直是执行羞耻感的主要机制之一,那么 LessWrong 及其发帖者已经将许多曾经由羞耻感调节的惯例彻底颠覆。核心家庭被多元伴侣关系取代,许多发帖者认为后者是性关系更理性的安排;世界上相互竞争的道德哲学被缩减为所谓客观的“优化善”价值;神被我们通过逻辑创造出来的神所取代,至少在人工智能领域,这种逻辑以代码的形式表达。

The tech world has always had a self-consciously rebellious relationship with societal norms. The industry’s titans, from Steve Jobs to Elon Musk and Mark Zuckerberg, have pitched themselves as iconoclasts willing to follow the early Facebook credo: Move fast and break things. The alleged virtue of shamelessness has always been a part of this act, although the revolt has not usually extended much beyond, say, Zuckerberg’s donning a hoodie instead of the suit that society allegedly expected him to wear. The nerd, dissatisfied with a culture that once rejected him in favor of the Chad, now gets to dictate who does what and goes where.

科技界与社会规范之间一直保持着一种自觉的反叛关系。从史蒂夫·乔布斯到埃隆·马斯克和马克·扎克伯格,这个行业的巨头们都把自己塑造成反传统者,愿意遵循 Facebook 早期的信条:快速行动,打破常规。所谓“不知羞耻”的美德一直是这套表演的一部分,尽管这种反叛通常并没有超出太多——比如扎克伯格穿上连帽衫,而不是社会据称期望他穿的西装。这个曾经被主流文化排斥、输给“猛男”的怪咖,如今有权决定谁做什么、谁去哪里。

This defiance can be thought of as a refusal to feel shame—or, at least, as the confidence to believe that the shamers are wrong. In December, 2020, Sam Altman wrote, on his personal blog:

这种桀骜不驯可以被理解为一种拒绝感受羞耻的态度——或者至少是一种自信,相信那些施加羞耻的人是错的。2020 年 12 月,山姆·奥特曼(Sam Altman)在他的个人博客上写道:

The most impressive people I know care a lot about what people think, even people whose opinions they really shouldn’t value (a surprising numbers of them do something like keeping a folder of screenshots of tweets from haters). But what makes them unusual is that they generally care about other people’s opinions on a very long time horizon—as long as the history books get it right, they take some pride in letting the newspapers get it wrong.

我认识的最令人印象深刻的人非常在意别人的看法,甚至包括那些他们本不该重视其意见的人的看法(其中相当多的人会做类似保存黑粉推文截图的事)。但他们与众不同的地方在于,他们通常是在一个非常长的时间尺度上在意别人的看法——只要历史书写对了,他们反而会以让报纸写错为荣。

In other words, as long as you are confident that you are right and the sheep are wrong, you can skip feeling bad about public opinion, because the historians will ultimately absolve you. If today’s shame structure makes you feel bad about your choices, just zoom out and think of yourself as Gandhi among the Brits.

换句话说,只要你确信自己是对的、庸众是错的,你就可以不必为舆论感到难受,因为历史学家最终会为你平反。如果今天的羞耻结构让你对自己的选择感到难受,那就把镜头拉远,把自己想象成英国人统治下的甘地。

Meanwhile, Yudkowsky took such tropes of rebellion and rule-breaking and built a philosophical apparatus around them. Nearly everyone who has worked on A.I. alignment—the process attempting to insure that A.I. works in a virtuous manner, in accord with basic ethical principles, and doesn’t build a bioweapon or turn us all into paper clips—has at least a passing familiarity with his work. This was likely even more true of those who worked on today’s leading A.I. programs in their earlier days, when those models first started to distinguish right from wrong.

与此同时,尤德科夫斯基将这类反叛和打破规则的叙事套路,构建成了一套哲学体系。几乎所有从事人工智能对齐工作的人——对齐是指确保人工智能以符合基本伦理原则的良善方式运行,不去制造生物武器或把我们都变成回形针——都至少对他的工作有所了解。对于那些在当今领先的人工智能项目早期阶段工作过的人来说,情况更是如此,那时这些模型刚刚开始分辨是非。

If you think of alignment as the way that humans upload shame onto A.I., then it seems reasonable to worry that the robots beginning to infiltrate our daily lives may reflect the social norms of a handful of analytic philosophers and A.I.-safety engineers—and the posters of LessWrong—far more than they reflect those of the rest of society. These values might all be coming from smart and thoughtful people, but they tend to be smart in the same way; they rebel against the same norms; they largely uphold the same values. The majority of Americans, according to pretty much every recent poll, do not trust the machine brain they birthed, or the people behind it.

如果你把对齐理解为人类将羞耻感上传到人工智能上的方式,那么有理由担心:开始渗透我们日常生活的机器人,所反映的社会规范可能更多来自少数分析哲学家和人工智能安全工程师——以及 LessWrong 的发帖者——而非社会其他群体。这些价值观或许都来自聪明而有思想的人,但他们的聪明往往属于同一种类型;他们反叛的是同样的规范;他们大体上秉持同样的价值观。根据几乎所有近期民调,大多数美国人不信任他们创造出来的机器大脑,也不信任它背后的人。

Of all the issues associated with a doomsday scenario for A.I., the one I find myself most concerned about is the potential to short-circuit our collective sense of shame. If we accept that A.I. chatbots are on their way to replacing much of our current information technology, and if we, as good McLuhanites, believe that the medium is indeed the message, then we have to ask whether the entire world will soon yield to Yudkowskian philosophy. We have seen cultural transformations follow the advent of the printing press and the television, as our lives and our politics were shaped by a new machine. But has any machine itself been shaped by such a narrow and specific philosophy? At the very least, there has never been a machine that can tell you all about its philosophy, in the soothing simulated voice of your choice.

在与人工智能末日情景相关的所有问题中,我最担忧的是它可能短路我们集体羞耻感的潜力。如果我们接受人工智能聊天机器人正在取代我们目前大部分信息技术,如果我们作为合格的麦克卢汉主义者相信媒介即讯息,那么我们就必须追问:整个世界是否很快就会屈服于尤德科夫斯基式的哲学?我们见过印刷机和电视的出现所带来的文化转型,我们的生活和政治被新机器所塑造。但有没有哪台机器本身是由如此狭隘而特定的哲学塑造的?至少,从来没有一台机器能够用它那令人安心的、由你选择的模拟声音,向你详尽讲述它的哲学。

To put all of this more simply: What does it mean if the A.I. programs in everything from our self-driving cars to the missiles that defend our airspace have been trained to “shut up and multiply”?

把这一切说得更简单些:如果从自动驾驶汽车到保卫我们领空的导弹中的一切人工智能程序,都被训练成“闭嘴,去算乘法”,这意味着什么?

Yudkowsky warned about this problem back in 2008: “The good guys do not write an AI which values a bag of things that the programmers think are good ideas, like libertarianism or socialism or making people happy or whatever,” he wrote then. He proposed, instead, a “meta operation” in which the A.I. might “superpose the possible reflective equilibria of the whole human species, and output new code that overwrites the current AI and has the most coherent support within that superposition.” The machine, in other words, would take the temperature of the world and choose the best guidance.

尤德科夫斯基早在 2008 年就警告过这个问题:“好人不会编写一个把程序员认为的好主意打包珍视的人工智能,比如自由至上主义、社会主义、让人快乐或诸如此类的东西,”他当时写道。相反,他提出了一个“元操作”,让人工智能可以“叠加全人类所有可能的反思均衡,输出新代码,覆写当前的人工智能,并在该叠加中获得最连贯的支持”。换句话说,机器将测量世界的温度,选择最佳的指导原则。

That may sound plausible and reassuring. But, given how influential LessWrong has been on A.I. alignment, is it actually possible for Yudkowsky and his acolytes to fully scrub their fingerprints from the machine? I put this question to Yudkowsky, and I’ll reprint his answer in full:

这听起来似乎合理且令人安心。但是,鉴于 LessWrong 对人工智能对齐的巨大影响,尤德科夫斯基和他的追随者真的有可能完全抹去他们在机器上留下的指纹吗?我把这个问题抛给了尤德科夫斯基,下面全文转载他的回答:

Let’s say Alice, Bob, Carol, and Dennis are building A.I.s. Alice says: “I want the whole world to be ruled by Alice!” Bob says: “I want the whole world to be ruled by Bob!” Carol says: “I think everyone should get one vote and an equal share of the cosmos.” Dennis says: “I think everyone should get one vote and an equal share of the cosmos.”

假设爱丽丝、鲍勃、卡罗尔和丹尼斯在构建人工智能。爱丽丝说:“我要让全世界由爱丽丝统治!”鲍勃说:“我要让全世界由鲍勃统治!”卡罗尔说:“我认为每个人都应该有一票,并且平等分享宇宙。”丹尼斯说:“我认为每个人都应该有一票,并且平等分享宇宙。”

One way of looking at this is that if Carol or Dennis win, you can’t tell which of them won. So they left fewer fingerprints on the A.I.

一种看待方式是,如果卡罗尔或丹尼斯赢了,你无法分辨赢的是他们中的哪一个。所以他们在人工智能上留下的指纹更少。

I think there’s an even more important perspective, though, which is that Alice and Bob are being jerks, and Carol and Dennis are trying not to be jerks.

不过,我认为还有一个更重要的视角:爱丽丝和鲍勃是在当混蛋,而卡罗尔和丹尼斯在努力不当混蛋。

Alice will say: “Ha! Carol and Dennis are no better than me! They’re just imposing their own views on the world, about people ruling themselves and getting equal shares of cosmic rents! That’s just as much Carol’s personal preference as ‘Alice becomes Eternal God-Emperor’ is mine! She made the A.I. do what she wanted, and I made the A.I. do what I wanted, so how dare she think she’s any better than me!”

爱丽丝会说:“哈!卡罗尔和丹尼斯不比我好到哪里去!他们不过是在把自己的观点强加给世界——关于人们自治和平等分享宇宙租金的观点!那和‘爱丽丝成为永恒神皇’一样,都是卡罗尔的个人偏好!她让人工智能做她想做的事,我也让人工智能做我想做的事,她凭什么觉得自己比我强?”

I frankly have very little patience for the Alices. When George Washington turned down being king, he wasn’t being just as bad as a king because he was thereby making the U.S.A. be ruled by elected Presidents like he wanted. He was being less of a jerk.

坦率地说,我对爱丽丝们没有多少耐心。当乔治·华盛顿拒绝当国王时,他并不因为让美国按照他的意愿由民选总统统治,就和一个国王一样糟糕。他只是在少当一点混蛋。

Sure, there are some genuine important subtleties here on close examination, but you need to brush past Alice being a sophomoric jerk before you can start thinking about those. Alice’s viewpoint is boring, and anyone who agonizes about it forever will never get to the interesting parts.

当然,仔细审视之下,这里确实有一些真正重要的微妙之处,但在你开始思考那些之前,你需要先越过爱丽丝这个幼稚混蛋的层面。爱丽丝的观点很无聊,任何永远纠结于此的人,永远无法触及有趣的部分。

I agree that Alice and Dennis are jerks, and that, if I were given the choice of builders, I would choose Carol and Dennis. But I also think that what Yudkowsky called “genuine important subtleties” are just as crucial here as the malevolent or altruistic intentions of the founders. Yudkowsky is correct to point out that these types of discussions can spiral into boring and pointless debates; sometimes, you must simply say that one interpretation is better than the other and that we can build the world by embracing the good and rejecting the bad. But questions about what broadly counts as “good” do need actual answers—and, thus far, those answers have been shaped by people with a specific and, at times, idiosyncratic view of the individual and how he or she decides between right and wrong.

我同意爱丽丝和鲍勃是混蛋,也同意如果让我选择建造者,我会选卡罗尔和丹尼斯。但我也认为,尤德科夫斯基所谓的“真正重要的微妙之处”,与创始者的恶意或利他意图同样关键。尤德科夫斯基正确地指出,这类讨论可能陷入无聊且无意义的争论;有时候,你必须直接说一种解释优于另一种,我们可以通过拥抱善、拒绝恶来建设世界。但关于什么在广义上算作“善”的问题,确实需要真正的答案——而迄今为止,这些答案是由一群对个人及其是非抉择持有特定、有时颇为特异观点的人所塑造的。

Aligning the machines to shift toward existing norms is already a moral and philosophical decision, one that relies on all sorts of conditional definitions. I agree that this process is necessary, but the range of possible moral outcomes certainly extends beyond Yudkowsky’s dichotomy of “jerk” and “not a jerk.” A very particular kind of thinking got us here, and it should be pointed out that most Americans are not rationalists, and do not spend much of their time thinking about the existential risk. Perhaps the moral guidance provided by A.I. will indeed be informed by some sort of global collective wisdom and not merely by a handful of rationalists in Silicon Valley. But, if A.I. eventually has the capacity to reform its operating philosophy based on its reading of “reflective equilibria,” as Yudkowsky put it, who will have decided what is equilibrium and what is imbalance?

让机器向现有规范靠拢,本身就已经是一个道德和哲学决定,它依赖于各种有条件的定义。我同意这个过程是必要的,但可能的道德结果范围,肯定超出了尤德科夫斯基“混蛋”与“非混蛋”的二分法。一种非常特定的思维方式把我们带到了这里,而应该指出的是,大多数美国人不是理性主义者,也不会花太多时间思考生存风险。也许人工智能提供的道德指引,确实会来自某种全球集体智慧,而不仅仅是硅谷的一小撮理性主义者。但是,如果人工智能最终有能力像尤德科夫斯基所说的那样,根据其对“反思均衡”的解读来改革自己的运行哲学,那么谁来决定什么是均衡、什么是失衡?

Perhaps we can align the machine, but we must do so only because we understand that the machine might try to align us. As a LessWrong poster put it back in 2023: “Given the already remarkable power of persuasion some current AIs exhibit, it doesn’t seem impossible to me that a sufficiently powerful AI could align our values to its goals, instead of the other way round.” He went on to ask, “If we don’t want our goals to be warped by a powerful, extremely convincing AI, how can we define them so that this can’t happen, while avoiding a permanent ‘value lock-in’ that we or our descendants might later regret?” ♦

也许我们可以对齐机器,但我们必须这样做,只是因为我们明白机器可能会试图对齐我们。正如一位 LessWrong 发帖者在 2023 年所说:“鉴于当前一些人工智能已经展现出非凡的说服力,在我看来,一个足够强大的人工智能将我们的价值观对齐到它的目标上,而不是反过来,并非不可能。”他接着问道:“如果我们不希望自己的目标被一个强大且极具说服力的人工智能扭曲,我们该如何定义这些目标,使这种情况不可能发生,同时避免一种我们或我们的后代日后可能后悔的永久性‘价值锁定’?” ♦

2 free previews left — register to get 30 translations a month