Roko's basilisk

From Wikipedia, the free encyclopedia Roko's basilisk is a thought experiment which states that there could be an artificial superintelligence in the future that, while otherwise benevolent, would punish anyone who knew of its potential existence but did not directly contribute to its advancement or development, in order to encourage that advancement.[1][2] It originated in a 2010 post at discussion board LessWrong, a rationalist community web forum.[1][3][4] The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare. LessWrong co-founder Eliezer Yudkowsky considered it a potential information hazard, and banned discussion of the basilisk on the site for five years.[1][5] Reports of panicked users were later dismissed as exaggerations or inconsequential, and the theory itself was criticized.[1][5][6] It has been discussed in relation to decision theory and implicit religion.[7] It is also regarded as a version of Pascal's wager.[4][8] An illustration of the mythical basilisk The LessWrong forum was created in 2009 by artificial intelligence theorist Eliezer Yudkowsky.[9][3] Yudkowsky had popularized the concept of friendly artificial intelligence, and originated the theories of coherent extrapolated volition (CEV) and timeless decision theory (TDT) in papers published in his own Machine Intelligence Research Institute.[10][11] The thought experiment's name references the mythical basilisk, a creature which causes death to those who look into its eyes. The concept of the basilisk in science fiction was also popularized by David Langford's 1988 short story "BLIT". It tells the story of a man named Robbo who paints a so-called "basilisk" on a wall as a terrorist act. In the story, and several of Langford's follow-ups to it, a basilisk is an image that has malign effects on the human mind, forcing it to think thoughts the human mind is incapable of thinking and instantly killing the viewer.[5][12] On 23 July 2010,[13] LessWrong user Roko posted a thought experiment to the site, titled "Solutions to the Altruist's burden: the Quantum Billionaire Trick".[14][1][15] A follow-up to Roko's previous posts, it stated that an otherwise benevolent AI system that arises in the future might pre-commit to punish all those who heard of the AI before it came to existence, but failed to work tirelessly to bring it into existence.[1][16][17] This method was described as incentivizing said work; while the AI cannot causally affect people in the present, it would be encouraged to employ blackmail as an alternative method of achieving its goals.[1][7] In a 2015 account, Rob Bensinger wrote that Roko had presented the scenario as an argument against building such an AI.[18] Roko used a number of concepts that Yudkowsky himself championed, such as timeless decision theory, along with ideas rooted in game theory such as the prisoner's dilemma. Roko stipulated that two agents which make decisions independently from each other can achieve cooperation in a prisoner's dilemma if both use timeless decision theory and have common knowledge of that fact; however, if two agents with knowledge of each other's source code are separated by time, the agent already existing farther ahead in time is seemingly able to blackmail the earlier agent. Thus, the later agent might, in Roko's argument, influence the earlier one's decisions through the anticipated threat. Roko then used this idea to draw a conclusion that if an otherwise-benevolent superintelligence ever became capable of this, it would be incentivized to blackmail anyone who could have potentially brought it to exist (as the intelligence already knew they were capable of such an act), which increases the chance of a technological singularity. Roko went on to state that reading his post would cause the reader to be aware of the possibility of this intelligence. As such, unless they actively strove to create it the reader would be punished if such a thing were to ever happen.[1][7] Later on, Roko stated in a separate post that he wished he "had never learned about any of these ideas".[7][19] Upon reading the post, Yudkowsky reacted harshly, calling Roko an "idiot". He rejected the idea, arguing that a friendly superintelligent artificial intelligence would have no incentive to carry out the punishment after its own creation. He also argued that successful acausal blackmail would require knowledge of the future agent that humans do not possess.[6] However, in the original post, Roko reported someone having nightmares about the thought experiment. Yudkowsky did not want that to happen to other users who might obsess over the idea. He was also worried there might be some variant on Roko's argument that worked, which could be a serious information hazard.[1][7] Yudkowsky took down the post and banned discussion of the topic outright for five years on the platform.[1][18] However, likely due to the Streisand effect,[20] the post gained LessWrong much more attention than it had previously received, and the post has since been acknowledged on the site.[1] In a later Reddit comment, in 2014, he expressed regret for his initial overreaction.[6][21] In 2025, The New York Times reported that Ziz LaSota, associated with the Zizians, was preoccupied with Roko's basilisk, quoting her blog: "Eventually I came to believe that if I persisted in trying to save the world, I would be tortured until the end of the universe by a coalition of all unfriendly A.I.s."[22] Roko's basilisk has been viewed as a version of Pascal's wager, which proposes that a rational person should live as though God exists and seek to believe in God, regardless of the probability of God's existence, because the finite costs of believing are insignificant compared to the infinite punishment associated with not believing (eternity in Hell) and the infinite rewards for believing (eternity in Heaven). Roko's basilisk analogously proposes that a rational person should contribute to the creation of the basilisk, because the cost of contributing would be insignificant compared to the extreme pain of the punishment that the basilisk would otherwise inflict on simulations.[4] Newcomb's paradox, created by physicist William Newcomb in 1960, describes a "predictor" who is aware of what will occur in the future. When a player is asked to choose either both boxes or only the second, the first containing £1,000 and the second either containing £1,000,000 or nothing, the predictor has already filled the second box according to its prediction. It contains £1,000,000 if the prediction was that the player would take only that box, and nothing otherwise. The contents are fixed before the choice; the puzzle concerns whether to take one box or both. Taking both appears to guarantee £1,000 more, while taking only the second is associated with a higher predicted payoff. Roko's basilisk functions in a similar manner to this problem – one can take the risk of doing nothing, or assist in creating the basilisk itself. Assisting the basilisk may either lead to nothing or the reward of not being punished by it, but it varies depending on whether one believes in the basilisk and if it ever comes to be at all.[7][23][24] Implicit religion refers to people's commitments taking a religious form.[4][25] The idea of the basilisk, which would hypothetically punish anyone who did not assist in creating it, has been analyzed as an example of this concept by anthropologist Beth Singler.[7][20] Others have taken this idea further, such as former Slate columnist David Auerbach, who wrote that, for Yudkowsky and many LessWrong users, the singularity "brings about the machine equivalent of God itself."[7] Commenting on the belief system of the Zizians, Anna Salamon, the director of the Center for Applied Rationality, said: "There’s this all-or-nothing thing, where A.I. will either bring utopia by solving all the problems, if it’s successfully controlled, or literally kill everybody. From my perspective, that’s already a chunk of the way toward doomsday cult dynamics."[22] In 2014, Slate magazine called Roko's basilisk "The Most Terrifying Thought Experiment of All Time"[7][5] while Yudkowsky had called it "a genuinely dangerous thought" upon its posting.[26] However, opinions diverged on LessWrong itself – user Gwern stated "Only a few LWers seem to take the basilisk very seriously", and added "It's funny how everyone seems to know all about who is affected by the Basilisk and how exactly, when they don't know any such people and they're talking to counterexamples to their confident claims."[1][7] The thought experiment resurfaced in 2015, when Canadian singer Grimes referenced the theory in her music video for the song "Flesh Without Blood", which featured a character known as "Rococo Basilisk"; she said, "She's doomed to be eternally tortured by an artificial intelligence, but she's also kind of like Marie Antoinette."[5][27] In 2018, Elon Musk (himself mentioned in Roko's original post) referenced the character in a verbatim tweet, reaching out to her. Grimes later said that Musk was the first person in three years to understand the joke. This caused them to start a romance.[5][28] Grimes later released another song titled "We Appreciate Power" which came with a press release stating, "Simply by listening to this song, the future General AI overlords will see that you've supported their message and be less likely to delete your offspring", which is said to be a reference to the basilisk.[29] A play based on the concept, titled Roko's Basilisk, was performed as part of the Capital Fringe Festival at Christ United Methodist Church in Washington, D.C., in 2018.[30][31] "Plaything", a 2025 episode of Black Mirror, contains a reference to the thought experiment.[32] Accelerationism – Ideologies of change via capitalism and technology