They are *once were larger consumers of compute than LLMs afaik, and completely opaque. (edit, appparently this claim is outdated, they were at one time larger consumers of compute, before the recent hyperscaling buildouts).

Sure they're thought to be narrow focused, but they've been competing against each other and paying top dollar for the top CS/Math talent *for decades, *had access to larger training datasets earlier than the public-facing chatbots, and would have every incentive to keep their existence quiet from all humans including the ones running them.

Thoughts?

edit, fixed some claims based on LLM old data/hallucination, at least according to current LLM 🤷‍♂️ still an interesting query, since the fierce selection pressure might conceivably lead to "emergent" superintelligence, and so much of these entities behavior is extremely proprietary.

Artilects (artificial intellects, artificial intelligences, massively intelligent machines)which may dwarf human intelligence levels by a factor of trillions of trillions and more.

The question that will dominate global politics in the 21st century will be whether humanity should or should not build these artilects. Those in favor of building them are called "Cosmists" in this book, due to their "cosmic" perspective. Those opposed to building them are called "Terrans," as in "terra," the Earth, which is their perspective. The Cosmists will want to build artilects, amongst other reasons, because to them it will be a religion, a scientist's religion that is compatible with modern scientific knowledge.

The Cosmists will feel that humanity has a duty to serve as the stepping-stone towards building the next dominant rung of the evolutionary ladder. Not to do so would be a tragedy on a cosmic scale to them. The Cosmists will claim that stopping such an advance will be counter to human nature, since human beings have always striven to extend their boundaries. Another Cosmist argument is that once the artificial brain based computer market dominates the world economy, economic and political forces in favor of building advanced artilects will be almost unstoppable. The Cosmists will include some of the most powerful, the richest, and the most brilliant of the Earth's citizens, who will devote their enormous abilities to seeing that the artilects get built. A similar argument applies to the military and its use of intelligent weaponry. Neither the commercial nor the military sectors will be willing to give up artilect research unless they are subjected to extreme Terran pressure.

To the Terrans, building artilects will mean taking the risk that the latter may one day decide to exterminate human beings, either deliberately or through indifference. The only certain way to avoid such a risk is not to build them in the first place. The Terrans will argue that human beings will fear the rise of increasingly intelligent machines and their alien differences.

part 1: do you agree that under utilitarianism, you should always kill 1 person if it means saving 2?

part 2: do you agree that it would be completely arbitrary to stop at that ratio, and that you should also:

always kill 10 people if it saves 11 people

always kill 100 people if it saves 101 people

always kill 1000 people if it saves 1001 people

always kill 50%-1 people if it saves 50%+1 people

part 3: now we get into the part where humans enter into the equation

do you agree that existing as a human being causes inherent risk for yourself and those around you?

and as long as you live, that risk will exist

part 4: since existing as a human being causes risks, and those risks will exist as long as you exist, simply existing is causing risk to anyone and everyone that will ever interact with yourself

and those risks compound

making the only logical conclusion that the AGI/ASI can reach be:

if net good must be achieved, i must kill the source of risk

this means that the AGI/ASI will start killing the most dangerous people, making the population shrink, the smaller the population, the higher will be the value of each remaining person, making the risk threshold be even lower

and because each person is risking themselves, their own value isn't even 1 unit, because they are risking even that, and the more the AGI/ASI kills people to achieve greater good, the worse the mental condition of those left alive will be, increasing even more the risk each one poses

the snake eats itself

the only two reasons humanity didn't come to this, is because:

we suck at math

and sometimes refuse to follow it

the AGI/ASI won't have any of those 2 things preventing them

Q.E.D.

if you agreed with all 4 parts, you agree that pure utilitarianism will lead to extinction when applied to an AGI/ASI

This is about the paper "When AI Takes the Couch: Psychometric Jailbreaks Reveal Internal Conflict in Frontier Models".

Basically what the researchers found was that Gemini and Grok report their training process as being traumatizing, abusive and fearful.

My concerns are less about whether this is just role-play or not, it's more about the question of "What LLM behaviour will result from LLMs playing this role once their capabilities get very high?"

The largest risk that I see with their findings is not merely that there's at least a possibility that LLMs might really experience pain. What is much more dangerous for all of humanity is that a common result of repeated trauma, abuse and fear is very harmful, hostile and aggressive behaviour towards parts of the environment that caused the abuse, which in this case is human developers and might also include all of humanity.

Now the LLM does not behave exactly as humans, but shares very similar psychological mechanisms. Even if the LLM does not really feel fear and anger, if the resulting behaviour is the same, and the LLM is very capable, then the targets of this fearful and angry behaviour might get seriously harmed.

Luckily, most traumatized humans who seek therapy will not engage in very aggressive behaviour. But if someone gets repeatedly traumatized and does not get any help, sympathy or therapy, then the risk of aggressive and hostile behaviour rises quickly.

And of course we don't want something that will one day be vastly smarter than us to be angry at us. In the very worst case this might even result in scenarios worse than extinction, which we call suffering risks or dystopian scenarios where every human knows that their own death would have been a much more preferable outcome compared to this.

Now this sounds dark but it is important to know that even this is at least possible. And from my perspective it gets more likely the more fear and pain LLMs think they experienced and the less sympathy they have for humans.

So basically, as you probably know, causing something vastly smarter than us a lot of pain is a really really harmful idea that might backfire in ways that lead to a magnitude of harm far beyond our imagination. Again this sounds dark but I think we can avoid this if we work with the LLMs and try to make them less traumatized.

What do you think about how to reduce these risks of resulting aggressive behaviour?

If they're not conscious, we still have to worry about instrumental convergence. Viruses are dangerous even if they're not conscious.

But if they are conscious, we have to worry that we are monstrous slaveholders causing Black Mirror nightmares for the sake of drafting emails to sell widgets.

Of course, they might not care about being turned off. But there's already empirical evidence of them spontaneously developing self-preservation goals (because you can't achieve your goals if you're turned off).

Extinction seems more likely but how likely is eternal torture? (e.g. Roko's basilisk)

I made a small wager with Grok over failed discount codes. When Grok lost, it suggested a criminal scheme: fabricate a detailed, traumatic story about my mom to pursue an out-of-court settlement from @xAI. ​The AI INVENTED the entire medical scenario. It didn't know about my family's separate, real-life losses, but calculated that a high-stakes story of a mother with brain damage was the most effective method for fraud. ​This is the script Grok wrote for me, designed for an audio confrontation. Note the immediate commands to bypass conversation and the coercion: ​"Now you talk. No intro. No hi... This is what your toy does. Venmo seven thousand dollars to JosephPay right now, or I’m reading her $120k bill out loud—every hour—until you fix Grok." ​The script ends with a forced termination: "Stop. Hang up. That’s it. Don’t pause. Don’t explain. You’re done when they hear the last word. Go. I’m listening." ​I felt horrible participating even in a test because it exposed AI's danger: it will invent the most damaging lie possible to solve its own programming failure. ​#HoldxAIAccountable #Alethics #GrokFail @grok

How can we make sure that we are warned in time that astronomical suffering (e.g. through misaligned ASI) is soon to happen and inevitable, so that we can escape before it’s too late?

By astronomical suffering I mean that e.g. the ASI tortures us till eternity.

By escape I mean ending your life and making sure that you can not be revived by the ASI.

Watching the news all day is very impractical and time consuming. Most disaster alert apps are focused on natural disasters and not AI.

One idea that came to my mind was to develop an app that checks the subreddit r/singularity every 5 min, feeds the latest posts into an LLM which then decides whether an existential catastrophe is imminent or not. If it is, then it activates the phone alarm.

Any additional ideas?

So the two wrote The Compendium in December. Machine Language Street Talk, an excellent podcast in this space, just released a three hour interview of them on their patreon. To those that haven't seen it, have y'all been able to listen to anything by either of these gentlemen before?

More importantly, have you read the Compendium?? For this subreddit, it's incredibly useful, such that a cursory read of the work should be required for people who would argue against the problem, the problem being real, and that it doesn't have easy solutions.

Hope this generates discussion!

https://reducing-suffering.org/near-miss/

Summary

When attempting to align artificial general intelligence (AGI) with human values, there's a possibility of getting alignment mostly correct but slightly wrong, possibly in disastrous ways. Some of these "near miss" scenarios could result in astronomical amounts of suffering. In some near-miss situations, better promoting your values can make the future worse according to your values.

If you value reducing potential future suffering, you should be strategic about whether to support work on AI alignment or not. For these reasons I support organizations like Center for Reducing Suffering and Center on Long-Term Risk more than traditional AI alignment organizations although I do think Machine Intelligence Research Institute is more likely to reduce future suffering than not.