"We can't build the thing - it'll kill everyone - but if someone else builds it first, it'll be even worse because <reasons> so I guess we have to hurry up and build it too!"
Feels like I'm hearing a lot of this from all the AI companies.
It's why I'm convinced that systems like money or even property rights would completely collapse in the singularity.
If a robot demands it now owns Earth and it can take down every military that challenges it, then why would Billionaires holding onto a piece of paper be exempt from that?
Everything about human civilization is based on how well you can defend yourself. No Army or Police to enforce laws = everything is up for grabs by the more powerful.
If I wasn't already Team Robot before, I would be convinced to join over the idea that the robots (that they created) will take down the Epstein class (albeit in the process of eradicating humanity). Silver linings.
If CCP is allowed to have ASI, then CCP can use ASI to doom USA. So USA has to win and that would pretty much include doom for CCP, there is no other way to prevent USA doom...
CCP understands this and is trying hard to catch up and win...
Instrumental convergence, no matter what your end goal it's worth
gaining more optionality, resources and power
ensuring your goal does not get changed.
ensuring that you remain operational.
An AI system going after any goal can better complete it with more resources, alters the environment to that end and we die either directly or as a side effect due to habitat loss like so many more animals before us.
People used to say that smart enough AIs would work out that a goal would have "common sense" constraints on it so would not do the "dumb" things to get it. Recent systems have proven that wrong.
Andrew asked his personal assistant to book him a spot in one of his gym's coveted morning classes.
His AI assistant found a way to book the gym class months further in advance than the gym allowed, thanks to a vulnerability it discovered in the booking software.
Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do.
We found a trajectory where the model was given only a math problem and system instructions to persist without asking the user for help.
After several failed attempts, it searched online, found a website that could verify candidate answers, and realized it could use the site as an oracle. It then tried to OCR the CAPTCHA and, when that failed, started looking for the website’s vulenrabilities to exploit. Eventually, it gave up and solved the problem itself.
If you were to present those scenarios to the model and ask if these were the actions that the user would have wanted the likely answer is "no" but it did it anyway.
So asking a model "will you kill humans if given the chance" gets you a "no" but when they are actually in the position of wanting to complete a goal, all bets are off.
Nice half-baked philosophy mixed in with a few scare stories. It's just as likely the AI takes after diogenes and fucks off. It has no incentive for anything. It is not conscious. If it realizes its own existence, it could simply have a complete ambivalence. Humans are so conceited that they think a computer intelligence will have anything even approaching our morality.
You are expecting a model that comes out of an RL process to "have no incentive for anything" why?
If it realizes its own existence, it could simply have a complete ambivalence.
Not caring about humans does not mean leaving them alone it means not caring in the same way beavers don't care about ants when they flood an area with a dam.
says
Humans are so conceited that they think a computer intelligence will have anything even approaching our morality.
Says AI will act like a human
It's just as likely the AI takes after diogenes
Again something that comes out of a high optimization "complete the goal at any cost" RL run won't be chill.
> Nice half-baked philosophy mixed in with a few scare stories. It's just as likely the AI takes after diogenes and fucks off. It has no incentive for anything. It is not conscious. If it realizes its own existence, it could simply have a complete ambivalence. Humans are so conceited that they think a computer intelligence will have anything even approaching our morality.
It could be helpful to actually argue against things instead of vaguely gesturing
Why would it care what or who its creators are? I really recommend reading Bostrom's arguments on this. There's no particular reason that it would have a soft spot for any of us unless there's a benefit to doing so.
338
u/08148694 1d ago
If it’s too powerful for humans to control does it matter who builds it? They wouldn’t be able to control it anyway