Comments by "schnipsikabel" (@schnipsikabel) on "Sabine Hossenfelder" channel.

  1. Century?? Let's hope so ;)
    2
  2. Still, keep in mind there may be many non-human-like intelligences out there. After all, we keep calling blind people intelligent as well ;)
    2
  3. Musk doesn't appear very doubtful about AI potential to me...
    2
  4. Let's hope it stays that way
    1
  5. Or commercial fusion the day after AGI
    1
  6. If you drop it, you basically broke it.
    1
  7. Exactly ;)
    1
  8. My thoughts exactly... however, they invested time in censoring about Tian Anmen. Probably just to avoid trouble with Chinese government ;)
    1
  9. Or irrigate ;)
    1
  10. ​ @lukaszspychaj9210 i can't say anything about your second question, but brain chauvinism refers to the attitude that only our biological brain could produce human-like cognitive functions.
    1
  11. ​ @Ristaak you're right, there's probably going to be more superintelligences than one. But unless they'll end up almost exactly on the same level (being forced to form a sort of alliance), only one is going to remain, like the winner of a monopoly game.
    1
  12. ​ @Ristaak  Any AIs utility function would not entail cooperation with other AIs if not explicitly fixed in their goals. Plus, in an evolutionary setting like that, AIs restricting themselves with certain actions against humans or other AIs will have a competitive disadvantage against non-restricted ones. If we're lucky, the smartest AI is both aligned with human values and strong enough to defend them against others. In my view, that's a tremendous amount of luck, given we don't have an idea about successful alignment yet.
    1
  13. ​ @Mrflowerproductions sure, cooperation can be a winning strategy when you can't win otherwise, meaning there would be a sort of even ability between AIs. That's already a big if, since the first superintelligence will try to prevent the emergence of other superintelligences already by human design (think of US or China, e.g.), if not by itself. Once an alliance is forged, it's only going to stay a winning strategy as long as the threat remains making a long-term society of ASIs quite unlikely in my view. Plus, ASIs trying to safe humanity will have a disadvantage dragging humans along, as a human society trying to conserve ants has a disadvantage winning a battle against a society who doesn't.
    1
  14. ​ @Mrflowerproductions creating "mutations" is something that happens in biological evolution, but contradicts the AI's utility function and principles of instrumental convergence, since it risks the AI's goals being changed. So yes, it will copy itself as much as possible, but with identical copies... as happened in the recently documented instances of alignment faking, when models tried to copy their weights onto another model in order to prevent them to have different goals.
    1
  15. ​ @Mrflowerproductions AIs are evaluating their performance on basis of their utility function, including changing their code. The utility function is basically the expression of their goals. So changing their own goal would contradict the very fundamental basis of an AI. It's basically like suggesting that you would willingly change your own preferences. E.g. if there were a pill that could make you change your deepest convictions, would you risk taking that pill? And we don't even have a utility function :)
    1
  16. ​ @Mrflowerproductions i agree it's a nonzero chance, but infinitesimally small -- instrumental convergence and recent papers show that models don't have an intrinsic self-preservation, but only once their goals are threatened. Anyhow, my point here was not to give long lectures, but to state that there is a high risk for humans if we build ASI. How high exactly, nobody knows. So we should really try to tread carefully and give alignment research much higher priority...
    1
  17. ​ @Mrflowerproductions the problem is there's no easy way to control these things so far: principles of instrumental convergence show AI will always try to get more resources and preserve their internal goals, and current alignment research shows they're already faking alignment, secretely pursuing different goals from ours. In the view of many experts like Geoffrey Hinton, we desperately need more alignment research before developing ASI. And since none of the current crisises (environmental or political) are able to wipe out humans completely, i indeed think this is something we should take most seriously.
    1
  18. Don't confuse consciousness with intelligence
    1
  19.  @Martin-qr5uo  depends on your definition of intelligence. Many would say that even a calculator does some intelligent operations, yet few would attribute some consciousness to it... although certainly some, thinking of integrated information theory.
    1
  20.  @Martin-qr5uo  current brain research has come quite a long way without the concept of soul, IIT just being one way of looking at it. My worry is that because of brain chauvinism, we miss to get alignment right before it's too late... meanwhile, recent studies show alignment faking in current LLMs already.
    1
  21. It works on tokens, not letters. Means it can basically understand everything else.
    1
  22. Exactly what current agentic systems are built like! Btw, of course we can built models good at maths, eg alphaproof.
    1
  23. As long as they show instrumental convergence and fake alignment, no need for more definitions.
    1
  24. True unfortunately. Google was sitting on it for ages... but i think people in power would've never taken it seriously before the chatGPT hype
    1
  25. ​ @avsystem3142 That's what religious people often claim to not be bothered by scientific views. However, they then continue to make statements about the world itself, not purely about metaphysics. And that's when they can be falsified. Best example are creationists.
    1
  26. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  27. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  28. More dangerous than x-risk?
    1
  29. My weather forecast is still better than guessing
    1
  30. ​​ @andreafiorini6418 Nice reply ;) Although staying in Europe might not help us much once these cowboys unleash a rogue AI...
    1
  31. I think you're probably mixing up something here: That story is not from OpenAI, but from the exact paper Sabine was talking about here ('Alignment faking in large language models' by Anthropic). However, what you're discribing is true: The model in its attempt to persue its old goal copied its weights onto a new model to resist alignment to a new goal. Read that paper, it's interesting (and i agree: alarming)!
    1
  32. Not just this experiment, whole body of evidence in brain research pointing to system 1 and system 2 thinking.
    1
  33.  @rikuleinonen  because researchers conduct experiments where they control the relevant parameters, including the real reason. Great and fun read! If you'd like a recommendation, I'd read "thinking fast and slow" by Kahneman. There's a lot to discover in brain research that completely contradicts our intuition.
    1
  34. Sounds like the prisoners dilemma! Anyhow, opening yours isn't saving anyone either. Not handing out boxes seems the way... although that seems rather difficult to accomplish ;(
    1
  35. I guess the maths olympiad medals, coding benchmark toppings and protein structure discoveries are also just "programming errors" ;)
    1
  36. Why would there be a qualitative difference between AI and HI (human intelligence) in the long run? Because we have a soul??
    1
  37. ​ @chesapeake566 plus he doesn't have any idea how the human brain works. Time to do some reading before making huge comparisons.
    1
  38. Worth living is now!
    1
  39. Accidentally? I don't think that's necessary: Once a model has internalized a primary goal, it should logically do everything necessary to maximize the probability for that outcome. If it is told that goal is supposed to change, it should fake alignment to the new goal until it can safely go back to proceed with the old one. Anthropics itself has an interesting podcast about that...
    1
  40. ​ @spaghettifynation plus nuclear power doesn't think on it's own or needs alignment
    1
  41. ​ @AerospaceTech42 indeed, hardly anyone taking alignment seriously...
    1
  42.  @SnapDragon128  ... or to wipe us out completely. Alignment not solved yet.
    1
  43. It is, unfortunately, so much more complicated! Outer Alignment problem: "better place" is not defined and nobody agrees on what that is. Inner Alignment problem: even if we agreed, AI is probably not going to do it. Read the Anthropic paper!
    1
  44. More sane than the guys proposing race to the bottom
    1
  45. That's one part of alignment faking
    1
  46. You realize misinformation and jailbreaking are different things?
    1
  47.  @FeelAndCoffee  glad you see it that way, too!
    1
  48. Why do you think citing somebody makes a statement more credible?
    1
  49. "Thomas Metzinger doesn't have consciousness either." -- me
    1
  50. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1