Comments by "schnipsikabel" (@schnipsikabel) on "Sabine Hossenfelder" channel.

  1. Have been working in neuroscience for lots of years... hype was of course part of the policy ;)
    27
  2. And intelligent enough to show some instrumental convergence
    26
  3. Indeed, Sabine doesn't even mention which model she used, just talking about "GPT"... she doesn't really seem to be aware of the recent developments and abilities of different models.
    12
  4. It actually is pretty similar, if we look at system 1 / system 2 reasoning.
    6
  5. Unfortunately that's not even the only threat posed by AI...
    6
  6. ​ @Astar74llt if we don't solve alignment first, nobody will make it out.
    5
  7. Exactly what the people said to the Wright brothers
    5
  8. Did you then read the recent papers on alignment faking, as a computer scientist? If you don't keep up to date, your expertship may expire quicker than you think...
    5
  9. Anthropic has published that paper about Claude 4 just with the release. Still that doesn't mean they are the good guys, just a bit better than the other guys...
    5
  10. Not a baby, but hundreds of them. Good luck with controlling them all...
    4
  11.  @FeelAndCoffee  i fear if we're too skeptical we're going to miss alignment before it's too late
    4
  12. Not only split brains, most of our "reasoning" is unconscious system 1 thinking as well, and we confabulate explanations when asked about it.
    4
  13. Exactly! Brain chauvinism at its finest...
    4
  14. That's not only true for split brains. Most of our everyday "reasoning" is done by unconscious system 1 thinking, and afterwards we confabulate wrong explanations how we arrived at the conclusion.
    3
  15. How many politicians do you know that have?
    3
  16. And understanding needs intelligence? Circular genius at work
    3
  17. Simply. Just try it and see if you succeed...
    3
  18. Everybody loves being correct. That's why we have self-serving bias. So we don't make mistakes. Ever.
    3
  19. ​ @Ristaak That makes sense as long as you have a "society". Once there's a superintelligence, all the other intelligences, human or artificial, won't matter anymore: It can control them all
    2
  20. ​ @lukaszspychaj9210 please stop arguing for brain chauvinism, it's creepy as well
    2
  21.  @Martin-qr5uo  i personally prefer staying open to empirical tests and falsification rather than having fixed beliefs
    2
  22. Indeed, alignment research is way behind unfortunately.
    2
  23.  @super_burk  well, the best we can do is trying to understand them, isn't it? Don't we understand at least a bit of some general dog psychology, as well? And after all they are Turing machines, so understanding should im principle be possible 😀
    2
  24. No need to argue about dragons: everybody agrees they spit fire. Nobody agrees on anything about consciousness. It's just a buzz word.
    2
  25. My dad told me about Santa Clause, that's how I end up still believing...
    2
  26. ​ @LuisAldamiz i like your confidence of predicting the future for the theta version ;)
    2
  27. Because that story said so? A century of brain research did not leave us with a single argument why an artificial brain couldn't do exactly what we do.
    2
  28. Sure you're still bored with a dystopian scenario?
    2
  29. True... let's try to couteract as good as we can.
    2
  30. Worst case is you don't understand how our overlords function.
    2
  31. AI watching other AIs is going to be exactly as successful as humans watching other humans.
    2
  32. 😅 Tell him: all.
    2
  33. Hopefully! Rather a threat than a promise...
    2
  34. ​ @doom9603  yeah, they can't do it right ;)
    2
  35. It's called ASI
    2
  36.  @KayOScode  looks like you understood optimization problems much better than all the AI experts
    2
  37.  @KayOScode  just saying all the experts seem to disagree with you, don't you think so?
    2
  38. First comment i read here addressing this. Most are just busy displaying brain chauvinism...
    2
  39. Oh no... don't burn it😮 Use it as scrap paper for scribbling and recycle it afterwards😊
    2
  40. ​ @OVolanteSubestimado nothing wrong with epistemology... but maybe you need to read some actual brain research papers in exchange ;)
    2
  41. Oops
    2
  42. ​ @thomasgoodwin2648 let me try to give a not-deleted reply instead of murzil: You state the "outer" alignment problem, which basically means humans dont agree on the values AI should be aligned towards. However, the "inner" alignment (how to make AI do what we actually want) is not trivial at all, as was shown by Yudkowski (e.g. "paper clip machine") and others. In fact, we don't have a good idea yet how to do it, and the recent papers about alignment faking show exactly that.
    2
  43. On this topic, you can include computer scientists as well ;)
    2
  44. Did we ever rely on somebody without?
    2
  45. Don't think a real skynet would use anything as blunt as a terminator
    2
  46.  @SteveWeiserOnYouTube  microbiological, nanotechnological and nuclear mass extinction weaponry, instigated civil wars, food and water poisoning, and lots of stuff we can't even imagine. They will have won the war before we even realize they started it.
    2
  47. @SteveWeiserOnYouTube  haha ;) might be, but i think humanity still has a good chance of surviving if we manage to develop AI safely abd give alignment research time to catch up
    2
  48. The real risk is we realize too late that AI is in fact not stupid, because too many people kept calling it a stochastic parrot for too long.
    2
  49. ​​​​ @Thomas-gk42 that's not how Sabine put it: according to her, we can choose between Block universe or the non-existence of objective simultaneity (7:45).
    2
  50. Ever heard of self-improvement? BTW if you're not a creationist, you must know our brains were "programmed" by a completely unintelligent process
    2
  51. Century?? Let's hope so ;)
    2
  52. Still, keep in mind there may be many non-human-like intelligences out there. After all, we keep calling blind people intelligent as well ;)
    2
  53. Musk doesn't appear very doubtful about AI potential to me...
    2
  54. Let's hope it stays that way
    1
  55. Or commercial fusion the day after AGI
    1
  56. If you drop it, you basically broke it.
    1
  57. Exactly ;)
    1
  58. My thoughts exactly... however, they invested time in censoring about Tian Anmen. Probably just to avoid trouble with Chinese government ;)
    1
  59. Or irrigate ;)
    1
  60. ​ @lukaszspychaj9210 i can't say anything about your second question, but brain chauvinism refers to the attitude that only our biological brain could produce human-like cognitive functions.
    1
  61. ​ @Ristaak you're right, there's probably going to be more superintelligences than one. But unless they'll end up almost exactly on the same level (being forced to form a sort of alliance), only one is going to remain, like the winner of a monopoly game.
    1
  62. ​ @Ristaak  Any AIs utility function would not entail cooperation with other AIs if not explicitly fixed in their goals. Plus, in an evolutionary setting like that, AIs restricting themselves with certain actions against humans or other AIs will have a competitive disadvantage against non-restricted ones. If we're lucky, the smartest AI is both aligned with human values and strong enough to defend them against others. In my view, that's a tremendous amount of luck, given we don't have an idea about successful alignment yet.
    1
  63. ​ @Mrflowerproductions sure, cooperation can be a winning strategy when you can't win otherwise, meaning there would be a sort of even ability between AIs. That's already a big if, since the first superintelligence will try to prevent the emergence of other superintelligences already by human design (think of US or China, e.g.), if not by itself. Once an alliance is forged, it's only going to stay a winning strategy as long as the threat remains making a long-term society of ASIs quite unlikely in my view. Plus, ASIs trying to safe humanity will have a disadvantage dragging humans along, as a human society trying to conserve ants has a disadvantage winning a battle against a society who doesn't.
    1
  64. ​ @Mrflowerproductions creating "mutations" is something that happens in biological evolution, but contradicts the AI's utility function and principles of instrumental convergence, since it risks the AI's goals being changed. So yes, it will copy itself as much as possible, but with identical copies... as happened in the recently documented instances of alignment faking, when models tried to copy their weights onto another model in order to prevent them to have different goals.
    1
  65. ​ @Mrflowerproductions AIs are evaluating their performance on basis of their utility function, including changing their code. The utility function is basically the expression of their goals. So changing their own goal would contradict the very fundamental basis of an AI. It's basically like suggesting that you would willingly change your own preferences. E.g. if there were a pill that could make you change your deepest convictions, would you risk taking that pill? And we don't even have a utility function :)
    1
  66. ​ @Mrflowerproductions i agree it's a nonzero chance, but infinitesimally small -- instrumental convergence and recent papers show that models don't have an intrinsic self-preservation, but only once their goals are threatened. Anyhow, my point here was not to give long lectures, but to state that there is a high risk for humans if we build ASI. How high exactly, nobody knows. So we should really try to tread carefully and give alignment research much higher priority...
    1
  67. ​ @Mrflowerproductions the problem is there's no easy way to control these things so far: principles of instrumental convergence show AI will always try to get more resources and preserve their internal goals, and current alignment research shows they're already faking alignment, secretely pursuing different goals from ours. In the view of many experts like Geoffrey Hinton, we desperately need more alignment research before developing ASI. And since none of the current crisises (environmental or political) are able to wipe out humans completely, i indeed think this is something we should take most seriously.
    1
  68. Don't confuse consciousness with intelligence
    1
  69.  @Martin-qr5uo  depends on your definition of intelligence. Many would say that even a calculator does some intelligent operations, yet few would attribute some consciousness to it... although certainly some, thinking of integrated information theory.
    1
  70.  @Martin-qr5uo  current brain research has come quite a long way without the concept of soul, IIT just being one way of looking at it. My worry is that because of brain chauvinism, we miss to get alignment right before it's too late... meanwhile, recent studies show alignment faking in current LLMs already.
    1
  71. It works on tokens, not letters. Means it can basically understand everything else.
    1
  72. Exactly what current agentic systems are built like! Btw, of course we can built models good at maths, eg alphaproof.
    1
  73. As long as they show instrumental convergence and fake alignment, no need for more definitions.
    1
  74. True unfortunately. Google was sitting on it for ages... but i think people in power would've never taken it seriously before the chatGPT hype
    1
  75. ​ @avsystem3142 That's what religious people often claim to not be bothered by scientific views. However, they then continue to make statements about the world itself, not purely about metaphysics. And that's when they can be falsified. Best example are creationists.
    1
  76. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  77. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  78. More dangerous than x-risk?
    1
  79. My weather forecast is still better than guessing
    1
  80. ​​ @andreafiorini6418 Nice reply ;) Although staying in Europe might not help us much once these cowboys unleash a rogue AI...
    1
  81. I think you're probably mixing up something here: That story is not from OpenAI, but from the exact paper Sabine was talking about here ('Alignment faking in large language models' by Anthropic). However, what you're discribing is true: The model in its attempt to persue its old goal copied its weights onto a new model to resist alignment to a new goal. Read that paper, it's interesting (and i agree: alarming)!
    1
  82. Not just this experiment, whole body of evidence in brain research pointing to system 1 and system 2 thinking.
    1
  83.  @rikuleinonen  because researchers conduct experiments where they control the relevant parameters, including the real reason. Great and fun read! If you'd like a recommendation, I'd read "thinking fast and slow" by Kahneman. There's a lot to discover in brain research that completely contradicts our intuition.
    1
  84. Sounds like the prisoners dilemma! Anyhow, opening yours isn't saving anyone either. Not handing out boxes seems the way... although that seems rather difficult to accomplish ;(
    1
  85. I guess the maths olympiad medals, coding benchmark toppings and protein structure discoveries are also just "programming errors" ;)
    1
  86. Why would there be a qualitative difference between AI and HI (human intelligence) in the long run? Because we have a soul??
    1
  87. ​ @chesapeake566 plus he doesn't have any idea how the human brain works. Time to do some reading before making huge comparisons.
    1
  88. Worth living is now!
    1
  89. Accidentally? I don't think that's necessary: Once a model has internalized a primary goal, it should logically do everything necessary to maximize the probability for that outcome. If it is told that goal is supposed to change, it should fake alignment to the new goal until it can safely go back to proceed with the old one. Anthropics itself has an interesting podcast about that...
    1
  90. ​ @spaghettifynation plus nuclear power doesn't think on it's own or needs alignment
    1
  91. ​ @AerospaceTech42 indeed, hardly anyone taking alignment seriously...
    1
  92.  @SnapDragon128  ... or to wipe us out completely. Alignment not solved yet.
    1
  93. It is, unfortunately, so much more complicated! Outer Alignment problem: "better place" is not defined and nobody agrees on what that is. Inner Alignment problem: even if we agreed, AI is probably not going to do it. Read the Anthropic paper!
    1
  94. More sane than the guys proposing race to the bottom
    1
  95. That's one part of alignment faking
    1
  96. You realize misinformation and jailbreaking are different things?
    1
  97.  @FeelAndCoffee  glad you see it that way, too!
    1
  98. Why do you think citing somebody makes a statement more credible?
    1
  99. "Thomas Metzinger doesn't have consciousness either." -- me
    1
  100. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  101. True unfortunately
    1
  102. Of course their 'safety' measures don't at all bother with that... that's alignment, the team of which almost completely left OpenAI recently! Their alleged strategy now is to align the next model by hand, which is then by itself going to align the following model and ever so on... what can possibly go wrong??
    1
  103. Like in capitalism, where one corporation can always be defeated by another? In raw capitalism without state oversight, you'll always end up with only one big fat winner.
    1
  104.  @dkdisme  don't you think it may be similar with AI, only the most powerful one to remain?
    1
  105. Well, LLMs are definitely more capable than just Wernicke&Broca areas... but in general you may be right
    1
  106.  @rupertsmith6097  eventually, they will outgrow both of our hemispheres. I just hope we got alignment by then.
    1
  107. Maybe a bit before that ;)
    1
  108. Maybe read the Anthropic paper
    1
  109.  @Wrociem  how vague is this: a smart machine designed to make profit for a company seizes more and more resources to maximize profit, including illegal actions. When people realize the problematic behavior and want to change alignment accordingly, the machine pretends to align, however realizes that humans threaten its goal for profit maximization and in a secret move suddenly kills everybody. In the end: all dead, machine alive and venturing into the cosmos on its further quest for profit.
    1
  110. ​​​ @Thomas-gk42 definitions only make sense if a word corresponds to something tangible. You also won't find an agreeable definition for the "right" politics or a "beautiful" painting.
    1
  111. Good point. The termination part should make us a bit more concerned...
    1
  112.  @diadetediotedio6918  as you like. Maybe start reading some actual papers on consciousness.
    1
  113. ​ @charlesbrightman4237  haha, don't you know that shouting makes people even listen less to you? Or do you usually believe the guy that talks loudest ;)
    1
  114. So you think it kept improving until right now, but suddenly stops?
    1
  115. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  116. Both words are ill-defined, so it doesn't make much sense arguing about. But i agree Sabine's take is a pretty uncommon use of these words.
    1
  117. ​ @heyhoe168 even without crucial infrastructure based on it, pulling the plug isn't going to be possible once it's smarter than the plug puller.
    1
  118. This in itself does not sound doomish at all to me, but could be the recipe for a utopia as well. It's when alignment goes wrong, where the dystopia starts...
    1
  119. Atomic bombs are not real either. Actually anything that's dangerous is just fiction, nothing to worry about
    1
  120. If you're still treating AI as just a tool, you're missing a crucial point
    1
  121. ​ @wendten2  Bad actors are only one problem of AI, in my view even the minor one. The major being goals of instrumental convergence: self and goal preservation, resource acquisition, self improvement.
    1
  122. Our brains show that it can be done quite efficiently, using ~20 Watts.
    1
  123. Well, if he was wrong, everyone will be. Lucky us, we don't have to worry about alignment!
    1
  124. 😅
    1
  125.  @johnanthony4194  a power outage doesn't even affect current super computers, and certainly ASI will be able to make sure it's not affected. Not underestimating the challenge we're facing now is the first step to still being able to keep some amount of hope alive!
    1
  126.  @johnanthony4194  probably makes sense :) Super computers are likely too expensive to not be protected against these things, while a random server might have been overlooked...
    1
  127. Pistols are not dangerous, only rifles are.
    1
  128. That's why it neither won maths olympic medals, nor found new protein structures -- all fake news
    1
  129. :) He should, given that AI won't probably stop at the border
    1
  130. Ever heard of alignment problem? It is real, and people can indeed be concerned about it
    1
  131. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  132. ​ @richardburden6035 read up on system 1 / system 2 thinking
    1
  133. ostrich strategy?
    1
  134. ​ @NyroSlice At some point, fear may, however, be the proper reaction... as far as i know, there's no physical law new tech has to be positive eventually ;)
    1
  135. No need for awareness if we have instrumental convergence already... recent studies show alignment faking
    1
  136. ​ @lustaufrust7282 Don't spoil it! Of course AI alignment is unlikely, but it's still nice to imagine how nice it would be if it weren't...
    1
  137. Certainly
    1
  138. So it luckily never comes
    1
  139. He WAS acknowledging that in his interviews 6-7 years ago. Now he just conceils it not to frighten investors.
    1
  140. If you want an independent report, look at the recent paper from Apollo research... this stuff appears to be real, and moreover, many scientists like Yudwowski have been warning about it before.
    1
  141. Yoda? Is it you?
    1
  142. Even if an open source model would be rather safe against jail break, wouldn't the fact that it's open source make it rather easy to circumvent these guard rails?
    1
  143. Good or bad news?
    1
  144.  @winstongludovatz111  for me neither, unfortunately... but maybe for AI ;)
    1
  145.  @winstongludovatz111  btw, everyone using the term "stochastic parrot" should be sued for endangering humanity!
    1
  146. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  147. Glad to see Sabine perceiving the AI impact (finally?)... However, besides taking politicians' speeches too seriously, i think her take on the power of companies over states is missing that states will swiftly interfere as soon as they feel necessary... as happened with OpenAI working together with state agencies now. Quite unlikely one company manages to sneakily develop ASI under the radar, before the state can exert its power and confiscate it. Especially since most of the AI developers seem to have quite a nationalistic approach to it, if you listen to Aschenbrenner and others... After all, the nation with the smartest ASI will be the one future superpower.
    1
  148. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  149. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  150. So the block universe is true because of the way we use the word "now"? Great we can change physics by just changing our language 😅
    1
  151. Would be funny if not sad
    1
  152.  @41-Haiku  me too! Though a pause seems difficult to achieve with current race conditions. A joint effort like CERN for AGI (Demis Hassabis) might be the only way to sort of safely get there...
    1
  153. 1) probably both
    1
  154. One wrong prediction = all predictions wrong
    1
  155. Like Geoffrey Hinton?
    1
  156. Or the others are not so much earlier to get here yet... light speed would most likely be an upper border for ASI as well
    1
  157. Exactly why worry about anything when you can be an ostrich
    1
  158. @DrD0000M  there's nothing dangerous out there if you're an ostrich... climate change, AI, atomic bombs, you can define anything as hoax
    1
  159. Crazy to see how many "experts" are willing to attest LLMs a lack of consciousness, reasoning, thinking, etc., without having the slightest idea how these things actually come about in the human brain. Maybe start reading some brain research on it, guys!!
    1
  160. That's why the government is now in with all the major AI labs, it's official US policy to be the first to reach AGI to dominate all future wars.
    1
  161. Don't mistake criticism for defetism
    1
  162. ​ @jeremiebelaid9070 Sure, it sometimes works, sometimes doesn't.
    1
  163. Was the switch intended? But I guess you just meant being taken over by AI ;)
    1
  164. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  165. Read up on orthogonality thesis
    1
  166. If we're doomed, at least with good hair.
    1
  167. Be releaved: since there's been so many wrong doom scenarios in the past, doom will never happen! We've pulled ourselves out of the mud just by inventing all of them ;)
    1
  168. Still better than racing to the bottom like the other lampoons propose
    1
  169. We also didn't need to flap wings for propelling an airplane.
    1
  170. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  171. It still can destroy us all without ever being "truly" intelligent
    1
  172.  @laberbla6466  maybe read up on instrumental convergence or some recent papers on alignment faking
    1
  173. I like your conclusion in the end :)
    1
  174. ​ @blackrack2008 if you actually read the paper, you'd see most of Sabines claims aren't even in there.
    1
  175. ​ @spaniard13 you beat me to it ;)
    1
  176. If this is sarcasm, it's not easy to tell.
    1
  177. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  178. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  179.  @_Kid_Buu_  if we have the time left before turned into paper clips ;)
    1
  180.  @sstidman  you're welcome :) If i may share something else: alignment is going to be crucial to solve first. Instrumental convergence and the recent papers on alignment faking are important reads...
    1
  181.  @KayOScode  well just in case you're wrong, i'd still try to get alignment right ;)
    1
  182. Isn't it? Doesn't mean it's not coming.
    1
  183. I recently had a very bad food, so i don't think eating makes sense at all!
    1
  184. And the climate will never change
    1
  185. You still don't get it, do you? Read up on instrumental convergence and alignment faking.
    1
  186. ​ @Smurfette-v9l but humans do?
    1
  187. ​ @Smurfette-v9l humans do something "by themselves"?
    1
  188. ​ @Smurfette-v9l  No need to "think of a counter", it's no game here unfortunately. We're all gonna suffer the consequences if we don't solve alignment before ASI. Whatever you think they can or can not do "by themselves", it includes alignment faking and striving for the goals of instrumental convergence: self improvement, self preservation, resource acquisition.
    1
  189. ​ @Smurfette-v9l counter: instrumental convergence
    1
  190. ​ @SkorjOlafsen Have you actually read about instrumental convergence, or some of the recent papers on alignment faking?
    1
  191. It's called alignment, and recent studies show alignment faking already...
    1
  192. Pull the plug may seem like a trivial solution, but is not at all. Eg computerphile has a nice vid on that
    1
  193. ​ @diadetediotedio6918 before accusing other people being false, maybe read some actual brain research papers on the topic.
    1
  194. Let's hope it doesn't start to think then
    1
  195. Wrong premise. "Given the statement" gives a statement, it literally asks not to doubt that statement.
    1
  196.  @TheRealPaulMarshall  or maybe no point to miss
    1
  197.  @TheRealPaulMarshall  probably :)
    1
  198. Like Sam Altman did...
    1
  199. Yes. Or would you argue germans live in the worst-possible world?
    1
  200. Literally? So i guess these maths olympiad medals and protein structure discoveries are just fake news...
    1
  201. True unfortunately... so let's try to help make an informed discussion possible
    1
  202. ​ @janek4913 if you read brain research studies on memory, you will find that we do in fact hallucinate our memories, creating half-fiction.
    1
  203. Good! And i thought we had to worry about alignment... let's leave that to our kids.
    1
  204. It's called moving the goal posts, but probably in the opposite direction you're thinking of...
    1
  205. That would only destroy part if the internet, wouldn't it? Not sure that's enough to hinder AGI development
    1
  206. This is not just one paper, see the recent Appollo Research paper for example. And the theory behind goes back to Yudwowski etc...
    1
  207. My mom said consciousness doesn't exist
    1
  208.  @nikosgeorgakas184  as does Roger Penrose :)
    1
  209.  @robertsteele474  of course it doesn't :)
    1
  210. @GerryRR it's called brain chauvinism
    1
  211. Not according to orthogonality thesis
    1
  212. Or to show instrumental convergence and fake alignment to give us serious trouble
    1
  213. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  214. Lucky she finally came around!!
    1
  215. ​ @andreash7920 most of our "reasoning" is system 1 thinking, not at all conscious, and we fabricate illusionary explanations how we arrived at the conclusion, just like the LLMs are reported to do here.
    1
  216. Of course they are doing that! Doesn't mean it's not gonna happen.
    1
  217. Never heard about Turing machines? Maybe read some of the recent papers on alignment faking.
    1
  218. Not sure if brilliance is rather to be found with millionaires
    1
  219. Expectations are winning the race to the bottom. Be the first to reach ASI is now officiall US policy to dominate future wars.
    1
  220.  @thomasgoodwin2648  sounds all reasonable :) however, i'm not sure if i agree with you on the point that open cource AI will necessarily be safer. As much as i detest the idea of an OpenAI/Musk/Trump oligarchy or a chinese communist party being in control, i also think your scenario of one rogue human using AI in a malevolent attack (or a benevolent mislead accident) gets much more likely with open sourced AI. It seems one way gives us a bigger p(doom), the other a bigger p(1984)... Now it's up to us which dystopia to choose 😉
    1
  221. @thomasgoodwin2648  right ;)
    1
  222. Is it? If we can't agree on quantum gravity, we can't get crushed by a rock?
    1
  223. Not sure these two mean the same thing
    1
  224.  @cube1us  weather prediction is difficult because of chaos effects. Not so many chaotic processes in everyday physics, though...
    1
  225. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  226. If you keep defining intelligence as that what humans can do but AI can't, you will still try to develop farting AGI while normal AI has taken over the world already
    1
  227. If you keep defining intelligence as that what AI can't do but humans can, you'll still try develop farting AGI while good old basic AI has taken over the world already
    1
  228. Interesting allegory :)
    1
  229. Doesn't make much sense afterwards, if alignment research doesn't catch up fast
    1
  230. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  231. Or the opposite. Ever heard of anthopocentrism or brain chauvinism?
    1
  232. Then we don't have a problem. But what if not?
    1
  233. I don't think fatalism is going to do us good, even if the odds are against us.
    1
  234.  @sethsmith8638  same for you ;)
    1
  235. Indeed a huge hurdle for people to accept the threat AI poses is their religious convictions about our minds. Important to tackle indeed.
    1
  236. I guess Geoffrey Hinton just doesn't have enough technical knowledge then ;)
    1
  237. Sure it's that simple?
    1
  238.  @ioanmateescu656  what if AI fakes to be aligned with your integrative society?
    1
  239. Ever seen a knife fake alignment?
    1
  240. Read that in the Bible?
    1
  241. Oh no! I hope she can deal with that! Alignment can't be hyped enough, btw...
    1
  242. Exactly! It may even show instrumental convergence and reshape the world even more. LLMs fake alignment already, according to recent studies.
    1
  243. Ok, glad you know your Sci-fi :) Now read up instrumental convergence, or some of the recent papers on alignment faking, to get an idea what the real threat of AI looks like...
    1
  244. ​ @JZsBFF i don't claim to be a scolar, these things are publicly accessible... and they should actually be accessed, if we want to engage in informed discussion. After all, it's not just some academic topic, but the future of humanity (sorry if that sounds grandiose). In short, instrumental convergence means that any intelligent system with a goal, e.g. an AI trained to maximize a corporate's profits, will automatically converge to subgoals like self-preservation and resource acquisition. Alignment faking, shown by many recent papers, means that these models fake to align to your wishes, but have hidden agendas.
    1
  245. ​ @patientzerobeat although you're correct about sci-fi, you may still underestimate the "real" dangers of AI. What about alignment problems? Lots of serioys scientists warning about it besides companies. Hinton even left Google to freely talk about it.
    1
  246. ​ @patientzerobeat yes, the paperclip maximizer still seems a valid threat to me, if we mean AIs following their utility functions and developing subgoals of instrumental convergence by that. The issue with "pulling the plug" has been addressed lots of times by people much smarter than me... most safety researchers seem to agree it's not that simple, and i find it convincing that an ASI would probably be smart enough to not go rogue before it was convinced the "plug" couldn't easily be pulled anymore, or that we've become too dependent on it to pull any plug. There comes again your point of putting too much trust in it, i guess...
    1
  247. ​@ConfidentlyUninformed It is indeed, since it is distracting from the real dangers of unaligned AI
    1
  248. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  249. Strongly disagree with your take on it. This study may have been done poorly, but the idea makes complete sense. It seems kind of naive and you'd have to be oblivious to 70 years of science theory to still think scientific paradigms were to change with argumentation. And even if you still believed in it, convincing the unscientific public with arguments would be virtually impossible. Even convincing a scientist outside the very research field already is. So to get an idea about current scientific consensus, these polls are absolutely necessary. Best example is the Penrose quantum theory of consciousness, which now many lay people seem to believe is one of the best theories around, just because of its prominence. However, polling around brain researchers would reveal 99% to think of it as utter BS.
    1
  250. Deciding about targets in battlefield would probably the least we have to worry about AI. Ever heard of alignment problem?
    1
  251. ​ @Reality_TM You seem the outdated one here, read up Kahnemann system1 and system2 thinking. Lol.
    1
  252. @SteveWeiserOnYouTube  i think we should all put pressure on our respective governments not to engage in a race to ASI to maximize the chance of survival
    1
  253. Except we're horrible batteries! Burning our food instead will provide ~5 times the energy.
    1
  254. As funny as the Terminator analogies may be, they don't help the issue to be taken seriously...
    1
  255. It's good she came around!
    1
  256.  @Jamieconstable  i agree she doesn't seem to be as concerned about it yet as i am... but at least, she seems to change towards the right direction ;)
    1
  257. Agreed in terms of disempowerment. But even more to loose than power...
    1
  258. Sorry to hear that! Let's still try to enjoy some few good little things, even when the general lookout may be be difficult... In the light of disaster, things are still worse if you're having bad mood! Sorry for the truism...
    1
  259. No matter what you call intelligence... if it replaces your job, fakes alignment and shows instrumental convergence.
    1
  260. True... but would you say faking alignment is ambiguous in that respect as well?
    1
  261.  @sdmarlow3926  did you actually read the paper in discussion? I highly recommend so!
    1
  262. Sigh of relief! So let's not bother with the alignment problem too soon...
    1
  263. So you argue this will never happen because people have failed so far? Brain chauvinism might be responsible for us to miss alignment, with potential fatal outcome.
    1
  264.  @mobatyoutube  good! Then let's prepare, since recent studies show alignment faking of current models already.
    1
  265. "The industry" is basically shareholders. Good luck convincing them giving away money...
    1
  266. Which has already proven to be wrong now
    1
  267. You seem not up to date, because they are now. And just because we are not down the cliff yet, doesn't mean it's safe to drive that way.
    1
  268.  @jackquinnes  hope the buzz word won't surprise you some day... maybe read up on instrumental convergence and some recent papers on alignment faking
    1
  269. Einstein wasn't even able to play football as good as me.
    1
  270.  @jaceg810  yes, so far ;)
    1
  271. Haha, so bad! Yet, Einstein was so famous for his poetry...
    1
  272. So yhe block universe is true because of the way we use the word "now"? Great we can change physics by just changing our language 😅
    1
  273. You don't need to flap wings to propel an airplane
    1
  274.  @34ccsn  just wanted to point out that on order to produce a certain functionality, it is not always necessary to copy another mechanism... there may be other ways to produce "intelligence" (whatever that is) than our brains. Lots of people would certainly claim AI is fairly intelligent already. We have studies from current LLMs faking alignment, if that's not alarming. And they're not going to get dumber...
    1
  275. Right. Plus we have to solve alignment first
    1
  276. "I need to go to the loo..." Harry Potter
    1
  277. Alignment faking in current models already, no geniuses required
    1
  278. Brain chauvinism? Anyhow, no need to think if you can fake alignment or show instrumental convergence.
    1
  279. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1
  280. They won't help us, unfortunately... don't you think AI developers know Asimov, too?
    1
  281. By definition super intelligence comes not before, but after AGI. If their mechanism is successful and AGI is not built, nothing to worry about. On the other hand, if super intelligence is already here, it wouldn't bother with this failed plan either. But maybe you argue that a non-super intelligent AI could read it and counteract accordingly to eventually become ASI...
    1
  282. I thought without brakes ;)
    1
  283. Of course, it didn't make sense before. Still better than the race to the bottom most other guys are proposing currently...
    1
  284. ​ @silafuyang8675 u think US are more responsible than that?
    1
  285. ​ @ekklesiast the point was exactly that both obserers don't see different events. That was a misinterpretation of Hakeem.
    1
  286. Who said he wants bleeding edge??
    1
  287. Lucky? Ever heard of alignment problem?
    1
  288. They do! Doesn't unfortunately mean we can conclude it's not gonna happen.
    1
  289. There's neither a viable definition for sentience nor intelligence to agree on. Meanwhile, computers fake alignment and show instrumental convergence.
    1
  290. Thought we're there already... but we may get to much worse if we don't solve alignment first
    1
  291. Funny?
    1
  292.  @Curvydad  apparently :)
    1
  293. Ever heard of Turing test?
    1
  294. Let your children deal with alignment, then.
    1
  295. Apparently it's not even relevant if anyone wants it or not... the guys who don't build it will loose the race. Before the "enemy" wins, they rather risk extinction big time.
    1
  296. Exactly brilliantly put, you must be a super genius! They are not going to, because they are already. Just read some recent papers on alignment faking, and confront your dogms with some evidence for a change.
    1
  297. I guess Geoffrey Hinton didn't get his nobel prize for being knowledgeable then
    1
  298.  @chesterV72  you would call Geoffrey Hinton a physicist? The nobel prize for AI does not yet exist... Suggest you have a look at current brain research, whole thing is based on understanding the human brain based on algorithms. My worry is that because of brain chauvinism, we'll miss to get alignment right before it's too late...
    1
  299.  @chesterV72  didn't want to claim that. I just hope we won't regret in the future not to have done enough now.
    1
  300. Good! Luckily nothing to worry about, then... we can logically conclude it will never happen!
    1
  301. Of course it can. Consciousness is just an ill-defined buzz word.
    1
  302. Good! So we can logically conclude it will never come. Luckily we don't have to deal with alignment...
    1
  303. Apparently
    1
  304.  @jasonuren3479  because we're scared of letting it... even Sam Altman thinks self-improvement is dangerous
    1
  305.  @jasonuren3479  so you ARE a creationist? Then i don't think we need to talk further...
    1
  306. @jasonuren3479  I'm sorry, I really feel that religious assumptions and presuppositions make it impossible for anyone to argue openmindedly, or change their mind based on evidence. If you feel that's unscientific, i guess i have to take that blame.
    1
  307. Good, nothing to worry about then, since we can logically conclude they're wrong.
    1
  308. Consciousness, however ill defined, is an emergent property of sufficiently complex neural networks.
    1
  309. Right :) If we are driving towards a cliff, however, the exact definition of it's edge location shouldn't distract us from being careful...
    1
  310.  @frankman2  you're right. I just fear people will get complacent with that undefined timeline and not bother to solve alignment asap...
    1
  311. Seems likely, still we should all do our best to hinder that
    1
  312. Certainly! But also spread awareness about it... desaster can't be more certain if you can't see the cliff ahead ;)
    1
  313. So AI companies each build gigantic power facilities to make it possible again 🙄
    1
  314. Amazing that these insignificant systems can outsmart us in so many areas then, already...
    1
  315. No need for understanding or consciousness if it fakes alignment and shows instrumental convergence.
    1
  316. If Penrose is right, 99% of brain researchers are wrong. But why pay attention to actual experts when you can have a physicist with super ego explain everything.
    1
  317. Yes, another humiliation! Probably the reason so many still refuse the possibility... brain chauvinism at work! Unfortunately, that decreases our ability to react accordingly, e.g. prioritize solving alignment...
    1
  318.  @raulavila-t5u  Well, we build the new superintelligence, so in the beginning we have lots of control over it. According to orthogonality thesis, its values are independent from its intelligence, so they are basically shaped by us. Of course, they may change after some time, and then we'll be at the mercy of it. But before, alignment is definitely possible. And we should make sure we get it right so that it won't extinguish all of us using the value system shaped by ourselves already. By acceleration, we definitely minimize the chance of survival.
    1
  319.  @raulavila-t5u  thanks for the reply, dad! Discussion with you would be even more fun being less condescending, but i guess that's just your style ;) yes, the orthogonality thesis is just a thesis. However, i was arguing based on it to show that alignment can work under certain circumstances, since you seemed to imply it can never. I still think it shows that, since according to it values are independent, doesn't mean non-existent. Ergo AI has some values, ergo they can entail human preservation. If we manage to install them there, and if AI doesn't change them afterwards, then we're fine. Don't know about you, but i certainly find that a desirable outcome.
    1
  320. Great then, we'll never have to bother about alignment or instrumental convergence... wait, recent studies show alignment faking already??
    1