Youtube comments of (@geohotarchive).
-
109
-
102
-
81
-
75
-
63
-
59
-
55
-
54
-
49
-
46
-
44
-
39
-
39
-
37
-
36
-
34
-
Unfortunately, twitter provides only 1280x720 at 60 fps for replay :((( | Follow George on twitter.com/realGeorgeHotz & instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog | Support George by subscribing twitch.tv/subs/georgehotz | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad
Chapters:
00:00:00 intro
00:01:25 twitter media studio
00:02:25 twitter nodes
00:03:20 javascript off
00:03:45 javascript code
00:05:00 front end noob
00:05:20 twimg.com
00:07:02 copy all
00:08:30 graphql
00:09:00 how to go live on twitter
00:09:45 realapi.py
00:10:30 no expectations
00:12:15 requests api
00:12:40 403 unauthorized
00:14:00 binary search by hand
00:16:45 how to do this
00:17:00 hadoop, scala
00:17:50 not logged in
00:20:30 Alex
00:23:10 the following feature couldn't be null
00:25:20 copy paste how to learn
00:27:55 json
00:29:15 python pretty print json
00:30:30 my user id
00:31:02 400 status code
00:31:30 custom twitter client
00:32:10 signup for twitter api
00:32:45 base64 decode
00:34:40 variables features graphql
00:34:45 graphql twitter api
00:35:35 userbyrestid twintproject/twint
00:35:50 No authentication. No API. No limits.
00:37:30 authorization, guest token
00:39:10 user by screen name
00:39:30 same api for internal and public
00:40:25 reading comments
00:41:15 p1atdev/twitterql
00:42:00 trash on twitter
00:42:30 20 engineers enough for running twitter
00:43:30 query violation
00:44:10 tracing trace_id graphql
00:44:30 backend or scala
00:45:00 hadoop cluster with 10k nodes, overengineered
00:45:20 twitter vs other services
00:45:30 youtube storage requirements
00:45:57 number of tweets per day (data)
00:46:35 cdn for media
00:48:30 image ingestion server
00:48:50 understanding the backend of the company
00:49:10 graphql request path
00:50:10 manhattan twitter
00:50:50 20, 40 engineers, why services hard
00:50:50 horizontally scalable, whatsapp
00:51:25 twitter algorithmic feed
00:52:30 400 billion events
00:53:05 vertica
00:54:00 hadoop, hdfs
00:54:55 apache kafka
00:55:05 minikeyvalue
00:55:55 vaping
00:57:15 reading comments
00:57:40 userbyrestid
00:59:05 vs code prettify json
01:01:15 twitter/elephant-bird
01:01:50 prettify
01:03:05 can't access suspends accounts via api
01:04:50 createtweetdownvote
01:06:00 grep operationname
01:08:10 reading comments
01:08:50 graphql introspection
01:09:40 graphql introspection how to
01:11:10 3 spaces
01:12:00 don't know react
01:12:20 backend details
01:13:50 miniray comma ai
01:14:30 hackerrank scala
01:18:40 bad scala skills
01:19:40 twitter scala
01:21:15 python 3.11
01:23:10 scala one liner
01:26:50 king of scala
01:29:30 scala installed
01:29:45 slow scala
01:30:10 reading the timeline
01:30:40 who is the media
01:34:21 more about backend
01:34:45 real money
01:35:05 here to help the cause
01:36:20 free speech on the internet
01:36:45 internet going downhill
01:37:40 core nouns
01:39:20 mysql plugin
01:40:20 write amplification
01:41:25 twitter manhattan open source
01:42:08 manhattan, rocksdb
01:47:10 chrome network not showing requests
01:49:25 guide.json
01:50:00 bed time
01:50:15 conclusion, helping the cause
34
-
33
-
32
-
32
-
31
-
30
-
30
-
30
-
29
-
29
-
28
-
27
-
27
-
27
-
26
-
26
-
25
-
Respect to Tesla team & Elon 00:00:55 & 00:03:49 & 00:04:40 & 00:16:35 | 00:54:30 Dojo George's favorite part of presentation | Buy www.commabody.com & comma.ai/shop/products/three to get your own robot! | Support George by subscribing on twitch.tv/subs/georgehotz | Follow the official youtube.com/commaai & twitch.tv/georgehotz for more livestreams. Stay up to date by following twitter.com/comma_ai & instagram.com/georgehotz | Support comma.ai at comma.ai/shop | Every George Hotz Video🡿 https://www.youtube.com/playlist?list=PLzFUMGbVxlQsVt4r4hSGJ00SW9Tb6VpF4
Chapters:
00:00:00 intro
00:00:55 respect to Tesla & Elon
00:03:25 cringe governance comment
00:03:49 big Elon fan, Axel Nix
00:04:40 no disrespect, real pr, Elon is real
00:05:22 loser definition
00:06:50 weird governance comment
00:07:10 Tesla bot moving
00:07:42 Tesla bot parts
00:07:52 Tesla bot non fluid movement
00:08:10 boston dynamics atlas, boston dynamics great youtube channel
00:09:00 atlas vs optimus differences
00:09:39 weird demos - tesla autopilot segnet
00:09:55 V2 optimus
00:10:15 making hardware before software
00:10:40 competitor to optimus comma body
00:11:35 price points, robotics software problem
00:12:05 degrees of freedom, lifelike
00:13:50 optimus walking
00:14:13 human form, hardware spec
00:14:45 overkill hands, lobster hands
00:15:18 mechanical simulations
00:15:45 end to end learning
00:16:35 disclaimer respect to Tesla
00:17:02 no reinforcement learning in Tesla AI presentation
00:18:40 on device learning
00:19:20 mass production, fancy mechanical engineering
00:19:50 open sourcing optimus
00:20:15 fancy hands, adapting training stuff to body
00:20:35 reward function for cooking, cleaning
00:21:04 why we need robots, I build technology, luddites lose
00:21:58 same hardware, software as autopliot, wernher von braun
00:23:05 ethics statements in papers
00:23:42 model predictive control
00:24:05 trajectory simulators, MuJoCo open source, deepmind
00:25:40 gazebo ros, ros ecosystem, ros 2.0
00:26:25 willow garage, Kittyhawk VTOL
00:28:19 tesla does not believe in end to end
00:28:28 foundation models for geometry and physics
00:28:43 tesla vs openpilot stack
00:30:35 neural networks not used for planning
00:30:45 occupancy network, software lidar
00:31:40 ai compiler & inference, tinygrad
00:32:00 moving off qualcomm, qualcomm sales processes, emailing ceo
00:33:30 some chance on making chips, qualcomm losing robotics, ai
00:34:20 difference between George and other startup guys
00:34:35 two nvidia's high stock price and low price stock nvidia
00:35:50 moore's law is dead, karpathy not working at Tesla, karpathy twitter fun
00:38:30 unnecessary featurespace hand engineered
00:39:20 no human understandable feature space for perception
00:40:20 neural planner, software lidar
00:41:25 odd image before ingestion, image compressor, occupancy network
00:42:35 how much training is needed tesla vs comma
00:43:15 loading data, custom file format, accelerated video library
00:43:45 cheat sheet for speeding up video training
00:44:25 lane net, custom language for lanes, unnecessary just use end to end
00:44:55 NVDEC, T600, .smol file format, IOPS, disable SMT, iommu, torch_cudnn_v8_api_enabled, avx2, cuda graph, channels last
00:49:14 sorry for the engineers working on this, handcoded, super complex
00:49:38 network latency comparison to comma, power usage
00:49:53 compiler, multi-trip reconstiction, hd maps, adding complexity to ground truthing stack
00:50:19 phantom braking in tesla, language models for lanes
00:51:10 language inside neural net, extract meaning with neural net, biggest misunderstanding about how human brain works
00:52:25 how to make it run on chip, autolabel
00:52:55 simulator, simulator world creator, data engine
00:53:20 tesla vs comma training set, comma10k, driving monitoring
00:54:30 dojo favorite part of presentation
00:54:49 voltage regulator, back on the chip, closer = less latency
00:56:20 104 kW, equal to first whole comma compute cluster
00:56:50 cooling, 2x 104 kW per cabinet
00:57:43 sad dojo not open source, cutting edge for now
00:58:05 google lost the cutting edge infra
00:58:55 Tesla will not be a serious cloud company, only Amazon, Microsoft, Google
00:59:24 advantage to open source, holding back by not open-sourcing
01:00:29 kubernetes is google, hadoop yahoo, react facebook, google tensorflow, facebook pytorch
01:02:10 facebook made something, owning meta stock
01:02:24 200 kW per cabinet, don't know why Tesla don't opensource
01:03:10 compiler, cool reduce
01:03:55 nvidia open source, cuda is closed, ai art, EMostaque is a boss
01:05:38 bad subscriber link
01:06:22 how stupid all handcoded stuff is
01:06:40 shkreli hanging out, karpaty stream invitation
01:07:19 attracting talent, serious Elon misunderstandings about ai, karpaty knows ai
01:07:48 not debating destiny, diss track Axel Nix, rap lyrics in court, pat stay rone compliment battle
01:12:15 being unpredictable, escaping the meta, life tips
01:14:03 call options, putting money into losing pie, bad comments from chat, Elon should open source it
01:15:00 nuclear war bets, nord stream pipes, life sucks right now, fan of the recession
01:16:04 john carmack bet, truth after nuclear war, never admiting their wrong, controlled opposition
01:18:42 Palmer Luckey, good people, organisational ability the key ability to have power, persistent stack
01:20:35 Elon messages to Parag, Jack bitcoin maximalist, new twitter
01:22:45 censoring ethereum transactions, mevwatch.info
01:24:58 where wokeness comes from, positive vision of the future
01:26:50 Elon comments on AI day not that inspiring, stock comments, Mars a lot more inspiring
01:29:18 what if we are the bad guys, western world, russia, general shit talking hour
01:30:38 decentralization, crypto people as disgusting as late startup people, no real value to produce there
01:32:15 vitalik fan, thinking twitch chat, women will give you answers, only answers from deep self search, off grid, psychedelics, ubi
01:34:00 birthday tomorrow, elder ring too big to download, best time to be alive, do not vote, alex jones
01:40:10 subscriber only answers, biking around iceland, why people think I want to expand the company, starting a second company, starting a family
01:44:20 apply to comma for work, no part time, no remote, no difference between the genders, believe one thing acting another way = loser
01:46:55 movie where journalists are the bad guys, in a recession truth comes out
01:49:50 we are going to solve self driving cars, the victory does not feel good
01:52:00 stable diffusion elmo bombing the ukrainian pipeline, tinygrad shutout to madebyollin, cerebras big chip tinygrad
25
-
25
-
25
-
23
-
23
-
23
-
22
-
22
-
22
-
21
-
21
-
Update! https://twitter.com/LisaSu/status/1669848494637735936 Thanks for connecting @realGeorgeHotz. Appreciate the work you and tiny corp are doing. We are committed to working with the community and improving our support. More to come on ROCm on @radeon soon. Lots of work ahead but excited about what we can do together.
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | Thank you to @alex_comma for being a great stream assistant! | 00:39:05 desk setup & 01:06:00 bookshelf reveal | From $1499 buy comma.ai/shop/comma-three | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | github.com/NVIDIA/open-gpu-kernel-modules/discussions/494 (PCI-E P2P support on 3090 GPU discussion that George opened) Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:00:48 Alex
00:02:30 light setup
00:04:25 horizontal stream
00:05:15 ASRock Rack ROMED8-2T/BCM, AMD EPYC 7662 64 Core
00:05:50 Samsung RDIMM 8GB DDR4-3200 Sticks
00:06:07 AMD ROCm, goal of tinycorp, bad Nvidia
00:06:25 MLPerf, competitor to Nvidia, AMD developer friendly corporation
00:06:48 AMD RDNA 3 open source, AMD Radeon™ RX 7900 XTX
00:07:15 bitcoin mining case, pcie x16 extenders
00:07:40 motherboard price, cheap gen2 epyc cpus
00:08:18 tinygrad discord, $6000 2019 CPU for $800
00:08:40 $38 8GB RAM sticks, no need for RAM
00:09:25 no sponsors, it's not about money
00:10:30 good mood George
00:10:45 EVGA Power Supply 220-PP-1600-X1, Dell Power Supply
00:11:30 Noctua fans secret 12V fans runs on 5V
00:12:45 no twitch or comma technical support
00:13:00 airflow direction
00:13:30 missed installing the processor, thermal paste
00:14:40 Noctua struggle, fan color
00:15:50 power cables, fully modular power supply
00:16:50 overview of the stream
00:18:15 tinycorp corporate headquarters
00:18:30 Alex's favorite pokemon
00:18:50 pirate flag
00:19:30 tinycorp not hiring, working for tinycorp
00:19:55 big goal for tinycorp, training library competitive with pytorch on AMD hardware
00:20:05 $5000 computer, 5 more GPUs, multimachine mellanox
00:20:57 advantage by using AMD GPUs, nvidia consumer GPUs do not support P2P
00:21:20 sponsorship from AMD
00:22:00 current feuds
00:22:35 $350 EVGA, Dell Server PSU $55
00:23:08 Moving computer back, large rubber feet for computer
00:24:10 Alex's favorite George feud, jack in the box
00:24:55 gucci candle
00:25:20 George always kind to service people
00:26:20 George's haircut, barber
00:28:05 Alex distracted by chat
00:28:40 first computer George build, George does not remember the past
00:29:30 Hiball Lemon Lime
00:30:40 George's flight rules, good planes 787, Airbus A350, A380
00:31:35 video update, being famous, wealth, india
00:31:15 computer overview, 128 lanes of PCIe Gen 4
00:33:50 bad nvidia, price gouging
00:34:30 AMD peer-to-peer support
00:35:00 turning the computer on
00:36:10 quiet computer, hotswap Samsung SSD 990 Pro 2TB
00:38:25 real facts George's height
00:38:35 correct M2 slot
00:39:05 George's desk setup
00:40:28 George is not on drugs, Alex showing George's stuff
00:45:00 Computer starting up, connecting to the monitor, keyboard, mouse
00:50:50 finally boot screen on monitor
00:53:50 tarallini snack, looking for usb stick, maple almond butter filled
00:57:15 $100 paypal to George gmail for drug test to take on stream
00:59:10 computer for crypto
01:01:45 will George use HIP on AMD?
01:04:05 Ubuntu 20.04, 22.04?
01:06:00 book shelf review (Thank you Alex!!!)
01:07:09 making bootable Ubuntu 22.04.2 LTS on Samsung 870 QVO 8TB, Richard Stallman, GNU
01:14:30 booting from USB
01:18:00 wrong click on keyboard, installing ubuntu
01:22:00 computer name, no ubuntu pro, meta stock loss
01:25:25 computer power distorting the camera, ai regulation, bios/boot delay
01:31:00 Alex bringing pizza
01:33:00 sudo update-pciids, lspci, uname -a, cat /proc/cpuinfo, /proc/meminfo
01:37:30 6 GPU cards, LLM on AMD, 1 pcie for network
01:40:00 ROCm vs Cuda, 7GB LLaMA, dma weights from nvme, data parallel
01:41:10 goal 13GB LLaMA float16 across two cards at reasonable speed
01:41:40 stream from macbook
01:43:35 securing the computer, disable ssh password, 50-cloud-init.conf, sudoers
01:53:30 hardware error from APEI Generic Hardware Error Source 516
01:55:08 amd install gpu drivers ubuntu, amdgpu-pro
01:59:00 will make amd drivers good, apt get install amdgpu, amdgpu rocm install
02:05:35 number of devices 0 opencl, RX 7900 XTX not on rocm support list
02:10:30 github.com/NVIDIA/open-gpu-kernel-modules/discussions/494, researching ROCm
02:14:45 tmux config, rocm with open amdgpu driver
02:19:10 ROCclr, hsa-runtime64, AMD biggest fan, how to be friends with George
02:31:40 libhsa-runtime-dev, AMD GPU user experience vs nvidia, clang
02:45:15 mi300, custum llvm needed to build rocm
02:48:10 GPT4 rant, openai lame
02:52:00 building clang, big htop, lag on computer
02:57:00 need more ram, AMD uProf
03:05:00 make install, Executable ld.lld doesn't exist
03:19:45 fatal error: hsakmt.h: No such file or directory
03:23:00 reading the readme, ROCclr, amd_comgr, rpm package requires rpmbuild, dpkg see files
03:35:10 back to hsakmt error, fatal error: X11/Xlib.h, comgr not installed, ROCm-CompilerSupport
03:54:35 docker container, ROCm Installation Guide, amd kernel module
04:03:10 amdgpu-dkms, build amdgpu driver module
04:05:22 repo.radeon.com/rocm/apt/.apt_5.5
04:10:40 ubuntu 23.04, updating kernel
04:18:50 it was the kernel issue
04:22:30 add to group ubuntu
04:23:10 installing tinygrad
04:25:00 tinygrad llama.py
04:25:30 it did not build, kernel upgrade worked
04:28:25 all you need for tinygrad is opencl
04:28:40 90% of a problem was the old kernel, chatbots inside llama, macbook 10gbe network adapter
04:34:25 tinygrad llama up and running on amd RX 7900 XTX, optimization needed
21
-
21
-
21
-
20
-
20
-
20
-
20
-
Read twitter.com/realGeorgeHotz/status/1770238969663594607 | github.com/geohot/7900xtx
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1050 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:00:40 blog post
00:02:00 AMD open source driver, bugs
00:04:25 pre-orders, cheapETH, George changes jobs quickly
00:06:30 why no one use AMD? kernel space and user space
00:08:25 not understanding, business platitudes, unpaid QA team
00:09:40 bugs are not in the driver, kernel driver does not talk to the hardware
00:12:45 Nvidia GPU System Processor and AMD MicroEngine Scheduler (MES)
00:13:50 llvm amdgpu launching a kernel, firmware blobs
00:17:00 amd on mlperf
00:18:08 AMDGPU kernel driver change log, amd firmware will never get better unless
00:19:05 companies do not work how Elon companies work, they don't make good things
00:19:20 people who succeed in companies do not care
00:21:15 believing wrong things, change, correcting
00:21:55 better comma.ai, scaling comma.ai models, throwing money at comma.ai, $1 mil per month
00:24:05 fundamentally correct, life wasted on wrong things
00:24:35 kernel dispatch on AMD, AMD Command Processor (CP)
00:26:08 technically wrong
00:27:49 deadlock, root causing
00:29:10 why this firmware will never get better
00:30:40 props to AMD for releasing this, fixing the issues
00:31:20 MES correct description
00:31:50 normal bug
00:32:40 oh god, appreciation for software development, horrifying solution, why is this a flag
00:33:40 this why AMD drivers have been shit for 25 years
00:34:07 should be an if statement in the allocator not a flag
00:35:20 offer to AMD, downside risk, worries about bad PR
00:38:40 sony, suing someone, led by emotions
00:41:10 AMD instinct, Nvidia generations, Apple monopoly
00:45:45 Nvidia blackwell chip, RDNA 4, Dojo
00:48:50 tenstorrent software stack, mlperf
00:50:45 removal of turing completeness
00:51:00 Jim Keller
00:52:26 AMD will not open source 7900XTX firmware, Elon can execute
00:53:30 12xIntel Arc, Groq LPU scam demo, Nvidia king
00:55:00 fixing AMD culture hard
00:55:40 fixing qualcomm firing the whole sales division
00:57:05 don't want to work with George, Elon genius
00:58:44 JH7110, RK3588, comma solved the qualcomm situation
01:00:25 oneplus folding phone money back
01:02:50 AMD situation stressing George out, clean ubuntu install
01:04:40 hsakmt debug level, business decisions
01:07:50 amd dynamic debug how to
01:10:10 hip graph vs cuda graph
01:13:40 loading static libs with ctypes, libdrm
01:20:20 moving on from AMD
01:23:55 hsa_queue_create
01:25:20 GPU user spaces very complex
01:27:00 how many exploits with George find
01:33:50 building tiny boxes, AMD open source timeline
01:37:20 user space crash
01:41:40 ctypes uint8_t pointer from address
01:48:30 random queue
01:55:10 how do you fuzz things
01:55:35 teaching hacking, what is this thing
02:00:40 messages from the kernel, ubuntu rebuilding kernel
02:06:40 heven, are you a good person
02:09:10 unload, load, module,
02:09:25 hacking is 50% setting up your env, knowing a lot, binary search by hand
02:18:20 amd gpu smu mes, phoronix are legit, gfx1100
02:26:05 amd firmware architecture
02:26:30 kernel driver, tinygrad goal minimal all the way to the hardware
02:29:00 RDNA3 firmware architecture
02:32:18 Lana_Lux raid!
02:33:40 recap of the stream, driver not opensource
02:34:30 firmware closed source, signed
02:36:20 mistake was talking with AMD, unpaid QA team
02:37:05 work with the product, ordering food
02:39:40 MES support
02:40:20 if AMD open source anything by monday it's over, hacking is addictive
02:44:20 it's not like qualcomm
02:46:20 what did AMD give to George, talking at AMD
02:52:15 Ghidra, food
03:11:44 verbose
03:24:30 size_bytes, ucode, assembler and disassembler
03:35:45 F32 disassembler, f32 microcode
03:43:10 0 technical info from AMD, Sam Altman
03:50:20 giving up AMD GPUs, reverse engineering not sustainable
03:58:40 Evil AMD plot, un-paid QA team
04:02:00 break
04:08:50 amd gpu, psp, dcn
04:17:00 working on interoperability, nvidia consumer cards for commercial purposes
04:22:50 sas, ta, sos, amd vcn, sdma
04:29:40 secretive amd, imu gc, this is dumb
04:33:40 muted
04:50:00 build one module in linux kernel tree, why reboot
05:37:20 what would happen if we corrupt it
05:47:50 amd psp function, amd gpu bootloader
05:53:00 attacking signature check, amdgpu disable dma debug
06:14:25 talking to intel, no business development people
06:15:25 shill for comma, toyota pd0wm
06:20:00 better understanding of security model
06:26:10 tenstorrent people, rdna3 emulator tinygrad
06:34:00 break
06:41:40 7900XTX vs 4090 compute, nvidia nerfing the firmware, market forces
06:50:30 HSA_STATUS_ERROR_INVALID_ARGUMENT, AMD release docs
07:09:00 ROCm does not work, hard decision, let's fix all of this, crypto miner
07:15:25 nvidia bugs, docs to drive the gpu, amd_hip_bfloat16
07:47:10 message to AMD, amd imu gc, amd stock, gpu market share, nvidia, elon musk
08:04:10 boeing, british company, brazil, arm ai chip, google, jensen huang, sam altman
20
-
20
-
Update! https://twitter.com/LisaSu/status/1669848494637735936 Thanks for connecting @realGeorgeHotz. Appreciate the work you and tiny corp are doing. We are committed to working with the community and improving our support. More to come on ROCm on @radeon soon. Lots of work ahead but excited about what we can do together.
tiny corp is hiring! -> tinygrad.org | Submit good github.com/geohot/tinygrad/pulls | AMD does not care about their kernel drivers 00:02:45 | 00:07:53 | 00:08:32 | 00:09:55 | 00:12:05 | let them know how bad their kernel driver is twitter.com/amd | twitter.com/AMDRadeon | twitter.com/lisasu | From $1499 buy comma.ai/shop/comma-three | Check out and star github.com/geohot/tinygrad
Issues submitted by George to AMD:
https://github.com/RadeonOpenCompute/ROCm/issues/2198 ([Driver] BUG: unable to handle page fault for address: ffffa7c13eaffff8)
https://github.com/RadeonOpenCompute/ROCm/issues/2205 ([Driver] amdgpu 0000:c3:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:12 pasid:32773, for process rocm-bandwidth- pid 8285 thread rocm-bandwidth- pid 8285))
https://github.com/RadeonOpenCompute/ROCm/issues/2204 ([Driver] WARNING: CPU: 2 PID: 479 at drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c:471 amdgpu_bo_move+0x261/0x2e0 [amdgpu])
https://github.com/RadeonOpenCompute/ROCm/issues/2197 ([Driver] WARNING: CPU: 96 PID: 903 at drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c:600 amdgpu_irq_put+0x9f/0xb0 [amdgpu])
https://github.com/RadeonOpenCompute/ROCm/issues/2196 ([Driver] ERROR MES failed to response msg=2)
https://news.ycombinator.com/item?id=36189705
https://old.reddit.com/r/Amd/comments/140uct5/geohot_giving_up_on_amd_gpus_for_compute/
tinygrad/tinycorp archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQsh0fFZ2QKOBY25lz04A3hi
Chapters:
00:00:00 intro
00:01:20 motherboard
00:02:06 we giving up on AMD
00:02:45 AMD drivers, bad AMD patch
00:04:35 Intel vs AMD driver patches
00:07:17 3 editions of tinybox
00:07:53 AMD driver will never be good
00:08:32 kernel panic not acceptable, contact at AMD, capture mindshare
00:09:55 should I buy AMD or Intel for ML
00:12:05 top priority for gpu division at AMD
00:12:55 Intel stable kernel driver, ROCm
00:13:25 did not expect running AMD demo app in loop to crash the system
00:13:40 Lisa Su respond to email, Jensen Huang
00:14:10 would fix AMD, no docs, short AMD
00:16:35 AMD does not care about their driver
00:17:37 Pat Gelsinger trying to turn around Intel
00:18:25 plan what to do, get tinygrad on mlperf
00:19:13 3090 vs 4090, NVLink, price/perf/power, jetson orin
00:20:25 how to fix AMD graphic division, callture
00:21:20 Lisa Su do you want to compete against Nvidia
00:22:05 meditation, Linus adding AMD GPU driver in kernel
00:25:54 user space, driver need to be stable, internship at AMD
00:27:23 bad culture, mediocre work not accepted, middle management
00:28:25 pytorch team, facebook, fb product stupid, twitter bad development environment
00:29:25 fire middle management tomorrow, Yann LeCun, Geoffrey Hinton
00:30:12 whisper openai, ai safety, open ai bad practices, Sam Altman
00:31:22 brand vs performance advertising, do not work at open ai Andrej Karpathy
00:34:00 whisper.py, locked tinygrad bounties, Twitter needed help, trust Elon, neurolink
00:36:00 AMD kernel driver 3 people team, hundred middle management, comma IPO
00:38:45 finding the whisper model
00:44:18 apology for Nvidia, how much we love Nvidia
00:45:50 no safety checker, no alignment, don't talk to the government
00:54:00 writing whisper.py
00:59:00 AI Drake, Lil Mabu, Lil Windex, music generator
01:14:40 attention blocks
01:25:40 class Whisper, bach, chiptune
01:37:20 load, prep audio
01:41:37 goal by the end of the weekend, talking and words on screen
01:43:00 friendship with computers, 2030 girlfriend company, ai safety
01:47:50 Sam Altman vibe, normies thoughts on ai
01:49:00 Ford Foundation, inequality, power
01:52:23 Industrial Society and Its Future
02:03:00 life of homeless, oligarchy and not the oligarchy, us gov budget
02:07:30 pizza, west needs god, jina, conspiracy theories, NGOs
02:14:00 getting cancelled, military, taxation
02:17:15 colonies on mars, moon, fusion, hyperloops, homeless
02:20:20 challenging social contract, better soda, eugenics
02:25:00 tinygrad welcomes good pull requests, good genes
02:30:15 based department, being on plane guided by morons
02:38:00 reading the bible, cultural catholic, quran
02:45:45 pizza, power over nature, people
02:50:55 England, losing all the money
02:52:55 why Elon got so rich, worked hard, smart, did what it takes
02:54:30 targeted campaign attack against Elon, Kanye West, Biden, Podcasts
03:05:00 the primagen, dr disrespect, Martin Shkreli
03:11:00 ADHD, tweakers, alcoholics
03:17:00 sine wave, mozart
03:21:00 qkv_attention
03:39:30 TextDecoder
03:46:45 languages
03:54:20 tiktoken
03:59:35 good contributors in tinygrad
04:00:50 using comma three whisper llama
04:02:20 attention block
04:03:20 disktensor speed on mac
04:06:45 dataset audio
04:11:30 visualizing sound
04:12:45 load_state_dict
04:18:00 AssertionError can't expand
04:19:30 code working, love transformers
04:21:10 commit whispering
04:23:30 python capture audio
04:31:10 hello twitch can you hear me
04:32:35 how great is whisper
04:43:20 pyaudio buffer size
04:47:00 can't pickle local object
04:48:45 real time speech test, changing offset
04:54:35 true dat, working, laggy
04:55:20 key value cache
04:57:45 testing and finding bugs
05:04:44 not working anymore
05:10:00 it works
05:11:25 ai safety
05:12:40 piping whisper into llama, output llama into say, sexy gan
05:13:40 fixing bugs
05:24:40 multiple times decoding
05:31:00 need to merge speedups for realtime
05:31:40 thank you for watching
05:32:00 prison hierarchy
19
-
19
-
19
-
19
-
16:15 !!!WARNING loud noise!!!
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
https://youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | https://youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00 intro
01:04 crisis of meaning
02:15 bright colors, 2024
03:05 winning
05:40 harvard over, 1971
07:18 how do we win?
07:55 earth, sun, space
09:10 2010s - FANG
09:45 farming, technology
11:00 Africa
11:55 liberal arts, humanities
12:35 marxists
13:10 Alex
14:15 internet commenters, systemic racism
15:25 Alex, 1$ dollars
16:15 !!!WARNING loud noise!!! money rain
16:50 housing market
18:50 technology, printing houses
19:15 crypto
20:55 freedom, cheap
21:20 war, cold wars
23:00 proxy wars, drone war
24:00 moon, space war
24:47 marc andreessen
25:30 christianity, evolution is real
26:15 wokeism, churches
27:07 embracing technology
27:25 conservatives, luddites
28:45 decel vs accel
30:20 energy use, population
31:25 opt out, preventing tech
32:30 flourishing, deceleration
33:10 meaning, being in time, critique of pure reason
34:12 winning over nature
35:35 definition of racism
36:45 god
38:15 pro tech future
38:35 fixing housing, land owners, renters
40:25 trust in us government over time
41:05 AOC
41:45 uniform housing projects, nature
42:40 1950 appliances, transformers
42:55 mosfet, efficiency
46:30 real liberal majors working on meaning
47:00 what can engineers do
48:40 FANG, john deere
49:20 stand up for what you believe in
50:15 Elon interactions, no filter
50:30 Mark Zuckerberg, Jeff Bezos, Jack Dorsey
51:20 having fun, if you don't love technology quick technology
51:50 progressivism, AI, doom
52:35 heat death of the universe
53:25 technology challenges
53:52 benefits of technology
54:45 other browsers for iOS, chrome webview
55:40 regulatory capture
55:50 thinking of technology, mosfet
56:40 happy 2024 new year!
57:00 solving meaning, alternative to harvard
19
-
18
-
18
-
18
-
18
-
17
-
17
-
17
-
17
-
16
-
16
-
16
-
16
-
16
-
16
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | George is being nice to noobs. Will the noobs be nice to George and subscribe on https://twitch.tv/subs/georgehotz ? | Code is available on https://github.com/geohot/tinygrad/tree/gemm | https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:10 quiet computer
00:02:10 no adderall joke
00:03:00 noob day
00:04:00 how to multiply a matrix
00:06:00 big matrix
00:07:10 j_blow raid George
00:07:45 how much compute is matrix multiplication
00:08:25 how to do matrix multiplication
00:09:50 FLOPS, time.monotonic
00:11:50 SI prefixes
00:14:00 hype titles, freedom units
00:15:00 CPU TFLOP/S, threadripper, ryzen
00:17:55 AMD Radeon RX 6900 XT
00:18:35 SGEMM, DGEMM, MADDNESS
00:20:10 github.com/dblalock/bolt
00:21:50 Theoretical GFLOPS
00:23:30 Same performance in C
00:26:30 multiply a matrix in C
00:28:05 timer in C
00:33:45 python,C performance, tiling
00:35:00 today's lesson (cache aware algorithm)
00:35:50 order of for loops
00:37:30 still slow
00:44:55 avx2 instructions c
00:49:30 FMA3, VFMADD
00:50:50 don't use strassen, cpu instructions, FMA
00:56:00 avx2 only about integers, we need FMA, thank you @paranon1
00:57:40 real
00:59:00 segmentation fault, align(64)
01:04:00 is that wrong?
01:09:00 still slow, threads
01:11:10 1 thread speed
01:14:30 visualizing what is it doing
01:15:20 _m256 init to 0, _mm256_fmadd_ps
01:23:04 time for printf's
01:26:00 short break, should we play wonderwall on a guitar
01:27:20 tweet about downsizing apartments
01:27:50 gdb
01:29:00 this is illegal, suing clang
01:30:29 not suing clang
01:31:30 that one is always 0 that can't be right
01:33:30 whiteboard missing
01:35:10 gemm tinygrad branch
01:37:25 internet broken
01:38:10 extract _m256, a bit faster
01:42:45 tracking down segmentation fault
01:43:55 data not aligned, dumbass
01:44:40 it's always your fault
01:45:10 good speed, alignment bytes
01:48:30 fan spinup
01:50:20 zen microarchitecture
01:54:30 something about this is slow
01:58:00 another way to do this
02:07:50 without and with ffast-math
02:12:30 too early for optimization
02:22:50 visualizing
02:27:20 will work but stupid
02:32:10 number of ymm registers, ymm matmul
02:35:40 not getting the numpy performance
02:38:20 slower, second fma unit,
02:39:40 it's faster now, don't trust -O3
02:44:40 lag on stream, turning off the dryer
02:47:15 hard to make faster
02:52:20 profile cache stalls x86
02:58:20 that loop looks fast
03:06:05 cpu cache sizes
03:15:50 cache coherence, how is it slower
03:22:38 short break
03:28:20 tweet about adderall, drug test, people without skills
03:32:20 zen microarchitecture, optimization
03:39:35 L1 only 32 kB
03:46:40 we are trying to do fast matrix multiply
03:54:40 openblas haswell gemm
03:59:45 online whiteboard
04:02:15 no sarcasm allowed subscriber get's a timeout
04:02:50 removing code, _mm256_broadcast_ss
04:16:45 just persistent
04:19:00 whiteboard time, better understanding
04:25:40 don't want to reorded matrix
04:32:00 strassen = ban, wrong and slow
04:38:25 coherent meaning, access memory in order better
04:46:15 same number of fma as broadcasts
04:48:20 it's fast now
04:51:05 how to get the same fma adds
04:54:15 beating numpy
05:00:45 multithreading check, max clock, pragma
05:06:35 theoretical maximum on cpu
05:12:40 crushing numpy, real threads in C
05:22:10 double the speed, even more speed
05:24:10 overhead, semaphore
05:28:40 we cheated
05:29:30 no TFLOP
05:43:50 Alex is home, stupid question timeout
05:49:00 beautiful htop, throttling
05:51:40 theoretical maximum
05:53:40 cpu power draw
05:57:30 cpu temperature
06:01:00 disable throttling
16
-
15
-
15
-
15
-
15
-
15
-
15
-
15
-
15
-
14
-
14
-
14
-
14
-
14
-
14
-
14
-
14
-
14
-
14
-
Documentation for the 7900XTX -> github.com/geohot/7900xtx PRs (Pull requests) welcome! | phrack.org/issues/7/3.html | www.catb.org/~esr/faqs/hacker-howto.html | github.com/fail0verflow/radeon-tools | github.com/NationalSecurityAgency/ghidra | www.cs.sfu.ca/~ashriram/Courses/CS295/assets/notebooks/RISCV/RISCV_CARD.pdf
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org/#tinybox | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:01:50 meditation
00:04:10 questions, AQL packets, radeon southern islands
00:06:00 two code paths
00:10:10 amd vulkan, dispatch_indirect
00:18:30 packet queue = AQL packets, amdgpu eop
00:22:10 AQL_CONTROL
00:30:55 blog part 3
00:37:00 resnet training job to crash
00:40:00 DEBUG=2
00:43:00 getting rid of AQL
00:45:30 engineer wrote something specific and project manager changed the words
00:47:10 selected pipe, queue does not matter
00:48:10 different queues
00:50:15 PM4, PGM_RSRC2
00:56:10 EVENT_WRITE.partial_flush comment
00:58:40 more testing of KFD, moving away from AQL
01:01:20 gpu_driver.cc
01:05:30 does someone want's to write PM4?
01:09:45 when I make the GPU
01:10:20 nvidia and amd very similar
01:10:50 firmware absolutely simple as possible, how easy is updating different tier of software
01:11:20 why AQL exists?
01:19:45 we got the crash
01:29:00 double banned, umr, smc bits read_smc
01:33:00 30% chance PM4 does not have bugs
01:33:40 rosetta stone style translation F32 and RS64
01:38:10 radeon_tools marcan
01:38:20 is AMD going to open source anything useful
01:40:40 ghost locomotion, argo ai
01:42:40 argo the next thing, can everyone stop, cleaning solution
01:44:30 can we all just be more honest
01:45:35 AMD NDA, infringing patents,
01:52:15 being afraid of getting sued
01:55:30 DISPATCH_DIRECT packet
02:00:40 amd gpus codenames, fiji
02:02:55 linux-firmware license, not a lawyer
02:06:10 nvidia gsp license, cuda license no datacenter use
02:07:55 risk averseness of people
02:10:40 linux-firmware
02:20:40 rosetta stone, F32, RS64
02:37:00 launching specific kernel
02:40:40 first GPU to support AQL
02:46:50 NSA, Ghidra, IDA Pro, Russian tools
02:48:30 respect NSA github
02:48:50 NSA better at open source then AMD
02:49:14 NSA more open source than AMD
02:49:30 backdoors
02:54:20 drake rosetta stone song
02:57:30 credit marcan for writing the disassembler
03:15:45 binvis
03:19:30 fail0verflow radeon talk
03:39:25 F32
03:48:30 gfx1101
03:53:20 amd rs64
04:10:15 gfx rs64
04:17:55 python os lseek
04:23:15 amdgpu_regs, mmio
04:34:35 decompiling now
04:38:50 all the different addresses
04:42:30 dump_ip.c, upset writing C
04:48:25 c example mmap fd
04:51:18 read without advancing file pointer
05:03:20 bugs in the dispatch
05:20:25 frequency analysis on code
05:31:45 next 4 bytes are register
05:47:00 amd_emu_mode
05:57:35 we broke the gpu, what did we learn today?
06:00:05 I have given you the tools you need to solve the problem
06:05:40 switch to nvidia if you have AMD crashes
06:09:05 risc-v opcode groups
06:12:30 capstone risc-v disassembler
06:18:20 skip instructions, CS_ARCH_RISCV mode
06:22:20 bad at ghidra
06:23:30 don't spoil the risc-v
06:33:00 if labeled easier to read, launch instructions, almost just compile
06:33:55 figuring out the first instruction
06:38:30 load this register map
06:52:00 ghidra load map file
07:35:20 jump table that goes there, some CP function
07:54:25 cmd 3 packets pm4
08:05:00 it's cool that is risc-v
08:23:40 now we can know if AMD open sources anything real
08:29:50 I bet this did not support AQL
08:33:50 we need to stup
08:42:45 I have found why there is so many bugs in this
08:49:15 back in the days of bonaire
08:56:50 HSA was a marketing campaign
08:57:30 internet issues
14
-
14
-
Update! https://twitter.com/LisaSu/status/1669848494637735936 Thanks for connecting @realGeorgeHotz. Appreciate the work you and tiny corp are doing. We are committed to working with the community and improving our support. More to come on ROCm on @radeon soon. Lots of work ahead but excited about what we can do together.
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | AMD is hiring Open-Source Linux/Mesa Developer https://www.phoronix.com/news/AMD-Another-Mesa-Dev-2023 | Reach out/tweet to AMD about how bad their drivers are via all the channels twitter.com/amd | twitter.com/AMDRadeon | twitter.com/lisasu | #sponsortinycorp George will fix the drivers | Crash error https://gist.github.com/geohot/59ac46f3740647d2fdbcb2c8f29a6920 | From $1499 buy comma.ai/shop/comma-three | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters Part 1:
00:00:00 intro
00:00:20 red team good green team bad, rocm bugs, tinygrad benchmark
00:01:30 7900xtx true flops, mojo, off by factor of 4, opencl
00:03:00 shader, bad assembler, v_dual_fnac_f32, ashrrev_i32_e32, arithmetic shift right
00:04:30 epyc, nice things about amd gpu, rocm-bandwith-test, bidirectional peak copy bandwidth
00:05:28 amd driver kernel NULL pointer reference, reboot, renting a modem from cox
00:07:00 terrible assembly code, not optimized, cheapETH not coming back
00:09:05 how gpu work, warps, simd machines, single instruction, multiple data
00:09:49 ADD,MUL=DSP, ADD,MUL,LOAD,STORE=GPU, ADD,MUL,STORE,COMPARE,BRANCH=CPU, load, not a load
00:10:55 WGP=2CU, warp - single instruction across 32, dual issue VALU
00:12:10 playing assembly in tinygrad, extra rocm, rdna3, enable_early_exec, bad driver, kernel bad
00:13:05 staging worse, the only kernel and dkms that only kind of works
00:13:35 compiling empty program and getting the binary out, stuff to compile rocm
00:14:55 llvm-objdump -d cc.elf, assembler in tinygrad asm.py, llvm-mc - LLVM Machine Code Playground, obj file, lld to link it
00:17:04 failed to get code object ISA name, if you have something to send via email to geohot, llvm-readelf cc.elf
00:18:25 assemble a shader from RDNA3 language, thanks amd, RDNA3, tensor cores float32 throughput vs non tensor cores, llvm target to amdgpu
00:19:45 amdgpu triple, llvm target triples, failed to get metadata
00:21:55 wavefront_size, how wide simd path is, NT_AMDGPU_METADATA, Message Pack [MsgPack] binary format
00:23:45 searching over all rocm open source code, thanks amd, amd_comgr_get_data_metadata
00:25:35 kernel crash again, rmmod amdgpu, reseting pci device
00:28:15 metadata generated, wavefront_size 128, failed to read ip discovery
00:30:35 git commit assembler maybe, removing the bus, computer rebooted
00:32:20 driver bad, dkms latest driver, stock ubuntu 22.04 kernel, connecting bmc
00:33:50 your connection is not private, thisisunsafe, connecting to asrockrack mb
00:35:20 event_log, pci serr critical error, powered off, no signal, vbios
00:36:50 debugfs to restart gpu, bmc remote kvm
00:40:00 pci serr, no boot delay, amd experience, pcie calibration
00:43:30 computer now in the case, checking for pcie errors
00:45:22 if this happens again we buy Intel+Nvidia, I really want like AMD
00:46:00 asm.py OUT_OF_HOST_MEMORY, normal crash, AMD_LOG_LEVEL=4, roadata
00:54:20 llvm, gnu assembly export symbol, .global
00:57:50 pyopencl does not bundle an opencl, symbol code.kd, word gnu assembler
01:04:40 OBJECT FUNC symbol type assembler, .type code.kd, STT_OBJECT, it works
01:09:00 why AMD? red team, creating a buffer, tinygrad and amd
01:11:25 args, running our custom assembly code, how to test, infinite loop
01:15:00 memory access fault by GPU node-1
01:23:10 missing an end program, thanks xXthebigfoXx
01:26:30 working, test number 2,commit custom asm, crashed gpu driver
01:28:00 how to reset gpu, echoing a string, one works one hangs, should I put the gpus in quiet
01:31:40 learning about what each command does, it's not the gpu it's the driver crashing
01:33:40 GLOBAL_STORE_B32, S_LOAD_B64, how complex is the driver?, writing our own driver, pro driver
01:37:15 does AMD care about this? no repeatable way to crash, why driver so bad
01:41:50 not one crash with CUDA did anything crashed the kernel driver
01:42:55 bad mb, driver suggestion from chat, everything new in pc
01:46:30 frustrating AMD, now on quiet, kernel 5.15.0-71, installed single 7900
01:47:50 updating rocm to 5.5, rocm_clone.sh, new computer, graveyard of gpus
01:49:30 if anyone from AMD is watching this, contract George, nvidia P2P not enabled
01:50:35 AMD should sponsor tinycorp, 7 gpu cards, rocm setup, llvm
01:52:50 no space in case for two gpus, rocm llvm, how hard is to write a gpu driver for compute, AMD do you want to be a competitor to Nvidia? sponsor tinycorp
01:55:10 you have to go through process, can make your card good
01:56:00 AMD call George, Lisa Su reasonable person, Nvidia 5x bigger company vs AMD and you make CPUs, hardcoded jammy
02:02:40 AMD would be worth more than Nvidia if they fixed their drivers
02:02:50 no matter how much I mess up in userspace you should never pagefault in kernel that crashes the driver
02:03:20 need dropping 0days on AMD to get them to listen, short AMD, exploit their driver, make money
02:03:36 will fix drivers for AMD, make compute stable on your own AMD CPU
02:03:55 pirate flag, pride flag, american flag, that is why drivers don't work
02:04:40 making money in good sustainable way, can't search twitter anymore without login why Elon?
02:04:55 I will fix AMD drivers, LBTQ+ friendly stream
02:07:00 nvidia makes more money on datacenter then gaming, libamdoci-orca64, more stable on this PC
02:13:35 AMD care, we love you, if the driver is not stable, I want to get to the bottom of this
02:16:50 we have a mystery to solve, why is it crappy on that computer, nvidia-nvlink
02:17:50 systemd disable nvidia persistenced, putting the gpu in epyc computer
02:19:40 could the pci port be bad, not enough pci lanes
02:23:20 product for tinycorp, home DGX, 5 GPU in box, home trainer box, 6 boxes to pro mb is 30 GPUs
02:26:45 TinyAMD driver selling to AMD, writing a fuzzer, fuzz.py
02:31:55 same crash, rocm-smi gpureset, minimum reproduction for this crash
02:36:30 map_queues_cpsch, dqm lock has to be locked before calling this function
02:38:48 upstream driver, staging driver, whole linux kernel, purge, arch linux not a solution
02:43:40 map_queues_cpsch, arch rocm, upset with amd not arch, solution is nvidia
02:45:40 gpu P2P disabled, cuda ioctl SM_ISSUE_RATE_MODIFIER_PARAMS, ada nvidia architecture, FP16 vs FP32 accumulate
02:47:30 AMD call George, AMD will become a competitor for deep learning, crash by just loops
02:48:25 if you want to suffer and struggle buy AMD, AMD driver crashes, amd hipcc apt-get
02:53:00 new crash, tinygrad using pyopencl, normal ubuntu, drivers just not work, linux 6.3, kernel crashes
02:57:00 3080 TI P2P, return AMD GPU?, jailbreak for Nvidia, Nvidia not that bad
03:02:30 Nvidia prohibit data center use, how make the companies to listen, AMD Baidu, who won in long run Apple or Jailbreak
03:04:00 build something better than they have, companies respond to fear, still use an iPhone
03:04:45 Android spyware, starting AI chip company, write AMD driver first, Jim Keller, tenstorrent can't buy card
03:06:00 as a company make drivers that work on AMD GPUs
03:07:00 company drivers better than nvidia it's Apple, apple 2.75T, nvidia 700B, AMD 144B, marketcap show the stability of their drivers
03:08:30 apple still not scummy yet, nvidia scummy, no competition, apple value of their brand, AMD good hardware
03:10:10 apple shader code clean, intel 129B crappy drivers
03:12:20 buying chips from qualcomm, will make chips and drivers that are stable, 4090 is a good card
03:12:55 advice to nvidia, enable P2P, disable mb dispach make your GPU 2x fast, remove banning from use in datacenter
03:13:35 Jensen who are you listening to?, 4080 stupid, Google TPU rant, Stevey's Google Platforms Rant
03:15:00 nvidia cards by the deal per TFLOP, do not buy 4080, depends on the chip
03:16:30 Alex home, amdgpu driver github, qualcomm gpu drivers good, kernel 6.3
03:26:30 booted the computer and it crashed, linux firmware amd install, amd website driver broken driver
03:36:15 rebuild initramfs, what linux for rocm, stable until it crashed, vertical screen for the long errors
03:43:46 normal install, RDNA2 card test, rocm 5.5
03:49:30 why apple does not use nvidia, bad sodering profile, Steve Jobs said we are never going to use nvidia GPU again
03:53:15 AMD EPYC cpu nice because no need for drivers, RDNA2 GPU, dual-issue elu commands on rdna 3
04:02:25 gpu-burn amd, mes might be in unrecoverable state, should we return AMD cards
04:07:25 AMD if you reach out to George about fixing this drivers he will keep the cards, need to make chips
04:08:00 AMD just show that you care about supporting this, AMD drivers don't work, we still haven't tried the latest driver
04:09:00 ubuntu build kernel from source, menuconfig, how did Linus let this crap in the kernel
04:12:25 7900xtx been out for a year and still broken, expect stable drivers
04:15:50 just building kernels and llvm instead of building AMD RDNA, menuconfig, driver build as a module,
13
-
Thank you to AMD KFD architect Felix Kuehling!!! -> github.com/tinygrad/tinygrad/discussions/4086 | Observations > gist.github.com/fxkamd/ffd02d66a2863e444ec208ea4f3adc48
Documentation for the 7900XTX -> github.com/geohot/7900xtx PRs welcome! | phrack.org/issues/7/3.html | www.catb.org/~esr/faqs/hacker-howto.html | www.cs.unc.edu/~otternes/papers/rtns2021.pdf | www.x.org/docs/AMD/old/R5xx_Acceleration_v1.2.pdf
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org/#tinybox | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:01:10 AMD open source announcement
00:01:50 AMD does not understand the spirit of open source
00:02:25 GPU quality, PR, praise AMD when they release source code
00:02:45 what will AMD open source?
00:03:45 AMD employees banned from tinygrad discord
00:04:14 KFD architect AMD Felix Kuehling
00:06:00 PR is bad, comma marketing safety
00:07:40 the spirit of open source, GPU quality will make money long term
00:08:25 ghost locomotion, $217 million gone
00:08:50 comma.ai business model
00:09:30 what companies get wrong today, influencing the narrative
00:10:30 truth, reality is reality
00:11:00 appreciate engaging on a technical level
00:11:30 internet lies, truth, being on your own
00:12:40 hacker manifesto, hacker culture eric raymond, hacker culture
00:13:50 responding to HN Hector Martin - marcan42, asahi linux
00:16:10 George Hotz security background
00:16:45 communication style, lone genius
00:17:20 response to 10 years sony story
00:18:18 political disagreement
00:19:25 hacker news doubt, marcan worked on asahi linux
00:20:30 lone geniuses, building on other people
00:21:25 Jürgen Schmidhuber credit for deep learning, Yoshua Bengio, Geoffrey Hinton, Yann LeCun
00:22:25 infosec people on twitter, most talented people ended up giving up
00:23:15 never give up never surrender, some people are better than other people
00:25:30 showing off in videos, depth in videos
00:26:10 Elon has no ego, I want to be right but not at expense of being wrong
00:27:10 views, entertainer
00:28:00 hacker manifesto, getting mad at AMD
00:29:00 compounding successes, spacex success story
00:29:45 Tesla FSD attitude, zen with GPUs
00:31:20 AMD change log, not high quality
00:32:00 ghost autonomy, call for ghost to open source, public domain
00:33:45 intellectual property
00:34:50 sonny bono copyright extension act
00:37:45 regression bug
00:41:30 computers don't feel pain
00:42:18 4 compute types, SDMA, COMPUTE_AQL, COMPUTE PM4
00:43:20 back to marcan hacker news response, AMD comunication
00:44:15 comma ai safety model, fear uncertainty doubt
00:45:50 watch what I do not what I say, geohot github
00:46:40 most honest response to a hater
00:47:15 CP old term, personal attacks
00:50:10 tomcr00se soundcloud civil war
00:50:50 Elon test, AMD hardware issue
00:52:25 how hard is to change yourself, other people
00:52:40 HWE = hardware enablement, food
00:53:50 what we did so far on the stream
00:54:45 AMD released a tweet
00:56:10 death spiral sequence, search engine, logging-in
00:57:10 kagi, Apple, no more iPhone, search engine MacOS, arc browser, xz utils
01:02:20 how to launch a kernel with PM4 packets
01:02:55 roct-thunk-interface
01:08:20 want the world to be better, personal gain, mechanized slaves, hvac
01:10:30 PM4 vs AQL, crash info, PM4 might bypass all this garbage
01:12:50 ai alignment
01:15:00 bug identified correctly, but not fixed properly
01:18:50 nvidia driver, umr, RS64
01:26:25 timeline for stable red tiny box, 20 conversions to green and conversions 2 to red
01:27:50 280 pre-orders, 40 cancellations
01:29:35 if we go to this level GPU will be stable, PM4
01:31:00 GPU-specific dispatch registers, quest for low level control stupid and contrary to all business objectives
01:33:00 has not crashed yet
01:40:00 Virtual Memory Identifiers (VMIDs)
01:43:00 exploring amd gpu scheduling details by experimenting with worst practices
01:45:00 downgrade firmware no crash, regression
01:53:15 PQ
01:56:40 amd global data share
02:02:40 how to dump this queue
02:13:00 favorite AMD employee is streaming
02:17:25 PQ, pipe queue, AI hallucination
02:19:59 PQ BASE, mac os x forwarding
02:32:45 gl x forwarding nvidia ubuntu 22.04
02:37:59 packet queue
02:42:55 ubuntu setup vnc, mac os x vnc client brew
02:47:55 control b stupid shortcut, control a
02:49:45 starting with computers today, if you are old it's over
02:54:30 vnc opengl
02:55:20 wayland, era of linux desktop, microsoft rdp
02:58:00 sdl2 x xquartz forwarding mac os x
03:01:20 launching gui for an hour
03:04:10 booting asahi linux macbook
03:07:10 sharing macbook screen
03:12:00 shower
13
-
13
-
13
-
13
-
13
-
13
-
13
-
13
-
13
-
13
-
12
-
12
-
12
-
12
-
12
-
12
-
12
-
12
-
12
-
11
-
11
-
11
-
11
-
11
-
11
-
11
-
11
-
11
-
11
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | Support George by subscribing on twitch.tv/subs/georgehotz | github.com/geohot/tinygrad | github.com/CompVis/stable-diffusion | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:50 sponsor for the channel
00:02:25 cond_stage_model
00:07:30 not a role model
00:11:00 blog post feedback
00:12:55 copyright music, cloudflare, kiwifarms
00:16:10 first they came for poem
00:21:20 centralized, entities with power
00:22:24 aws parler ban, ad friendly content, infrastructure deplatforming
00:24:15 cloudflare statements, changing the content, net neutrality
00:27:13 cloudflare defaced their website
00:28:05 alternatives to cloudflare, ddos-guard
00:29:20 harassment, patent troll, banned user on comma discord contacting comma investors
00:30:46 rational power, psychotic power, doxing, never swatted
00:32:15 axel nix, faulting the system, legal fees
00:35:04 illusion of choice, curtis yarvin, neoreaction
00:38:15 what you learn from hacking, quote from kiwifarms, tolerance
00:41:03 christian, atheism, removing bad things, free speech
00:42:50 internet freedom, good and evil, infrastructure layer
00:44:15 the daily stormer, matt taibbi, destiny, consume product
00:47:50 politics is interested in you
00:48:33 declaration of independence of cyberspace, justice trigger
00:51:00 abortion, ignoring politics, where is the line
00:53:30 real political advice, shutitdown.us, taking rights from other people
01:01:28 george hotz tonight
01:02:08 cloudflare competitor stormspark
01:03:40 raceism
01:05:08 sorry about talking about politics
01:07:05 in search of lost time
01:09:15 lidar = sub only chat
01:09:42 goal for today, people getting off topic
01:13:19 labor day, karpathy email, things to work on, maxpool, gpu execution bug, test to fix
01:16:03 boring and rational politics
01:18:40 why we build technology
01:19:38 idiots, can't have happy without sad
01:22:55 clipmlp, clipattention
01:31:44 clipencoderlayer
01:49:19 compare the outputs and make sure they are right
01:53:39 lawsuits, sue, private investigators
01:58:45 elon is ruthless
02:02:33 axel nix
02:04:27 wire, books, walls, dl slow
02:09:15 tiktok
02:12:35 song tomcr00se
02:14:25 justice, AI safety, taylor swift
02:22:30 stable_diffusion.md
02:25:28 girlfriend
02:31:40 making this work today
02:39:40 it's not the same
02:41:19 music
02:49:25 nutella
02:52:40 what to generate
02:58:35 layernorm
03:16:10 object has no attribute
03:19:00 struggle learning, john carmack lex, fake advice
03:23:50 whole thing match?
03:31:55 dark apple, code is law
03:42:20 monopolies, daylight savings, spacex
03:46:45 how to get rich quick
03:49:10 a horse sized cat eating a bagel
03:56:45 debugging time
04:01:10 information hazard, offended, offending
04:02:45 ubuntu site-packages
04:04:29 music
04:14:20 64?, black square
04:29:20 models together not working
05:03:20 unit tests, segfaults
06:00:40 math by hand, router overheated
06:07:08 bug in mps
06:18:30 no bug in tinygrad
06:26:30 two squares
06:29:40 no plan
06:50:55 noise
07:03:15 good noise, cat
07:16:15 resolve
07:22:43 tinygrad horse
07:34:10 faster inference
07:35:33 tinygrad cat eating a bagel
07:36:44 hard work, sampler, 10 steps
07:39:35 chat generating prompt
07:41:55 penguin with fire extinguisher
07:45:08 the penguin image
07:47:10 aptera
07:47:40 understanding stable diffusion
07:48:15 tinygrad libraries
07:52:00 3 days, kafka, no safety
07:53:20 dolly implementation
07:53:45 serious concerns in ai
07:55:00 can system prove it's aligned with you
07:55:25 effective altruism
07:56:25 next silicon valley
07:57:12 china, culture influences, no rights
08:00:03 immigration, tax, hiring, visa, value, text to video
08:02:40 EU bureaucracy
08:03:10 malaysia, singapore
08:04:05 california, san francisco
08:05:15 no remote, russia
08:06:39 50 iterations, dubai, cyber, freedom
08:15:25 end to end openpilot, agi, cbuffer, FSD, rotterdam
08:17:15 horse eating bagel
11
-
11
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | 00:19:45 01:29:00 beautiful tinygrad graphs | github.com/geohot/tinygrad | github.com/geohot/cuda_ioctl_sniffer | Support George by subscribing on twitch.tv/subs/georgehotz | Enjoy quickly made timestamps and suggest better ones in the comments Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:00:38 quotes
00:02:17 George moving out of apartment
00:02:42 refactor of tinygrad
00:04:15 how does data move, facts over feelings, Ben Shapiro
00:05:18 more info about refactor of tinygrad
00:09:10 we are going to make tinygrad lazy
00:11:25 pull requests tinygrad
00:12:35 fetching the openpilot model
00:14:00 error could not find pythonlibs
00:19:45 beautiful tinygrad graphs
00:21:05 not a safe space for off topic questions
00:21:35 tinygrad lazy branch
00:24:05 1000 lines
00:25:10 permute object has no attribute op
00:27:00 all the ops
00:29:20 the power of tinygrad
00:32:20 back to trying to install onnx
00:42:30 node
00:45:25 trying to setup twitch soundtrack
00:49:38 airpods music
00:51:30 green squirrel in pretty bad shape
00:56:30 George is feeling calm, questions
00:57:30 simple modes
01:14:10 TestOps.test_elu
01:24:40 split
01:29:00 graph of openpilot model
01:33:30 running model on gpu, torch cpu
01:36:30 flatten, MultiCAD support
01:49:15 onnx runtime
01:58:58 tests failing
02:15:40 asking for help on +=
02:20:35 stream for the weekend, land, export to onnx
02:21:45 idea of tinygrad
02:23:05 breaking of CI
02:31:05 idea
02:34:30 stream slogan keep hacking elite
02:36:40 non-sub chat quality, computer questions
02:37:55 food time
02:39:10 sub only society, taxes
02:40:00 always write tests, I love tests
02:40:31 people with skills
02:44:40 onnx resnet
02:52:10 working with food
03:00:15 onnx GlobalAveragePool
03:22:20 that fixed it
03:24:55 Converter is not implemented
03:39:50 bringing in the chicken image
03:54:50 looking up the chicken image
03:57:00 we should write more tests, piano
03:59:40 Unsupported Transpose
04:05:35 replacing onnx with better onnx
04:09:10 it's a lighter
04:12:15 incompatible constructor arguments
04:19:15 more errors
04:26:00 it's totally wrong
04:33:00 ShuffleNet
04:39:45 how much wrong could it be
04:40:58 explaining the problem with maxpool
04:53:30 hen
04:58:00 overview of the stream
04:59:00 recap
05:01:45 questions from subs
05:02:35 lot's of gifted subs
05:03:08 PRINT_LLOPS=1, optimization
05:04:30 tinygrad is a risc instruction set for neural networks
05:08:17 more gifted subs
05:08:45 tinyvoice, tinygrad, tinycuda = tinyfast
05:12:15 why pytorch can't load onnx models
05:13:45 the beauty of tinygrad
05:15:40 more gifted subs
05:16:40 strided max pool implementation
05:20:40 thank you for watching
11
-
10
-
10
-
10
-
10
-
10
-
10
-
10
-
10
-
10
-
Before asking bad questions read http://www.catb.org/~esr/faqs/smart-questions.html | 00:03:34 Loud sneeze by George careful headphone users! | github.com/geohot/cuda_ioctl_sniffer | Support George by subscribing on twitch.tv/subs/georgehotz | Timestamps took a lot of hours to make, suggest better ones in the comments Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:00:34 CUDA driver PTX program compiled to SASS
00:02:24 setting GPU name to sm_86
00:03:34 loud sneeze by George
00:04:57 cuobjdump output sass code
00:06:22 sass, nvidia assembly, bytes
00:06:51 dmacopy, memcpy,trivial, refactoring driver
00:13:37 banned subscriber
00:14:08 nice commit names
00:14:24 subscriber can get banned
00:15:28 goal of today
00:18:13 favorite lines from House
00:18:58 init function
00:22:19 octal is 7, top of food chain, postmodernism
00:23:47 NV_ESC_RM_ALLOC call
00:27:30 good comments by smurfd0, lunixbochs
00:28:16 sniffer not picking up why
01:02:45 getting duped scam GPU
01:12:15 j_blow raiding the stream
01:24:10 standing desk
01:25:12 one line in python to seperate something
01:32:50 that turned out to not matter
01:34:14 allocating TURING_USERMODE_A
01:39:37 hooking with driver
02:08:10 commit allocated
02:25:25 mapping things we care
02:26:29 did not work but it did not crash
02:26:56 Mike Stud - Closer, Hoodie Allen - Little League
02:28:05 something else that happens
02:39:32 driver does not work but the hook is firing
02:45:40 living advice
02:46:18 Elon working from home
02:48:20 there are no lessons
02:49:47 nvidia doorbell register
02:57:56 valid engine id must be specified
03:02:29 this might be it
03:05:16 engineType
03:06:49 we trying to learn
03:15:45 block VA_SPACE
03:29:45 status 0 got an object
03:39:48 this isn't acually right
03:44:30 what else do we need
03:59:19 my error it's right here
04:08:03 we are getting close
04:19:14 raid from provod
04:20:00 what are we doing
04:35:29 everyone can say things they don't believe
04:36:25 dumb questions warning
04:42:18 not initializing for some reason
04:48:37 getting close
04:54:05 comparing dmesg
05:03:12 what was that?
05:09:50 nvidia-uvm, shoutout to @lunixbochs and piano
05:21:12 timeout for bad commenter
05:24:10 could be uvm releated
05:35:31 share the device
05:42:29 we need the uvm
05:43:10 smart-questions.html
05:46:00 part2
05:46:55 hard work and being a genius
05:48:00 what uvm is
05:48:40 it's all copyrighted
05:56:35 Luna comment
05:59:38 gpu uuid and the rm control fd
06:06:25 missing break statements
06:09:38 interceptor not care about any fixed addresses
06:14:35 generic error invalid state
06:18:40 current device is not valid
06:22:38 uvm working, FERMI_CONTEXT_SHARE working
06:34:30 neck massage
06:35:30 generic error
06:39:55 how did I learn
06:45:08 legacy mode
06:54:50 being lazy
06:55:09 raid from sudomaze
06:59:10 channels
07:09:25 driver it's not working
07:27:00 pick a strategy and commit to it
07:28:10 halirutan gifting 10 subs
07:31:15 invalid state
07:49:10 KEPLER_CHANNEL_GROUP
08:00:00 smurfd0 get's promoted to VIP
08:19:25 Q is broken or the doorbell is broken
08:23:25 halirutan gives the non subs the power of talk
08:36:45 doing it again without linking cuda
08:39:45 without the driver the program does not run
08:43:50 intelligent questions
08:49:45 window B answer
08:51:18 memory overhead advantage
08:51:55 hiring Elon's fired people
08:52:20 remote work
08:53:55 the point of life is work
08:58:44 Ian Goodfellow
08:59:15 Vitalik Buterin
09:00:10 taxes
09:01:00 next steps in open cuda project
09:01:45 github copilot
09:03:25 AGI by 2030?
09:04:50 recession
09:06:50 trolling people who ask bad questions
10
-
10
-
10
-
Documentation for the 7900XTX -> github.com/geohot/7900xtx PRs welcome! | 06:56:30 job offer to github.com/gnif | $1000 to fix a reset of GPU for github.com/gnif | 07:02:15
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1050 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:01:25 new KFD backend
00:03:50 never give up
00:05:20 tinybox
00:07:00 looking for comment
00:11:35 docs for 7900XTX
00:13:40 university attendance over time us
00:22:00 vscode markdown preview
00:31:10 signature solar, solaredge optimizers
00:45:45 solar per watt over time
00:50:50 github.com/geohot/7900xtx, tenstorrent software
00:58:20 ubuntu kernel build source don't use apt src
01:01:00 AMD processes, Instinct MI 300, testing in isolation, crashes
01:03:00 no CI for hardware
01:05:50 planning, handling George as PR problem
01:15:00 nvidia taking AMD talent
01:15:50 old instinct cards
01:17:00 test_sdma
01:34:00 firmware file, registers dump
01:37:55 all this docs might be wrong
01:39:30 drivers from AND will not get better
01:46:00 US government people
01:48:10 product engineering, operations, engineering mindset manipulating things not people
01:49:00 everybody who creates is an engineer, administrator not an engineer
01:49:30 boeing, outsourcing, operations keeping systems running
01:51:30 how do you get more gold, more fiat, bitcoin, people
01:54:00 proof of work to proof of stake ethereum
01:56:00 gold works
02:09:00 diversity is our strength
02:19:55 DRM debugging
02:24:19 AMD brutal reviews
02:28:40 4090 tracker
02:39:05 hardware resources
02:48:00 LSDMA, system DMA and other DMA
03:00:00 codenames, UMR
03:05:10 perplexity, searching, search engine
03:09:50 RDNA 3
03:28:50 compute unit
03:35:50 gpu hang, crash, piano
03:39:20 amd community forums
03:41:10 AMD release register reference 79000 XTX
03:42:40 George helping AMD
04:01:35 plum bonito gfx1100
04:03:15 RS64
04:09:55 AMD gpu mailing list
04:13:40 break, shower
04:24:50 should you buy 79000 XTX
04:25:35 AMD PR team, disparaging, George's experience with GPU
04:27:50 the modern consumer
04:30:00 open pilot CI, comma ai failure rate
04:52:50 radeon f32 processor
04:54:20 leak, leak is a public domain
04:58:50 XML dump of bonaire register docs
05:00:00 internet is under attack, remember when google worked
05:01:00 2014
05:06:00 free speech, hate speech, iq vs free speech absolutism
05:17:40 MEC, amdgpu microengine compute
05:46:50 how to power cycle pci device
05:50:15 AMD I want the MEC
06:24:10 error: object file empty git
06:27:25 docs for 79000 XTX overview
06:35:40 SDMA+RLC are F32, RS64
06:38:10 AMD source about register access
06:41:55 ibm rs64 32-bit
06:46:10 rs64 soft core
06:50:50 Alex, RDNA 4
06:53:40 reading about workload managers
06:55:30 the AMDGPU code base
06:56:30 job offer to github.com/gnif
06:57:20 writing software, taping out chips
06:58:40 prediction on AMD high end GPUs
06:59:35 qualcomm reverse engineering
07:01:30 openmax, OMX
07:02:15 can people draw attention to this?
07:03:40 looking glass
07:05:35 internal AMD docs
07:06:40 tinygrad new driver, new style of driver, CommandQueue
07:07:20 $1000 to fix a reset of GPU for github.com/gnif
10
-
10
-
10
-
10
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | WARNING HEADPHONE USERS 00:53:50 | 02:29:00 | 03:06:05 | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:00 LLVM=1, no ShapeTracker yet, ethernet wire
00:02:30 defcon sux
00:03:28 llvmlite, event's to replace defcon, Chaos Computer Club (CCC)
00:04:05 code for implementing llvm backend, LLVM buffer, processing_op
00:05:15 tinygrad low level operations
00:07:14 add and sub could be the same op, could merge some
00:08:15 4 core ops of tinygrad, matrix multiply, matmul, derivative of convolution
00:09:36 why isn't tinygrad fast? memory access patterns
00:10:10 ShapeTracker, segfault
00:11:25 writing test_add_forward
00:12:00 LLVM, memory, single thread system
00:12:25 why it's not working, null pointer
00:14:00 object has no attribute address
00:18:26 segfault, cfunctype m1 ctypes
00:22:18 checking llvmlite version
00:23:13 cfunctiontype calling
00:25:20 as_array requires a shape argument
00:25:38 stupid argtypes
00:27:45 doing an ADD, a MUL and an NOOP
00:28:40 the first argument is the return type
00:29:30 nejintejag get's VIP
00:30:17 MUL should be OK, this NOOP doesn't work
00:31:20 shapetracked objects, shape, stride, offset
00:31:56 test_mul fails, strides 0's
00:32:08 what to implement today, memory might not be contiguous
00:32:53 break, more coffee
00:34:40 ShapeTracker object has not attribute shape_strides
00:35:08 question about Shape tracker
00:35:40 working on idx
00:37:40 rewriting expr, expr junk code
00:38:14 idx_to_llvm, idx_deref
00:41:58 don't use print statements for debugging
00:45:24 llvmlite fadd fmul, fneg
00:46:55 easy implement sub op
00:48:19 RELU not implemented
00:48:30 segfault
00:48:55 arm reciprocal instruction
00:49:18 segfault, why not implemented error?
00:53:50 WARNING HEADPHONE USERS
00:54:40 not implemented error
00:57:10 RELU not implemented
00:58:35 shipping comma ai into car mirrors?
01:03:30 chromecast for cars, just solve self driving
01:05:28 new Blink-182
01:07:35 Axel Nix diss, nathan for you episode, this is going be the greatest thing you ever do with your life
01:09:54 Axel Nix if you are watching this let that sink in
01:11:10 you don't want to punch downpunching down
01:14:08 Axel Nix business, reaching to his clients, don't hate the player hate the game
01:17:50 never ever settle a lawsuit, barnacle on yacht, Peter Thiel on Gawker
01:20:00 Alex Jones lawsuit
01:20:30 send Axel Nix this video, George is real
01:21:47 segfault
01:22:14 does George censor himself on twitch
01:25:00 taylor swift new album, nvidia unlaunching 4080
01:25:40 writing idx_deref
01:26:50 reason for ops_llvm
01:28:29 explaining idx, writing idx_deref
01:32:55 what is it doing, segfault
01:35:30 llvmlite pow exp
01:44:10 exp ops, no reduce ops
01:45:12 ontologicaltofu suggests Asan to check segfault
01:48:05 explaining ZeroView
01:48:55 segfault in python gc
01:50:45 segfault sometimes
01:53:50 what is getting gc to segfault
01:55:13 python faulthandler
02:02:18 python os exit
02:06:32 something is not getting saved
02:07:57 gc.set_debug(gc.DEBUG_COLLECTABLE)
02:10:00 disabling gc
02:17:35 segfault sometimes
02:27:00 wrong info from chat about gc
02:29:00 WARNING HEADPHONE USERS
02:31:05 Bitcoin__ asks about font size
02:33:10 George segfault rage
02:35:35 it was too tiny
02:40:20 loop was too long
02:40:42 memory corruption can show up as any kind of bug, always add some padding
02:44:05 Hailo-8 AI Processor
02:48:30 michael scott yacht quote from money
02:49:34 world without patents, copyright, we keep trademarks
02:50:20 robot lawyers
02:51:28 lesson about patents, copyright term extension act
02:54:55 canceling cox services
02:56:00 google duplex
03:01:00 promise to Alex
03:01:50 why video games, most movies sux
03:03:00 don't care about hacking
03:03:40 hamster to panda
03:04:05 cost of making movies, factorio
03:05:20 outer wilds, elder scrolls sux
03:06:05 WARNING HEADPHONE USERS
03:07:05 web3, comma body cooking meals
03:07:49 different between lot's ifs vs switch
03:08:17 brothel idea
03:09:35 google alpha tensor
03:10:33 Meta AITemplate GPU framework
03:11:00 Apple contacts Comma
03:12:05 raiding lana_lux
10
-
10
-
10
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | From $1499 buy comma.ai/shop/comma-three | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:45 intern at tiny corp
00:02:20 metal kernel, highball drink
00:02:35 dissecting the m1 gpu
00:04:25 india trip, mexico
00:10:20 gpu drivers crazy
00:11:40 metal kernels
00:12:50 opencl global local
00:14:20 IOConnectCallMethod
00:18:45 how gpu works
00:24:30 dev moved to mesa
00:26:55 the good news
00:31:30 apple cheaters
00:37:10 metal reduce kernel slow
00:40:00 bug
00:42:20 detecting not respecting local size
00:46:25 quantum localsize, changes state when observed
00:49:00 python metal, pip metalcompute
00:52:50 global system of capital
00:53:20 it build, almost cuda
00:57:00 writing swift code, xcode
00:57:38 playing the piano
00:58:50 upset or push on
00:59:00 low stars no trust
01:00:15 what is the point
01:03:00 how do you play a role in AI revolution
01:03:30 compute scaling, latency
01:06:20 if AI's are coming how to make sure power is not centralized
01:07:40 emad vs openai, azure
01:10:45 things don't scale always
01:11:45 the idea of comma, thank you tesla, elon
01:13:20 caring about safety, talking about safety
01:16:00 state actors
01:18:55 lil wayne - deep
01:20:05 paper clipping ai
01:22:45 why people join them
01:24:30 forbes sam abuelsmid comma.ai
01:26:35 openai draw, tech and power
01:30:20 what benefited people, no power over people
01:33:00 mark zuckerberg one of good ones, jack dorsey
01:35:25 money, power and love, spam, getting food
01:44:45 let's make some metal
01:57:30 use class methods to instantiate new objective
02:01:35 undo broken in vim plugin
02:06:40 food
02:31:50 metal in pure python
03:04:55 what we should do
03:31:00 metal backend
03:45:00 how reduce work, faster in opencl
03:52:00 bug, serious bug in test
04:02:55 half the time for quarter of the size
04:05:20 two stage reduce
04:07:55 metal timing, fix speed test
04:09:00 this looks fast
04:11:25 add is important
04:14:25 opencl atomic
04:21:10 opencl, cuda, metal concepts
04:23:40 amd close to the metal
04:27:15 timeout for bad chat user
04:30:20 threads by position by thread group
04:38:40 fast but wrong answer
04:42:20 11% speedup
04:46:45 timeout for stupid question
04:49:00 big flops wrong answer
05:02:05 amount of shared memory bandwidth
05:08:50 how to get stats on metal kernel
05:26:20 metal shader compiler verbose
05:28:40 lot's to learn about metal
05:37:30 speedup every day
05:41:15 tinygrad lines
05:43:35 what did we do today
05:47:45 bounty for pull request ($100-$1000)
05:51:35 real price $100
05:54:00 metal matmul
06:02:40 copy in fix, money if you solve really hard problems
06:11:01 how to scale it up
06:26:25 answer right but slow
06:34:30 never because of
06:37:00 hard to implement, slow, secret apple kernel
06:38:40 $500 for strassen implementation
06:39:00 implemented nice metal library
06:39:30 unabomber questions
06:40:25 lesson for everybody
06:41:30 every day a little faster, better
06:42:42 we got the flops
06:43:30 copyright music, addictive
06:44:25 the interesting things about the solution
06:47:20 brew breaking computer
06:47:45 secret api
06:56:45 m1 max benchmarks github
07:04:30 appreciation for flops
07:09:55 happy ending
07:12:50 mesa asahi
07:18:33 ninja build one target
07:30:33 apple shader disassembler asahi
07:30:49 dougall johnson
07:43:50 tinyurl service
07:47:45 commit dump shader
08:11:35 promised and delivered
08:12:00 rap ending
9
-
9
-
9
-
9
-
9
-
9
-
9
-
9
-
Go buy comma three from $1849 https://comma.ai/shop | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Follow George on twitter.com/realGeorgeHotz & instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog | Support George by subscribing twitch.tv/subs/georgehotz
Chapters:
00:00:00 intro
00:01:30 Finatra
00:02:30 installing sbt, life in san francisco
00:04:10 twitter intern, twitter codebase
00:05:45 new look
00:06:05 thank you to contributors on twitter, twitter intern salary
00:07:10 comma github
00:07:40 work full time at twitter, hate about twitter
00:08:10 working on search for twitter
00:08:25 getting rid of pop-ups on twitter search
00:09:45 http-server, thrift rpc
00:10:05 what a microservice is
00:10:16 opensource code
00:10:50 why thrift over protobufs
00:11:40 what is sbt
00:12:50 challenge for the haters (coding duel)
00:13:30 hate for Elon
00:13:40 directions from Elon to George
00:14:05 mr peanutbutter bojack horseman
00:14:15 why intern, career ladder, project manager
00:14:45 scala
00:15:40 Finatra vs Finagle
00:15:50 twitter server
00:16:00 never stop being curious, sf bumming me out
00:16:35 like coming in to the office
00:18:00 worrying about looking dumb
00:18:40 working hard on image instead of skill
00:20:20 getting twitter server running
00:23:20 adminhttp
00:23:35 company putting best foot forward with open source
00:23:58 figuring out twitter server
00:26:00 finagle
00:27:35 intern to make search better
00:27:45 don't want leadership position or to manage anybody
00:28:45 cool thing about scala return
00:29:30 sbt kill run
00:30:05 first time seeing sbt, pants, bazel
00:31:55 good on twitter for opensourcing
00:32:20 public stuff more useful then internal stuff
00:34:10 apache projects
00:34:20 your server as a function
00:36:05 finatra basics, hard to understand
00:37:15 svelte
00:39:00 java people broken
00:40:15 react is wonderful
00:41:10 meteor.js, fakemeteor.js
00:41:53 millionjs
00:42:25 bunjs, zig, frameworks broken
00:43:15 python fast, django
00:43:40 not understanding syntax
00:44:00 com twitter util await
00:46:20 sbt vs mill
00:47:40 asking github copilot to explain
00:48:00 twitter scala tutorial
00:48:20 12-week intern at twitter
00:49:00 two minute conversation with Elon
00:49:30 sf doomed, planning to ship small changes next week
00:50:05 media lying
00:50:45 scrooge
00:52:50 creating a server
00:53:50 pycap terrible
00:54:40 http server definition
00:57:40 wrong version of java
01:03:20 unable to load cache item
01:09:40 famous programmers, computer person challenge
01:10:25 goal for stream thrift server talking between scala and python
01:11:45 stupid media news, genius hacker, not a wizard
01:12:40 warning from friend, san francisco
01:14:00 sbt publish local
01:15:50 linkedin not the best dating app
01:16:50 scrooge github, thrift example
01:28:40 object example is not a member of package com.twitter
01:38:15 how is this not deliberately obtuse
01:41:00 coding the client in python
01:47:20 angry people in the world
01:47:40 how to enable debugging
01:54:30 what they are really angy about
01:55:20 levels.fyi, teamblind, eating the popcorn
01:56:15 day in a life of twitter employee
01:57:25 not speaking with any authority
01:57:45 want to see this experiment succeed
01:59:00 Solo0rTroll raid, what we are trying to do
02:00:30 public opensource stuff, free merch from twitter
02:03:05 no internal knowledge, it wasn't me, new people at twitter
02:05:10 bad media
02:06:40 way too complicated for a client
02:08:25 journalist learn to code, media did this to themselves
02:10:05 finatra logging debug
02:18:35 thrift protocol
02:23:10 catb.org the hacker attitude
02:24:30 Alex on the phone
02:28:20 call_multiply.py working, thrift at facebook
02:35:15 explosion of microservices, more lines of code
02:37:15 what finatra get's you, nodejs
02:38:15 flask, rust, elixir, go for microservices
02:39:30 c++ easy to mess up, Rob Pike on go, haskell
02:42:00 open source code better then internal, java
02:44:10 shill for comma three sale, spreading love, it's business
02:45:40 Elon's heart in the right place, mark zuckerberg, who run's TikTok, no MBAs, Project Managers
02:46:40 founders and investors made bad culture, keep control of your companies, build cool culture
9
-
9
-
9
-
9
-
9
-
9
-
9
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | https://youtu.be/MeE4Y2862FY (Google Coral Edge TPU part 1) | Check out and star github.com/geohot/tinygrad | github.com/geohot/edgetpuxray | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:18 Hacker News thread analyzing George, be able to talk about ideas
00:02:00 looking for ways to plug Coral Edge TPU to computer, appreciate good emails, the verge fake news
00:04:09 comma better without me, different person for 100M+ revenue company
00:04:50 George's limitations not patience just want to attack, Andrew Tate
00:06:12 python 3.11 excitement, comma it's a 20 person company, Harold, Alex, Adeeb running the company
00:07:02 comma has 7 millions in the bank, on track back to profitability, making lot's of comma three
00:07:22 comma trajectory good, not much for George to do
00:07:30 10x software engineers
00:08:12 why I am bored with comma, openpilot really good, have build it and works for me
00:08:40 not driven by other people using it
00:08:54 open source reason
00:09:15 go work at comma.ai/jobs
00:09:25 big potential for growth to openpilot product (sales), functionality some growth, pitching product to normal people
00:10:00 keeping board seat, approvals for large purchases, equity grants, wrappers, oreos disgusting
00:10:50 Google Coral correct output picture banana
00:11:01 Mark Zuckerberg supershares, meta stock, Andrej Karpathy lex podcast
00:11:50 respect to Mark Zuckerberg, fan of Elon buying twitter
00:12:03 George's a moderate person, politically Elon will make twitter fair for everybody
00:12:25 Elon calling out The New York Times, monopoly on information, Elon not a China puppet
00:13:18 no AGI in next 5 years, blog post amazon warehouse worker burnout
00:14:25 no OpenAI, DeepMind internship, influencing people
00:15:42 comma body, tiny corporation
00:16:30 Hacker News thread, Andrej Karpathy at comma, not interested in shipping products to people
00:18:03 Taylor Swift Anti-Hero, 4chan, Kanye West, ye
00:19:04 words taken too serious, reporter, friend reached out, not buying blue check mark
00:20:35 twitter improvements, not optimize for engagement, optimize for quality
00:21:17 unreleased song, not optimizing for recency, Alex still at comma
00:21:57 Trump ban, blue and red box explaining conservatives, liberals interpretation
00:23:25 tomcr00se show
00:24:05 Tiny Corporation overview
00:24:45 George leaving comma same as Karpathy leaving Tesla
00:25:04 wartime CEO, Rick and Morty got bored, George is not Elon
00:27:13 still own comma, still the board, explaining once more, total sum of salary
00:28:47 Jim Keller at tiny corporation, contracts
00:29:10 the idea of the tiny corporation, history of programming languages Assembly, Fortran, COBOL, ML frameworks, Alexnet, TF, PyTorch
00:30:25 C, python, new Machine Learning Framework, DSP, transformers, ResNet
00:32:30 openpilot uses tinygrad, what is new ML framework going to look like
00:33:25 TensorFlow, PyTorch high complexity, how much effort is to add new accelerator
00:34:08 tinygrad simple API, tinygrad operations, lazy.py - processing_op, tensor.py - matmul
00:35:45 nice syntax, stable_diffusion.py
00:36:35 Cherry on FPGA, supporting all accelerators
00:36:57 not making chips, getting to FPGA level, working with other companies that want to make chips
00:37:42 Only 3 people, Philippe Tillet, openai/triton
00:38:23 AI chip company contact George for good software, contracts
00:39:40 CEO with skill, humility, Mark Zuckerberg skill, how to judge a successful CEO
00:40:50 Zuckerberg stupid move, John Carmack
00:41:30 Polyhedral Computation, Triton’s documentation, AMX instructions, FSD Chip
00:43:58 low hanging fruit of new ML chip, software needs to be good first, NVDLA
00:45:10 tinygrad 3090 speed, lazy.py refactor, AST, _realize functions
00:47:08 George's Apple M1 Max computer, maxbenchmark_train_efficientnet.py, openai.com/blog/triton/, relu
00:52:30 comma not over, why do you think that?
00:53:00 explaining tinygrad calls, compute
00:55:44 Jax complexity, Levandowski law company looks like their codebase, Google bloated, Kubernetes from Google
00:57:50 no pypi, how to install tinygrad, goals for tinygrad 1.0
00:58:40 why wanted to talk today, the verge hit piece, metaseq, verge guy call me, another reporter reach out
01:00:53 reach out from Nvidia, AMD should sponsor tinygrad,
01:02:10 tinygrad attractive to companies, Theano
01:05:05 TechCrunch reach out, compile and run on edge tpu, reverse engineering the API
01:07:55 the goal of tinygrad LLVM backend, nnfs.io, lag while compiling
01:15:20 strategy build simple things then reverse engineer, stable diffusion Edge TPU support
01:26:43 google ruy, writing running custom model commit
01:35:50 output 6, is it using Edge TPU?, neural engine apple
01:42:10 proof that it runs on the Coral
01:45:00 RetireInProgress raid, exploring, writing driver for Coral
01:50:30 single operation, edgetpu-custom-op, input tensor 1 and output tensor 0
01:57:10 smurfd0 helping George, git clone TensorFlow, break
02:04:45 break 25 min
02:28:35 building TfLite bazel
02:35:35 wishing to be better about comma thing, sending journalists to Harald Schäfer CTO comma
02:36:15 trying to make it work
02:42:50 Should I buy a Porsche Boxster?, carfax, no electric car for George, Lambo
02:48:40 sarcastic George about why government love EV, government control, low range EV
02:49:30 modifying the tpu driver to see if this can be parsed
02:56:40 culture at remote organization
03:03:35 worldbuilding, private advice sessions, monetize advice
03:11:15 about leaving comma, still going to be involved, shipping 100k product, supply chain, real engineering
03:18:00 python program to mimic the C program is doing, edge tpu complier, driver
03:23:20 structuring society in new way that is not capitalism, economics, being public, painting in a nice light
03:27:10 edgetpu_compiler, cool thing to support Coral
03:32:45 docker, NSA backdoor, tinygrad for the startup, tinygrad for Nvidia
03:36:00 docker platform flag, worked, how amazing docker is
03:42:00 reading .tflite files, docker's licensing, using Ghidra
03:52:40 not doubling down on META stock
04:01:25 objective way to measure intelligence, brain machine interfaces
04:04:50 edge tpu decomplier
04:07:55 Google would you like to sponsor George to continue work on Coral Edge TPU
04:10:10 AMD you should sponsor George to support tinygrad
04:11:20 if you only can use Coral simply with tinygrad, apple detection demo on Edge TPU
04:13:45 Coral only supports ints, HBO season 2 white lotus
04:14:45 message to comma.ai archive about doing only things you enjoy
04:15:20 tiny corp is real, for engineer 1 you need insane experience, comma jobs, Google, Nvidia sponsor
04:18:10 chicken, hen on Google Coral, docker script cleanup
04:19:44 local, binary for compiler, how to turn debugging on, Christian channel
04:32:32 edgetpu compiler output logs, profiling
04:44:00 all hacking is strings, dumping firmware, DEBUG=1,
04:49:25 one flag away from magical solution, strings unicode
04:53:40 how did I missed this?, probably wrong approach
04:59:35 email to George debug output to get VIP on twitch channel
9
-
9
-
Before asking bad questions read http://www.catb.org/~esr/faqs/smart-questions.html | 02:10:10 George raiding https://www.twitch.tv/j_blow | github.com/geohot/cuda_ioctl_sniffer | Support George by subscribing on twitch.tv/subs/georgehotz | Timestamps took a lot of hours to make, suggest better ones in the comments Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro George is apologizing for doing things by hand
00:04:19 doing the right thing, clang
00:08:28 what we want to implement
00:09:22 subscriber getting banned
00:10:20 refactoring time, name for the project, cuda name copyright
00:27:25 what's the minimum you need to initialize cuda context
00:29:35 need to get gpu uuid or broken everywhere except my computer
00:37:20 GPU_UUID, nvidia-smi -L
00:40:25 spaces in vs code give me dots (mac only issue)
00:42:35 more refactoring
00:49:28 good comment two times included AMPERE_CHANNEL_GPFIFD_A
00:52:04 what do we want to do next, fast cuda initialization
00:53:45 doing assembly by hand
00:59:30 killing x, open cuda memory usage 2 MB (almost fit on cache)
01:01:30 nvidia driver broken
01:04:50 nvidia cuda driver memory usage (245 MB usage)
01:06:40 this should be a parameter somewhere
01:08:17 hook up with rdma
01:09:18 for 100 gpu, you are saving 20 gb vram
01:09:35 climate change, barrels of oil, carbon offsets
01:11:50 benchmarks, value, malloc
01:14:59 streamer for the body, complicated c++ program, no copies complicated, memcopy
01:16:28 we could try to reimplement the cuda driver, torch to work
01:21:33 smart parser, maintaining the code, investment in infrastructure
01:23:40 vs code advantage vs vim
01:24:35 working on infrastructure at comma, distributed file system
01:25:35 gigashuffle, reason why companies do not opensource
01:26:10 AMD drop in cuda replacement, ptx to amd, spir-v
01:27:50 PTX legal reasons
01:29:05 nice things about nvidia
01:29:46 there is no point in trying to defend a hill anymore
01:31:43 risc-v, lisa su, x86, arm64
01:33:30 qualcomm, selling chips, garbage cloud
01:37:25 TPUs are just fixed function hardware
01:38:50 tenstorrent, making chip is hard, Nervana Systems
01:41:01 what makes cuDNN slow, tools easy to see why it's slow
01:42:55 running two companies, Elon, Vitalik
01:43:33 nvidia profiler
01:44:10 slow reaction times
01:45:10 exactly what parts of the future do you want to own
01:46:40 take mentality, openpilot unstoppable, safety
01:49:35 singing
01:51:25 pytorch memory usage
01:52:25 tensorflow hidden state, JAX, MXNet,
01:53:48 not bad library
01:54:20 bloated software, rewrite the code you have, gigashuffle explained
01:55:38 accelerator people, own the stack, high end
01:57:10 Julia vs python
02:03:00 looking up Jonathan Blow stream
02:04:24 fast python, looking up Jai, different python interpreter
02:10:10 raiding j_blow
02:12:25 why should George use Jai
02:16:15 George ask j_blow why is python slow
9
-
9
-
9
-
9
-
9
-
9
-
9
-
9
-
9
-
Update! https://twitter.com/LisaSu/status/1669848494637735936 Thanks for connecting @realGeorgeHotz. Appreciate the work you and tiny corp are doing. We are committed to working with the community and improving our support. More to come on ROCm on @radeon soon. Lots of work ahead but excited about what we can do together.
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | RDNA3 Instruction Set Architecture Reference Guide https://www.amd.com/system/files/TechDocs/rdna3-shader-instruction-set-architecture-feb-2023_0.pdf | 01:57:15 enjoy your amd GPUs, AMD I want to make your GPUs amazing you should reach out to tinygrad | 01:57:43 will not send them back, I bought this GPUs and I am going to make them work | AMD is hiring Open-Source Linux/Mesa Developer https://www.phoronix.com/news/AMD-Another-Mesa-Dev-2023 Should George apply? | From $1499 buy comma.ai/shop/comma-three | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:00:15 RDNA3 instruction set architecture stream
00:00:30 absolute joy to work with AMD open source drivers
00:00:40 less then half time they crash, time machine that works half the time = amazing
00:01:45 RDNA3 assembly syntax highlighters
00:02:28 pathlib, quiet pc
00:04:45 running shaders
00:05:55 assembly highlighting
00:07:15 v_dual_mov_b32
00:08:30 s_sendmsg
00:10:10 amd driver is lovely, works 95% of the time
00:12:00 power_dpm_force_performance_level
00:12:50 rocm sniffer segmentation fault
00:14:05 big font
00:14:20 what is sniffer doing
00:16:05 gpu doorbells
00:18:30 we are going to write our own simple driver
00:21:20 explaining what we are doing assembly language for RDNA3 runtime io controls
00:22:20 sigemptyset
00:25:40 sigsegv handler inside python
00:26:50 operating between a level kernel and userspace, doorbell
00:28:50 sigaction threads
00:32:25 killed by SIGSEGV
00:32:50 mojo source, no talking about mojo until source released
00:33:25 sa.sa_flags = SA_SIGINFO
00:36:42 rt_sigaction sa_restorer
00:37:15 sigemptyset
00:38:49 sigaction(SIGSEGV, &sa, NULL)
00:39:38 sigaction isn't being caught
00:41:10 rt_sigprocmask
00:41:37 example sigprocmask
00:43:00 Alex
00:43:30 sigprocmask example
00:45:10 Alex bringing ramen
00:46:25 removing LOG_LEVEL=5
00:47:00 Alex, chair
00:49:45 queue_type 1, dma queue
00:50:00 spindrift, tortellini, grateful for ramen
00:50:52 positive upbeat stream, negative stream before, advent of scala part 1
00:51:20 thank you ptim for gifting 10 subs
00:51:40 KFD_IOC_QUEUE_TYPE_SDMA
00:52:15 JacobsM1 gifted 5 subs
00:53:10 HSA Async Copy staged
00:54:10 DMACopy
00:54:25 Alex and George working on GPUs
00:55:00 Alex feeling GPU energy around George all the time
00:56:00 cobracoba gifted 5 subs, mojo questions
00:56:35 it's not as good if it's secret in startups
00:57:05 Chris Lattner open source, modular $30 million
00:58:30 hsa_kernel_dispatch_packet_t
00:59:10 Alex patch & George hoodie matching
00:59:30 stream for entertainment not for education
01:00:00 the packet definitions
01:00:50 stormfrosty: ~4 years of being underpaid at AMD paid off
01:01:10 it's not ok not being aware of how little you know, taking pride in not knowing things
01:01:50 sunrise san diego, Alex get clothes ready for sunrise
01:02:55 Lana_Lux raiding stream
01:03:30 today we are looking at all the parts of the gpu driver in order to make a good gpu driver
01:03:41 doorbell, queue_type transfers, amd open source
01:05:15 PM4 packets, PGM_RSRC1
01:06:39 AMD's PAL github
01:09:35 hsa_status_t DmaCopy
01:11:00 DmaCopy(void dst, SubmitLiniarCopyCommand
01:11:25 HSA_PACKET_TYPE_BARRIER_AND
01:13:00 HSA_ENABLE_SDMA=0
01:13:55 CPDMA, how low level is this? it's at a user space level
01:14:30 enable_sdma
01:16:20 SDMA_PKT, SDMA_OP_TIMESTAMP, SDMA_OP_GCR
01:20:10 fatal error file not found
01:25:40 George got distracted, sdma_registers.h
01:26:30 offset
01:27:55 tinygrad code gen engine is good
01:28:05 unhandled packet type 17, unhandled packet type 1
01:30:00 end goal is to make it good
01:30:10 SDMA cool, different queue, unhandled packet type 10
01:30:55 User Mode Register Debugger
01:31:30 George like writing himself so he can learn about how it works, did the same with CUDA
01:32:22 SMDA_PKT_COPY_LINEAR
01:37:20 we understand how the buffer get's transferred
01:38:05 queue_type 1
01:40:45 this running the program, transferring buffer
01:41:00 copy kernel guess, dumb the args
01:41:25 hsa_kernel_dispatch_packet_t
01:43:00 tired happy for a nice stream, positive progress
01:43:15 if you had a time machine that worked 98% of the time
01:43:40 thank you for gifted subs, we got custom assembly running on gpu
01:44:10 dumping kernel arg address
01:46:20 ComputeABI initial-kernel-register-state
01:46:40 private segment buffer
01:47:45 what is the address
01:49:00 what address is actually in the gpu
01:49:35 you can step through the kernel assembly with gdb
01:50:15 github.com/RadeonOpenCompute/ROCm_Documentation/blob/master/ROCm_Tools/ROCgdb.rst
01:51:00 rocm-gdb
01:52:00 break point
01:52:40 could not attach a fatal error has occurred
01:53:40 global_store_b32
01:56:40 ds_load_b64, s_load_b64
01:57:15 enjoy your amd GPUs, AMD I want to make your GPUs amazing you should reach out to tinygrad
01:57:43 will not send them back, I bought this GPUs and I am going to make them work
9
-
9
-
8
-
8
-
8
-
8
-
8
-
8
-
8
-
8
-
8
-
8
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | From $1499 buy comma.ai/shop/comma-three | Go read and share the docs with friends -> https://github.com/geohot/tinygrad/tree/master/docs | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:00:45 tinygrad docs
00:01:10 triton rst docs
00:02:10 debt in tinygrad
00:03:05 great stream to learn how everything works
00:05:25 george teaching you programming
00:06:25 2+3 as Tensor
00:10:25 Types from Tensor
00:11:00 accountants problems, integrity
00:11:40 boilerplate imports
00:12:18 vscode python annotations %%
00:16:10 understanding the example 2+3 as a Tensor
00:17:50 derivatives in tinygrad
00:21:20 Tensor, relu, mlops example
00:24:45 LazyBuffer
00:25:20 pip or install it via git clone
00:26:10 LazyBuffer
00:29:55 lazydata
00:33:35 lazyop
00:41:10 hlop, mlop
00:41:25 thinking about this for 2.5 years
00:41:40 tinygrad hope
00:41:55 worth reading to understand hlop, mlop
00:42:25 tinygrad not slow, pytorch and tensorflow hundred of ops
00:43:15 getting pytorch support hard, why are things slow
00:44:00 worth reading
00:46:45 why can't FROMCPU be folded into ASTs?
00:48:45 example is tinygrad fast
00:49:30 DeviceBuffer
00:52:00 subclass c++
00:53:30 InterpreteredBuffer
00:54:35 don't worry about lazybuffer
00:55:10 DeviceBuffer
00:57:35 inops support
00:58:20 lazy.py
01:01:10 RawBuffer, 2+3 in raw clang
01:05:45 2+3 autogenerated clang code
01:10:55 ShapeTracker
01:23:55 no Tensor just a shape
01:24:25 variable in shape/symolic.py
01:33:00 theory of lines of code
01:33:50 new issues on github
01:34:08 JIT=1
01:37:35 what jit does
01:38:50 unittests
01:47:00 bug in tests
01:48:30 don't know anything about computers
01:49:10 tinygrad lines and garbage lines
01:50:00 2k lines, tiyngrad backends
01:51:55 5 lines for speed, shape, reshape
01:53:30 permute, reshape, expr_node
01:54:40 symbolic.py
01:55:20 fixing broken the tests
01:59:22 unittests parametrized test
02:06:00 looking at the docs
02:08:25 optimizers SGD, RMSprop, Adam
02:08:35 test_optim.py, challenge for pull request 2 line
02:09:00 lion optimizer, experimental drugs waiting
02:10:35 admiring mlops.py, looking how long is adam optimizer
02:13:50 llvm=1 on example, slow because copy overhead
02:17:40 tinygrad metal backend faster then pytorch backend
02:18:50 new goals for tinygrad, train imageNet
02:20:40 step to linearize the ast and codegen the ast
02:21:10 what is different about jax
02:29:20 numpy and finding a bug in tinygrad
02:43:15 adding test_div_numerator_negative
02:47:10 thinking about what to do
02:54:50 tinygrad symbolic implementation
03:00:35 removing tests
03:01:55 linking docs to tinygrad discord
03:02:55 commit bufs not none
03:04:45 some abstractions are good
03:05:15 codegen needs new version
03:06:10 goals for 0.6 release
03:07:00 code is a line, ast is a tree
03:07:30 reviewing code
03:08:55 tinygrad is not for openpilot, is competitor to pytorch
03:11:10 symolic is now a 6/10 due to the infinite loop
03:14:10 read and share the docs with friends
03:14:20 is tinygrad stupid? a lot of time invested into tinygrad
03:14:50 who is tinygrad for?
03:15:30 atan2_cpu
03:16:15 tinygrad stable diffusion
03:17:10 why python
03:18:15 transformers
03:18:33 pyright
03:22:00 tinygrad is just george and community
03:22:33 batman joke in javascript
03:24:10 ENABLE_METHOD_CACHE=0
03:24:40 pytorch is broken
03:26:30 workday start at comma
03:27:35 imagenet to c compile test
03:29:10 tinygrad will win, fast
03:30:00 agi does not exist
03:30:15 gpt-4 this week
03:30:50 ban for bad user
03:31:05 how readable is tinygrad code
03:32:20 plans for tinygrad inference and learning as the same thing
03:32:40 long term re-writing tinygrad in tinygrad
03:33:25 andrej karpathy on training vs inference
8
-
8
-
8
-
8
-
8
-
8
-
8
-
8
-
Update! https://twitter.com/LisaSu/status/1669848494637735936 Thanks for connecting @realGeorgeHotz. Appreciate the work you and tiny corp are doing. We are committed to working with the community and improving our support. More to come on ROCm on @radeon soon. Lots of work ahead but excited about what we can do together.
https://github.com/kposborne2 is the hero of the stream! https://github.com/geohot/tinygrad/pull/823 | The tinybox $100 preorder total cost $15,000-> buy.stripe.com/5kAaGL6lk9uX9nW144 | geohot.github.io/blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html | tiny corp hiring -> tinygrad.org
Issues submitted by George to AMD (AMD response patch https://youtu.be/Mr0rWJhv9jU?t=175):
https://github.com/RadeonOpenCompute/ROCm/issues/2198 ([Driver] BUG: unable to handle page fault for address: ffffa7c13eaffff8)
https://github.com/RadeonOpenCompute/ROCm/issues/2205 ([Driver] amdgpu 0000:c3:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:12 pasid:32773, for process rocm-bandwidth- pid 8285 thread rocm-bandwidth- pid 8285))
https://github.com/RadeonOpenCompute/ROCm/issues/2204 ([Driver] WARNING: CPU: 2 PID: 479 at drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c:471 amdgpu_bo_move+0x261/0x2e0 [amdgpu])
https://github.com/RadeonOpenCompute/ROCm/issues/2197 ([Driver] WARNING: CPU: 96 PID: 903 at drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c:600 amdgpu_irq_put+0x9f/0xb0 [amdgpu])
https://github.com/RadeonOpenCompute/ROCm/issues/2196 ([Driver] ERROR MES failed to response msg=2)
tinygrad/tinycorp archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQsh0fFZ2QKOBY25lz04A3hi
Chapters:
00:00:00 intro
00:00:45 tiny corp raised money, tinybox
00:01:40 tinygrad goals, developer experience
00:02:15 tiny corp programming challenge
00:03:00 don't waste George's time
00:04:15 tinybox almost 100 preorders
00:04:45 $100 preorder, $15,000 total price
00:05:25 tutorial.ipynb
00:08:40 set_debug does not work, python import global variable makes copy
00:16:30 lazy by default, should not lazy forever, enqueue
00:17:00 fixing debug problem, properly on global variable python
00:17:45 class debug, python magic methods
00:26:05 inherit from int python
00:28:50 1000 lines guideline
00:29:15 class debug
00:31:30 new debug behavior, clean clode
00:34:00 logging modules
00:34:15 contiguous
00:35:15 matrices, Tensor.eye
00:36:45 llama.py, debug on llama.py
00:38:15 quantization support, contribute to tinygrad
00:38:42 Conv2d, stable diffusion
00:39:25 $10000 bounty for stable diffusion on M1 faster then any existing implementation
00:40:35 a cat in a hat
00:40:47 bounty taxes
00:41:00 stable_diffusion.py and llama.py code
00:41:20 unit tests in tinygrad, ONNX tests
00:41:54 TORCH=1, external_text_onnx_backend.py, pytest
00:42:40 great way to good started
00:43:20 complexity management, self driving people dumb
00:43:46 revenue company, waymo and cruise, growth, investment, referral
00:47:25 smart pytorch people, not managing complexity
00:48:45 implement every op, conv_transpose2d
00:49:50 conv, reshape, sum
00:50:10 competing against smart people, developer experience, hard to develop on those libraries
00:50:50 jit.py, $500 bounty LLaMA python code without jit under 100ms
00:51:50 triton tutorials, conv2d example
00:56:55 mojo, einops-style API, $200 bounty Zero dim Tensor support
00:57:25 zero dimension tensors
00:57:50 haters Hacker News, uncountable infinities, cardinality, recursively enumerable
01:00:55 constructivist
01:01:25 matrix multiplication, a@b, cube
01:03:55 different matrix multiplication
01:05:20 kernel logs are so beautiful, colors
01:05:55 efficientnet.py, NUM=0, NUM=4, sd cat, cowboy hat
01:07:05 DEBUG=2, OPTLOCAL=1, kernel that are running
01:07:40 .realize ending the laziness, NUM=4 kernel
01:08:15 r = reduce kernels, e = element wise kernels
01:08:50 George friendly to noobs
01:10:00 autotune stuff, add mlperf bert model pr
01:11:30 adding tinygrad to huggingface
01:11:50 getting started contributing to tinygrad, fix ONNX tests
01:12:15 AMD driver stack, emailing AMD, intel amazing chip docs
01:12:30 intel a770 $349, XMX AI capabilities
01:14:55 fp16 flops on a770 vs 7900 xtx
01:15:24 I like AMD, driver is so bad
01:15:38 prototype tinybox, two gpus and driver crashes, intel
01:16:55 QLoRA, Falcon 40B model, Nvidia
01:18:25 500+ tokens per seconds, max 137 tokens per seconds
01:19:05 doing AMD work, free work, writing open source code
01:19:55 AMD more documentation open, memory architecture, scheduler, copy engine
01:20:30 why isn't AMD doing that already?
01:21:25 composable kernel, leapfrog nvidia, consumer hardware without gpu, m1
01:22:50 tinybox RAM bandwidth, AMD epyc, pcie lanes
01:24:00 RWKV, infinity fabric
01:25:15 how backends are implemented in tinygrad
01:34:00 transferring kernels to device
01:39:40 DCGAN
01:42:30 transfering img_align_celeba.zip
01:45:00 pre-commit hooks for tinygrad, zip and folder, jpeg
01:48:00 writing dcgan.py class Generator
01:50:50 class Discriminator
01:53:40 celeba_iterate
02:10:35 training tested very little
02:16:00 it uses a stride
02:17:20 training something, tensor object has no attribute _ctx
02:24:00 in theory this should work
02:30:35 need to make strides work
02:31:00 simple code, tinygrad, does not work
02:33:30 time on GPU, OPTLOCAL
02:40:15 do people like gan's?
02:42:42 if somebody fixes strides by tomorrow George will stream and finish dcgan
02:43:30 ugly permute, test_nn.py test_ops.py
02:44:20 big font today, depthwise convs
02:45:00 matrix decompositions, where we draw a line in tinygrad
02:46:00 no sorting, no scatter gather, if it's easy to implement
02:46:35 tinygrad ops, work together collaborate
02:47:20 tinygrad discord
02:47:30 how to deploy $5 million
02:48:25 tinybox prototype
02:49:15 limiting the hiring pool, changing the blog post
02:50:40 bad submission for programming challenge
02:51:44 tax, Joe Biden, remote work, budget of tiny crop
02:52:20 AMD on MLPerf, how Elon feels
02:53:10 sponsor visa, tiny corp is only George, rolls-royce, lost money on cheapETH
02:54:30 Lana_Lux raid, finishing DCGAN tomorrow if someone fixes strides
02:55:45 trying to raid Lana_Lux, enjoyed Lana stream, unity
02:57:05 brief overview of tiny corp, cow farms, chicken farms
03:04:30 avoiding monopolies on compute, ai safety, can't stop technology
03:10:05 Alex, tiny corp a company writing some open source machine learning software, making AMD competitive
8
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | George is streaming right now on https://www.twitch.tv/georgehotz part three of stable diffusion | github.com/geohot/tinygrad | github.com/CompVis/stable-diffusion | Support George by subscribing on twitch.tv/subs/georgehotz | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:02:06 quiet computer
00:02:25 how far did we get yesterday
00:04:08 cleanup
00:06:15 class Linear
00:07:50 class UnetModel, SiLU
00:10:25 fighting Mark Zuckerberg, Elon Musk, Vladimir Putin
00:13:10 hand to hand combat, Elon is cool
00:14:35 input_blocks
00:19:14 fusion power, stagnation
00:22:40 chemical engine, rocket fules
00:36:25 preview of blog post
00:51:10 class UNetModel, UnetResnetBlock, ResBlock
00:58:40 crypto is good for one thing
01:01:50 don't actually clip it, no clips
01:02:00 cloudflare bad move
01:02:50 blockchain will not replace sql database
01:03:45 crypto DAOs
01:05:20 meta stock
01:06:19 I take back the clip, don't want clips
01:07:23 blockchain things that will succceed
01:08:18 drugs are not cool anymore
01:08:45 unregistered securities
01:13:47 decentralization
01:14:40 cheapETH
01:17:20 what can you buy with money
01:17:58 crypto disrupting fiat currency
01:21:00 lack of respect to Russia
01:25:27 I don't know that much about this stuff
01:26:25 stop building technology
01:27:35 communism, vietnam, capitalism good or bad
01:28:24 christianity
01:31:32 population collapse
01:32:55 ubi
01:36:25 go protest outside cloudflare
01:37:45 two kinds of power
01:42:44 cloth masks
01:43:35 race relations
01:45:25 metaverse, love in the world
01:46:44 more Elon's in the world
01:47:00 scams in self driving space
01:47:50 politics is interested in you
01:51:20 you got to earn the right to an opinion
01:52:00 nn linear pytorch
02:12:50 are we learning?
02:14:20 understand the structure
02:17:20 why is that that
02:18:55 saying hello to Alex
02:19:18 rules for chat, rules for any social environment
02:22:18 middle block
02:26:35 kanye west
02:28:50 when people say something stupid
02:31:55 fan spinning up, slow
02:34:55 VIP for agifirealarm
02:37:50 what tensor.uniform doing
02:41:25 subs and non subs
02:41:45 done diffusion model, break
02:45:10 transformers are sample efficient world models
02:49:00 visual transformers paper
02:52:15 hutter price, understanding math in this book I could solve AI
02:58:05 job at huawei, american propaganda about china
03:02:55 optimism crypto
03:06:15 averaging weights leads to wider optima
03:07:30 can't reshape
03:21:00 object is not callable
03:25:18 just because you subscribed we are not going off-topic
03:26:45 AssertionError, has no attribute reshape, not callable
03:34:45 UNetModel
03:39:19 what is the dimension there
03:41:55 good idea
03:42:25 chunk
03:47:00 Alex call
03:50:13 torch chunk
03:56:10 np.random.uniform
04:01:50 was wrong
04:05:55 timeout for bad commenter
04:06:20 finding print
04:07:55 can't reshape
04:10:25 new problem, output blocks
04:12:45 it's a unet
04:16:30 it just ran
04:20:15 why it's not lazy
04:23:10 torch tinygrad fusion implementation
04:23:39 Alex bring food
04:24:55 load apple latent space
04:27:35 we are out of memory, tensor.no_grad=true
04:30:25 opencl=1
04:38:40 the problem was nograd
04:42:45 black apple, bajillion bugs
04:51:50 non subs talk
04:52:20 two things to do
04:57:35 more streams
04:59:18 deep understanding by doing
05:00:00 Pat Stay death
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
7
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | twitter.com/__tinygrad__/status/1729279842376613941/photo/1 | youtube.com/post/Ugkx49TlILe21dXuhJu6a74j5WCx3nGldqCC
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
github.com/tinygrad/tinygrad <- simple powerful deep learning framework
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:01:35 responding to youtube comments
00:02:05 C. S. Lewis quote
00:03:05 physiology and sociology, wokeism, trans people
00:04:35 harvard, ivy league educational institutions
00:07:15 tesla cybertruck
00:08:40 communism, anarchy
00:10:15 based elon
00:10:40 Amouranth, Sky Bri
00:12:20 love, hate
00:13:35 refactor stream
00:14:05 beautiful.py
00:14:50 memory objects in opencl, buffers and images
00:16:00 dtype, cast_image, device hack
00:17:35 LRUAllocator
00:22:18 Clang operations, MallocAllocator
00:23:20 missing 1 required positional argument
00:27:40 this is the actual George's work
00:28:45 LazyBuffer problem
00:32:00 numpy
00:32:45 Alex
00:33:20 auto generated program, float
00:35:00 tinycorp second intern
00:36:40 TORCH=1 test_ops.py
00:38:15 adding cast
00:38:30 unreadable code
00:41:00 UnaryOps_CAST
00:42:10 DiskAllocator dtype
00:46:20 git push
00:47:00 letting everyone chat
00:47:30 be nice to people
00:47:50 what is AGI
00:48:20 why care about views
00:49:00 beautiful.py, tinygrad is a tensor library
00:51:00 ask good questions, anyone understand this?
00:54:45 memoryview cast
01:00:00 copyin function
01:03:50 buffer, JITRunner
01:05:00 The Device, LazyOp
01:10:40 renderer
01:12:25 current line count of tinygrad, hiball energy
01:18:00 questions? does this make sense?
01:20:40 FROM_BUFFER=1 examples coder.py
01:22:10 10 runtimes in tinygrad
01:26:50 computation, memory
01:31:40 where you call the kernels, ClangProgram, to_program, CompiledASTRunner
01:33:20 abstractions.py outdated use beautiful.py instead
01:33:50 MULACC trigger, matmul
01:37:35 git push improve docs
01:38:30 t8 tinybox spec
01:39:00 jax_pmatmul.py
01:39:35 WD_BLACK 1TB SN850X
01:40:30 bidirectional copy peak bandwidth GB/s
01:42:00 infiniband connection between two tinyboxes
01:44:10 up to 18 monitors into your tinybox
01:44:40 shipping to EU
01:45:20 preorder tinybox today
01:46:20 stupid question from non-sub
01:46:35 mlperf, how fast you can train the models
01:47:50 15k GPU hours
01:49:20 the cloud, low priority CPU
01:50:10 tinybox for common user or business, secret price
01:51:10 tinybox wattage
01:55:20 Alex brining tea
01:57:10 hashcat example
02:06:25 4090 vs tinybox hashcat
02:09:40 Alex bringing lunch
02:11:45 hip for hashcat bug
02:12:20 best coin to mine with gpus
02:13:35 amd 8000
02:14:30 advent of code day one speedrun
02:19:20 AoC part 2, asking Quentin for help
02:26:00 write a regex to replace
02:30:20 good programming style
02:31:00 struggle in python console
02:37:15 reducing the lines of code
02:50:55 compiling
02:57:50 memory usage bug
03:06:00 external_test_alloc_all.py
03:11:40 analyze the issue in the comments
03:13:55 why tinygrad has to be so tiny
03:14:30 changing culture, gpt-fast
03:17:05 27 tinyboxes = person
03:17:25 teenyboxes = gaming computer
03:18:20 pro esports player
03:19:10 1Torch, unspecified launch failure
03:24:30 ???, OpenLLaMA, Nvidia
03:27:50 if George was CEO of OpenAI
03:30:25 opensource weights
03:30:50 company that lasts 1000 years
03:31:30 top 10 biggest companies
03:32:15 Apple M3 fastest single thread cpu
03:33:30 Apple GPU
03:35:50 high end M3 Max, don't buy low end M3
03:37:25 5 M3 Max or 1 tinybox
03:38:20 based Apple getting rid of touchbar
03:38:45 spec of tinybox
03:39:10 wokism, truth crime interrogation footage
6
-
6
-
Watch youtube.com/watch?v=iFUmWho7fBE - The AI Alignment Debate: Can We Develop Truly Beneficial AI?
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | tinygrad.org | github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/realGeorgeHotz & twitter.com/__tinygrad__ to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:00:50 comma 3x, watch COMMA_CON
00:01:30 don't use Qualcomm
00:02:20 friends with AMD, 60 GPUs
00:03:05 AMD ROCm github activity
00:03:54 Qualcomm will not improve, AMD will
00:04:40 Rockchip RK3588 chips for comma, sending $500k to Qualcomm
00:06:10 software vs sales people, lies
00:07:22 do not work with Qualcomm
00:08:10 AMD SoC, Ambarella, MediaTek
00:08:47 Qualcomm will sell you chips, 3 months later, thundercomm
00:11:10 get comma 3x while you can
00:11:30 company alive, Elon power level
00:12:00 X rebrand, still believe in twitter
00:12:39 Nvidia selling chips, Qualcomm, AMD stock
00:14:05 Meta stock, Qualcomm, Llama 2
00:15:35 George debate with Connor Leahy
00:16:30 don't use Qualcomm
00:17:15 bad people, trusting people
00:17:50 MediaTek, Rockchip, Orange Pi 5 Plus
00:18:25 Problem with HiSilicon, Exynos government
00:18:45 RK3588, Qualcomm chips off phones
00:20:10 comma 3x, Qualcomm hypervisor spyware
00:21:00 Connor Leahy other debates
00:21:20 Qualcomm licence to the chip
00:21:45 shipping 50k devices don't use Qualcomm
00:22:20 switch to Rockchip RK3588 or Ambarella
00:23:10 is Rockchip coming new better chip this year?
00:23:40 new 4-5nm Rockchip with 2-4x perf would be a big success
00:24:25 chips are very complex to build
00:24:50 looking to buy 100k chips, test order
00:25:47 who is buying Qualcomm, google ARM, Qualcomm perf
00:27:10 can't even get in touch with Samsung
00:27:30 why other companies still work with Qualcomm, integrity
00:28:10 comma lunch, Qualcomm stress, aggravation
00:28:40 software challenges with changing to different SoC
00:29:30 George cares, dealing with people I hate
00:30:00 how to stop depressed people from doing bad things
00:30:25 Qualcomm just lie, one more thing, legal document, bait and switch
00:31:15 patent troll saga, leave me alone
00:32:15 Qualcomm worst company I ever dealt with, Qualcomm via gray market
00:34:10 AMD get's better, Qualcomm will not get better
00:35:00 Intel is good, Intel GPU not powerful enough, Intel Software team
00:36:10 big activist investor intel, Jim Keller legacy
00:36:30 making SoC is super hard
00:36:40 Qualcomm legal document not enforceable
00:37:45 the process to buy Qualcomm chips
00:38:40 don't work with idiots, business world
00:38:55 bad incentives for sales people, Elon could turn Qualcomm around so fast
00:39:30 10.000 chips test order, bearish on tinygrad
00:39:50 country where people want to make world a better place
00:40:18 acceleration and deceleration
00:41:00 e/acc, jerry adams curve
00:43:50 world where energy usage is huge, global warming
00:44:40 getting cancelled, vaccines, bill gates, trans, augmented olympics
00:45:45 eugenics, hr department, boss at tiny crop
00:46:25 nootropic stack, getting canceled by God
00:47:15 lunch, decompressing about Qualcomm
00:47:35 Apple will never sell chips, only God can cancel me t-shirts
00:47:50 Alex Jones, George's politics, conspiracy theories, Alex Jones youtube
00:49:25 mainstream media so bad they took off Alex Jones
00:49:43 the soviet and american talk about propaganda
00:50:15 no andrew tate talk, sigma male discord
00:51:25 Unisoc T820 Processor
00:54:35 sigma male discord fake, business proposals via email
00:55:50 lost $5000 on cheapETH, great about haters
00:58:10 Llama 2 after back from lunch
00:58:22 how IBM still exist, zombie companies
00:59:20 google in bad era, microsoft turning around
01:00:20 history of woke era
01:01:50 sam altman, elon sense of humor
01:02:40 worldcoin orb
01:03:10 sam altman, gm shuts down cruise, microsoft shuts down openai
01:03:55 paperclipped by openai, stories about effective altruism
01:05:00 Lana_lux raid
01:05:40 Elon making the blue line going up, BAP
01:06:05 Nietzsche, Thus Spoke Zarathustra book
01:06:35 tiny box on full power noise
01:07:22 case for tiny box
01:07:45 tst_disk.sh
01:08:20 tst_gpu.sh
01:09:00 gpu_reset.sh
01:09:40 AMD software not the best
01:11:55 don't trust ROCm env
01:11:40 can't buy from hisilicon kirin
01:11:55 roko's basilisk
6
-
6
-
6
-
6
-
6
-
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1150 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org
github.com/tinygrad/teenygrad <- If tinygrad wasn't small enough for you...
github.com/tinygrad/tinygrad <- simple powerful deep learning framework
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:01:00 Apple M3 MAX
00:01:25 Logitech MX Anywhere
00:02:20 tiny corp has an intern
00:02:40 Sam Altman, Effective altruism, Eliezer Yudkowsky
00:06:30 openai, gpt4, EA
00:10:20 advice to openai, regulatory capture, contribute to open source
00:12:42 buy a comma 3x from $1150
00:14:40 people on reddit about openpilot
00:15:35 fire alarm for AGI, violence to stop AI
00:17:00 AI safety, fake AI safety
00:19:25 when AI takes over country
00:20:10 tinybox design reveal
00:20:25 EA, openai regulatory capture
00:22:18 first stream since big drama
00:23:46 problem with political philosophy
00:24:44 argentina currency
00:26:40 mad tasty, Logitech MX Anywhere, hhkb2
00:28:15 daily calm meditation
00:32:00 comma body
00:32:40 teenygrad
00:35:00 5000 lines warning light
00:35:35 what is teenygrad, front end of tinygrad
00:41:55 mnist.py
00:44:35 rawcpubuffer
00:46:50 GPU, apple neural engine
00:47:00 BEAM=2
00:50:35 what things do, what things are
00:53:20 rewrite code to be more clear
00:54:45 favorite test
00:57:50 adding type support to teenygrad
01:01:10 test_reshape
01:08:28 keyerror float64
01:16:30 dataclass
01:22:10 torch repeat
01:29:00 teenygrad is complete
01:31:10 CI failed
01:38:10 dtypes
01:40:40 trash code someone should improve it
01:44:30 adding a lot of lines to teenygrad
01:59:35 astype np
02:01:44 1000 lines
02:11:00 AMD tensor cores
02:12:24 test_real_world.py
02:13:07 mypy
02:30:30 teenygrad pre commit hooks
02:35:05 crappy gpt
02:37:50 yaml lint
02:45:00 nektos act github actions locally
02:47:40 tenstorrent fake buy botton
02:49:30 versions-manifest.json
02:52:35 --container-architecture linux/amd64
02:55:40 cool websites George reads everyday
02:58:05 tiny9
02:58:50 tinygrad discord colors
03:02:40 wandb
03:10:10 tinygrad intern, comma intern
03:12:50 what do you consider genius
03:15:00 IQ
03:19:25 what you can change, what you value
03:19:48 where you apply yourself, what do you believe in
03:25:10 useful advice
03:26:25 what did you change because your watch George
03:31:05 how much money do you need to live?
03:33:30 asshole, how George got money
03:40:50 motivation, find your passion
03:43:10 living in a van
03:44:10 kim dotcom us economy, diet coke
03:45:10 biggest regret, twitter
03:45:50 Elon monarchy is not perfect
03:46:38 2K a week for interns at tinycorp
03:49:50 twitter grok
03:50:05 How to not fail at life. quit bitching about the bitches edition.
6
-
Watch youtube.com/watch?v=iFUmWho7fBE - The AI Alignment Debate: Can We Develop Truly Beneficial AI? | tinygrad.org | github.com/geohot/tinygrad
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/realGeorgeHotz & twitter.com/__tinygrad__ to be up to date | Read George's geohot.github.io/blog/
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org
Chapters:
00:00:00 muted intro
00:01:02 un-muted intro
00:01:19 WARNING LOUD!!! qualcomm stock, sell chips
00:02:35 the debate, unpracticed in art of debate
00:03:15 George in echo chamber
00:03:35 doomer, intelligent AI in the future
00:04:25 to win debates rules for radicals
00:05:25 let's expand to the universe, generative tiktok
00:06:15 not e/acc, use more energy, colonize the moon
00:07:02 good mood George, AMD stock
00:07:40 tiny corp, banning people discord
00:08:01 warning for all of you, you should be aware of this
00:08:45 coordination technologies
00:08:55 tiny corp has money, future tinycorp
00:09:20 helping qualcomm, accumulating wealth in a dead world
00:10:20 hype about the future, mcdonald's on moon
00:11:22 invite for serious people to debate
00:11:35 bored with livestreaming, out of echo chamber
00:12:10 excitement about programming
00:13:15 never debating Destiny, Connor Leahy
00:14:05 debate Trump, livestreaming problem
00:15:10 better infra
00:15:25 founder that spends time coding
00:16:30 leverage, hire people, seriously building a company
00:18:00 bitcast PR
00:18:10 you want people to come to you
00:18:25 comma works because clear vison
00:18:55 tinycorp can be anything, tinybox, infra
00:19:35 people around when starting comma
00:19:46 escaping being alone in tinycorp, conviction on a vision
00:20:14 comma will solve fsd with e2e solution
00:20:24 weed every day, beat Elon when starting comma
00:21:16 san diego realtors, qualcomm, clear vision
00:22:05 richard s. sutton, the AI goat
00:22:25 leave san francisco, vegas
00:23:20 moon colonizing company, antarctica
00:24:20 prime intellect inc
00:24:40 hate san diego realtors, it's not the place
00:25:40 hire people and get serious
00:26:00 good open ai infra, accessible ai
00:27:30 putting time in not getting value out
00:28:00 back to simple tinygrad
00:28:15 enterprise customers, free software, train your ai on tinybox
00:28:35 500 pre-orders for tinyboxes at $15k each
00:29:00 time to build tinybox, margin, improving software
00:30:00 literally making people
00:30:40 hard starting a company, stable tiny box is hard
00:31:40 training and inference
00:32:20 tinygrad is a very good idea, DEBUG=2
00:32:50 being better at debates, somalia
00:33:30 bronze age mindset owned space concept
00:33:40 tiny box prototype tiny 5
00:34:10 comma will build tiny boxes, everyone will have agi vs nuclear bomb
00:35:10 connor leahy unique points
00:35:30 George is not comma
00:35:55 agi spam and psyop filter, home base for AI
00:36:40 coding on livestreams
00:37:10 software to replace professional managerial class
00:38:15 hard to hire, Marc Andreessen
00:39:10 not CEO, program and set a technical vision
00:39:30 operation role would get office, people, manage them
00:39:40 incompetent or scammer
00:40:50 want AIs, not existential crisis, sam altman
00:41:29 people that George would hire
00:42:00 pen testers comment trigger
00:42:40 George knows what he is saying, people at comma understand
00:43:10 people don't understand
00:43:30 pretending to be crazy
00:44:15 filter bubbles, the internet exposed people to different ideas
00:44:50 friends high school IQ, adverse selection effects
00:45:20 internet commenter vs average person
00:45:40 does Lex Fridman understand George
00:46:20 phone poster vs desktop poster IQ
00:46:48 EQ not real, Lex great interviewer
00:47:35 threads vs twitter replies
00:48:00 finding a stable place, not drifting forever
00:48:20 not selling bad things, not a scam, no timeline
00:48:55 LK99, how to scale up, limits, palladium
00:49:40 love America, first amendment, free speech absolutist
00:50:25 free speech meaning, dmca
00:50:42 europe does not have free speech
00:50:52 you don't own space in america
00:51:02 john mcafee chad
00:52:00 Iain M. Banks, The Player of Games, rich sutton reinforcement learning
00:52:35 not absolute free speech, AI that are aligned with you
00:53:33 the local vs the cloud plug, buy the tinybox
00:54:10 Say Anything - Psyche!
6
-
6
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | George is streaming right now on https://www.twitch.tv/georgehotz part two of stable diffusion | github.com/geohot/tinygrad | github.com/CompVis/stable-diffusion | Support George by subscribing on twitch.tv/subs/georgehotz | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:50 feelings
00:03:20 stable_diffusion.py
00:04:40 difference between George and Andrej
00:05:25 cannot create an OBJECT array from memory buffer
00:07:50 known bug in tinygrad
00:08:37 what is stable diffusion, opensource, AI safety
00:10:00 safety checker
00:11:35 what this models are
00:13:15 CompVis/stable-diffusion, reading the code
00:17:30 ready to give up, learning, advice, nathan for you, passion
00:18:34 32GB VRAM it's a mac and it's shared
00:18:48 UNet attention, where are the inputs and outputs
00:20:10 what parts of transformer matter, terminology
00:21:22 about stable diffusion, UNet attention
00:22:45 dictatorizing, problem with the world, democratizing
00:24:03 privileged, triggered
00:25:20 Axel Nix lawsuit comma, assholes in the world
00:27:12 Cool people in the world, rant about Aurora, Tesla wining self driving
00:29:25 Elon Musk, billionaires, zoox, Aurora scam, people like scams
00:30:45 stable diffusion paper, LDM
00:32:05 learn latest tricks in deep learning
00:32:21 glasses, astigmatism, health insurance premiums
00:33:05 why three pieces
00:35:25 how it works?
00:40:49 thanks to flavioschneider for real information
00:41:25 taking a look at Unet
00:43:40 Autoencoder, basic idea about diffusion, encoder
00:45:05 first stage model, transformers are sample efficient world models, eloialonso/iris
00:47:00 google-research/rliable
00:48:10 facebookresearch/hydra
00:48:45 how clip model get's to UNet
00:51:15 hating too much on the paper
00:51:40 a walk in the park learning to walk in 20 minutes
00:52:25 jax.readthedocs.io
00:53:30 training is insanely hard
00:54:30 implementation of stable diffusion
00:55:35 numpy assign, assignment, insert, np.copyTo
01:05:50 it match pytorch exactly, easier to load things
01:06:48 break, uber eats in iceland
01:13:48 banning a person in chat
01:17:00 it's outputting a distribution, freedom of speech can't be bought
01:17:30 quant_conv, nin_shortcut
01:21:05 class StableDiffusion, AutoencoderKL, Encoder, ResnetBlock
01:28:45 How to Catch Multiple Exceptions in Python
01:29:35 stable_diffusion.md
01:33:20 asymmetric padding supported in tinygrad, class Encoder, ResnetBlock, StableDiffusion
01:51:20 break, uber eats, class Decoder
02:35:25 iPhone to dad, apps manipulation
02:35:50 torch.nn.functional.interpolate
02:42:35 tuple object cannot be interpreted as integer
02:48:08 cannot handle this data type
02:49:30 black square because noting in text-prompt
02:50:45 need to write groupnorm
02:57:04 picture like art
02:58:05 break, the graph
03:00:52 class AttnBlock
03:06:00 image looks the same
03:08:40 torch layernorm
03:11:15 orange square, deep learning is not like cryptography
03:17:40 break, single color square, there is bugs,
03:22:50 don't want to do this properly
03:30:30 doing it the proper way
03:35:25 so much fun, kids you could get into computers, github copilot is not good
03:39:08 PyTorch image, noise is better then solid color
03:43:18 looks stupider, is it trained?
03:48:05 don't know if better or worse
03:49:00 it works, before and after image
03:55:00 Torch apple, we are nothing but script kiddies
04:00:30 negative apple, five times apple
04:05:15 we don't have anything working at all
04:09:05 let's dive in block one
04:18:00 bug in gpu implementation in tinygrad
04:22:30 make a test
04:27:20 overflowing int, this is not python
04:36:00 PyTorch vs tinygrad difference, opencl tinygrad backend
04:41:30 Alex on the call, clinfo on quiet computer
04:45:15 very serious bug in tinygrad
04:47:15 Lana_Lux raid, works on gpu in other computer
04:49:10 this is why Nvidia is king, thinking about simple fix
05:14:00 big victory for tinygrad is running openpilot model
05:18:05 Alex bringing drink, tinygrad code easier to read
05:25:50 explaining the problem
05:33:10 raiding Lana_Lux making twitch loop
05:38:25 cloudflare The Project Jengo Saga, free speech
05:40:15 here is something we can do AutoencoderKL
05:45:35 still to do diffusionmodules, FrozenCLIPERmbedder
6
-
6
-
Watch youtu.be/lO0KH-TgvbM (Peter Thiel on The Diversity Myth Lectures) | newcriterion.com/issues/2023/6/the-diversity-myth | italianfuturism.org/manifestos/foundingmanifesto | graymirror.substack.com/p/a-techno-pessimist-manifesto | geohot.github.io/blog/jekyll/update/2023/07/20/a-disgusting-playbook-copy.html | ecromata.com | palladiummag.com | longform.org | newcriterion.com | www.astralcodexten.com
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
https://youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | https://youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life | Read en.wikipedia.org/wiki/The_Elephant_in_the_Brain
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 muted intro
00:00:22 un-muted
00:01:40 everything that is wrong in your life it's your fault
00:02:44 a techno-pessimist manifesto, long term thinking
00:03:50 Tucker Carlson, Julian Assange
00:05:30 The Diversity Myth by Peter Thiel, Samsung Z Fold5
00:11:05 wokeness, universities, acceleration is good, wealth distribution
00:14:20 no progress in physics, energy usage, no progress in science
00:16:30 usa life expectancy
00:17:40 secrecy in startups, microeconomics problem, housing
00:21:00 no growth, Henry Adams curve, christianity
00:23:40 2010s, fed interest rates, high interest rate
00:27:50 progress, newcriterion, society complex
00:31:00 Griggs v. Duke Power Co., IQ test, correlation between SAT and IQ
00:32:35 human capital, practical engineering youtube
00:34:30 rasism, fix human capital, techno capital
00:35:35 tabo, secrecy, telling the truth
00:37:10 rent double in san francisco, energy usage usa
00:39:00 energy usage per capita china, germany
00:39:50 listen to The Diversity Myth by Peter Thiel
00:40:30 loss of status, harvard, ivy league
00:42:10 1915 plane, 747, 787 dreamliner, Airbus A380
00:44:30 executive functioning, business
00:45:50 argentina, el salvador, argentina currency, Bird Global, BRDS
00:48:15 nuclear power
00:48:40 a techno-pessimist manifesto, Marc Andreessen
00:50:32 futurist manifesto
00:52:30 left, right
00:53:15 phone usage hours per country
00:55:20 south africa
00:56:25 lil mabu
01:00:50 don't walk to George, socially awkward
01:01:30 accelerationism mass movement
01:02:50 william james, sent-down youth
01:04:30 George Hotz brand, making people useful
01:07:45 ecromata.com
01:09:00 what should I do?
01:09:45 robotics laborm, comma.ai/shop/body
01:10:30 living property developer, off grid, week in the woods
01:13:00 George vs animal, hunting
01:14:20 Kanye West city in the Middle East
01:18:10 Humans Need Not Apply
01:20:18 resource curse, UBI
01:22:45 long term ideology, human power over nature
01:24:10 George as a manager
01:24:33 don't talk to George unless you are cool, fans
01:25:20 Curtis Yarvin, Peter Thiel describing the present
01:25:50 cult, deploying money and human capital
01:26:25 what does Elon have that George does not, risk tolerance
01:27:10 Sam Altman, openai new blog post
01:28:20 AI alignment and capability issue
01:29:25 due to concerns about large language models
01:30:30 deploying capital ability, lying ability
01:31:55 a disgusting playbook blog post
01:32:46 lies do catch up with you
01:33:00 Elon internet space internet idea
01:34:20 bounties, business, utilitarian ethics
01:35:25 thymic energy
01:36:20 to partner with somebody
01:37:40 you can buy starlink
01:38:10 hyperloop and rail projects
01:40:05 showtune song, instagram
01:40:40 USleeve, BetterHelp, Squarespace, Surfshark
01:42:35 amazon firestick in a hotel, GTA leaks
01:44:40 Kevin Mitnick, 'code is law' ethos in crypto
01:45:45 make hacking legal
01:46:40 p vs e, p v p zone, binance smart chain
01:49:20 cheapETH, devETH, lost $5000 on cheapETH
01:51:30 no answers, happy holidays
01:52:14 atheism, merry christmas
6
-
6
-
6
-
6
-
From $1499 buy comma.ai/shop/comma-three | github.com/geohot/tinygrad/blob/llama/examples/llama.py | https://github.com/geohot/tinygrad/tree/llama | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 muted intro
00:00:50 missing the rant about bad mood
00:01:35 not family friendly content
00:01:55 hardware to run tinygrad
00:02:10 don't mess with george
00:02:35 what we did yesterday
00:05:00 where are we wasting time, not limited by gpu
00:06:20 python snakeprof
00:10:30 tinygrad discord llama channel, hacker news blow up
00:10:45 llama weights on torrent
00:11:20 does it run on nvidia 3090, ram
00:13:30 open assistant product, prompt engineering
00:21:10 no limits, don't believe in ai safety
00:21:40 no guns, ai stupid
00:23:55 what you need to torrent for weights
00:27:25 torch.triu
00:32:25 nice pytorch error messages
00:36:45 russian mode
00:38:00 python bottleneck
00:41:50 twitch chat not paying attention
00:42:50 pull request to facebook
00:43:50 hacker news, gotham chess plays chatgpt
00:45:30 chatbot loop
00:47:00 asking questions, bad enter
00:50:20 no ai filter
00:51:40 interesting output
00:53:50 sentencepieceprocessor decode_piece
00:58:00 removing extra enter
01:00:50 decode token to token, this is the problem
01:02:50 we learned how to take it out of russian mode
01:03:50 need demo answers, it's called llama chat
01:06:30 be verbose in your answers
01:08:00 call back humor answer
01:08:50 pastebin improved pre prompts
01:17:00 not Gale, your name is Brian, cookies
01:18:00 what is your prompt
01:18:35 arbitrary python
01:22:45 impressive large language models
01:30:50 running on M1 Max
01:36:30 llama link
01:37:30 Brian running wild
01:39:00 real talk about ai safety
01:40:20 chat llama open source, running in web browser
01:42:20 funny response
01:43:45 connecting to the internet
01:45:00 model too small, too many tokens
01:47:00 darth vader, used car salesman name
01:50:30 Lana_Lux raiding stream, explaining to new people
01:51:15 what makes chatgpt good
01:52:00 fun answer
01:53:00 Gary only has 7B weights
01:54:10 chatml, need smarter model
02:00:00 what we should do?
02:01:00 bad rapper
02:03:20 stacy
02:07:15 limits of 7B weights
02:08:50 good answer
02:09:25 assuming gender
02:11:25 searching snapchat mentee.dreams
02:12:12 replica terrible, asking more questions, rap
02:17:10 13B weights smart stacy, vram requirements
02:19:30 running on cpu, took a day to write this in tinygrad
02:21:55 comma build the best self driving car, tinygrad taking over the world
02:22:40 removing code, repetition penalty
02:30:10 testing stacy
02:36:10 sp_model.decode_pieces, bos_id
02:43:20 fun answer, need to make large work
02:50:10 reading the LLaMA paper
02:52:10 anyone running this on his mac?, code pushed
02:53:45 macbook air m2
02:59:15 pytorch split file
03:01:30 trying to load the bigger model
03:13:00 python readinto
03:16:20 load only 13GB because of gc
03:18:20 making copy = bad
03:20:00 apple m1, m2 max, ultra memory
03:21:30 v.shape, tensors
03:23:00 real offer sponsor, tinycorp honest corp
03:32:10 millionaire
03:34:50 bad chatter, DEBUG=3, pointer, constant folded
03:38:40 the problem and easy fix, mailing the cookie for anyone who knows, thai food
03:44:30 what type is the output buffer
03:47:45 does not use ram because of fake tensors
03:49:55 mid model, loading 13B weight model
03:55:30 bad chat WEIGHTS=0
03:59:30 closing apps, chrome, discord
04:01:50 mac process disable swap, metal buffer force ram resident
04:05:00 force os x buffer to not swap, metal makr bugger as not swap, MTLResourceStorageModeShared
04:09:55 mac os x disable swap for single process, apple gpu supported types, pci-e 4 16x bandwidth
04:16:20 improving prompts, strassen algorithm
04:23:00 let's go
04:27:25 getting food
04:35:00 marc andreessen
04:43:00 who is stacy
04:48:40 git commit add gary
04:49:00 context infinite
04:49:20 gary vs stacy
04:49:40 ai george hotz disappointment
04:50:20 stacy saying thanks for watching
04:51:00 llama branch 24GB ram min requirement, having fun with stacy this weekend
6
-
6
-
6
-
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
03:48:35 Hiring entire stack for tiny corp join if you are interested | 03:56:30 work major source of value in your life | Read en.wikipedia.org/wiki/The_Elephant_in_the_Brain
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:01:02 writing a Qualcomm GPU driver
00:01:54 ssh to the phone, code server, vs code server
00:02:22 user space GPU driver, graphics kernel
00:05:10 asking Quentin and Perplexity labs
00:13:00 regex pattern allow spaces
00:21:30 parsing struct
00:25:45 GPU driver, how long in C
00:31:05 tinygrad import Tensor
00:35:40 init, most of GPU not controlled in kernel space, single layer pointers
00:36:55 replay this GPU command
00:38:10 TinyJit, tinygrad prints
00:44:30 kgsl_gpu_command, msm_kgsl.py
00:49:10 Samsung Galaxy Z Fold5
00:49:45 who knows that green is executable +x
00:50:18 linux dma-fence.h
00:53:00 offset gpuaddr size flags id
00:58:20 struct_kgsl_command_object
01:06:25 building a driver before building a chip
01:07:00 jobs creation, comma lesson
01:08:25 thneed openpilot, mmap64
01:12:30 gpu command, parsing the command buffer
01:19:45 freedreno driver, freedreno cffdump
01:23:05 Perplexity labs hallucinating
01:25:40 rnndb database 404 link
01:28:25 real command buffer
01:31:00 PM4
01:35:20 pkt_is_type7
01:41:44 alloc memoryview writable
01:46:55 type 7 and type 4 packets
01:48:40 very cool freedreno people
01:49:05 qualcomm switching to opensource driver
01:50:00 writing registers, opcode
01:52:40 number of globals, and total number
01:54:40 adreno_pm4_type3,7_packets, 7 commands, 4 write registers
01:56:50 python format string add 0x to hex
01:57:55 opcode 17, CP_THREAD_CONTROL or IN_IB_PREFETCH_END
01:59:53 python parse xml, xml to json
02:07:15 not a real name, no leak for qualcomm driver, amd, nvidia open source
02:08:30 HLSQ_CONTROL_2_REG
02:09:25 nvidia open sourced the kernel stub that talks to gsp
02:11:20 CP_CONTEXT_REG_BUNCH2
02:13:45 CP_REG_TO_MEM
02:14:50 GPU performance counter to GPU memory
02:17:30 SP_CS_INSTRLEN, HLSQ_CS_NDRANGE_0
02:20:40 the simplest drivers that can dispatch compute kernels
02:23:00 copy out, unified memory space between CPU and GPU
02:26:00 objlist cmdlist, pkt types
02:36:00 stuct_kgsl_cmdbatch_profiling_buffer
02:42:20 printing in hex
02:45:20 python address of bytearray
02:47:00 making another copy
02:48:40 strace, copyin
02:52:30 two CP_RUN_OPENCL
03:01:00 CP_LOAD_STATE6_FRAG
03:04:20 a, 9
03:08:00 python hexdump
03:11:40 GPU drivers can be written in python
03:12:50 dumping the shader
03:21:25 freedreno shader registers r51
03:26:40 c20 hex 14
03:30:00 CP_LOAD_STATE6_0_STATE_TYPE
03:37:50 adreno registers
03:38:10 replacing the shader compilers, mesa for compute
03:41:30 replaying the command buffers
03:44:25 stream overview, bath, meditation
03:45:00 when tinygrad can run without dependencies
03:46:15 remote work, there is something missing
03:47:20 removing numpy from tinygrad
03:48:35 hiring entire stack for tiny corp
03:51:00 high equity, contribute to the project
03:53:15 trolls on twitter
03:54:00 comma is doing well, san diego
03:55:20 the dream of the internet, Elon is accessible
03:56:30 work major source of value in your life
03:57:40 sustainable business of open source
03:58:40 clear blog post about AI space, not trying to be understood
03:59:00 AMD mindshare, people at AMD, AMD bugs, issues
04:02:05 thinking longer term
04:04:33 college majors by iq
04:05:10 it doesn't change. if you are 30 your life is what it is, limitations forever
04:07:55 following your passion, your passion is stupid, singularity
04:12:10 against education, the elephant in the brain
04:19:30 christian religion, the bible, game over screen
04:21:55 the invisible rainbow, 5G
04:25:45 naming 10 books
04:29:40 not debating destiny
04:30:00 tiny keyboard
04:32:15 general knowledge science questions
04:41:20 human brain compute, wormhole
04:44:10 time dilation, sr vs jk flip flop
04:46:30 attention mechanism in transformers
04:47:40 e minor, supernova
04:48:50 Ra by qntm
04:50:10 milky way galaxy, quantum mechanics
04:52:34 power companies high voltage, black hole
04:55:20 polaritiazon in electro mechanics, einstein nobel prize
05:00:00 metal rod in space
05:01:20 universe not locally real
05:02:40 thank you for watching, human nature
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
6
-
Read www.incompleteideas.net/IncIdeas/BitterLesson.html | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 muted intro
00:00:40 un-muted intro
00:00:50 taiwan and india
00:01:50 how free is taiwan, china, check in hotel
00:03:20 free internet in taiwan
00:04:00 wechat in china
00:05:55 story time, iceland
00:07:00 safe singapur
00:09:20 brazil, south africa
00:11:10 George trying not be controversial
00:12:10 grants, fixing the empire
00:13:45 dismantling our late-stage capitalist society
00:14:40 George stating his political preference
00:15:30 Elon Musk the next Caesar, bike trip
00:17:00 Ukraine gdp per capita, taiwan gdp per capita
00:17:50 progressive, can't make everyone happy in politics
00:18:20 sarcasm
00:19:00 eastern europe bike trip, andrew tate
00:20:10 countries to avoid
00:23:05 shipping for non-US citizens for tinybox
00:23:45 sweden, dubai
00:28:25 direction of tinygrad
00:30:30 building a wall
00:31:00 declining empires
00:32:50 china spending power, kenya poor
00:35:00 favorite countries in europe, dubai
00:36:15 libertarian india
00:36:50 hand coding resnet 50
00:40:10 kernels in resnet
00:43:05 operations for the resnet, the power of the new tinygrad abstractions
00:46:10 simple abstractions in tinygrad = fast
00:46:20 init does lots of adds
00:47:25 renderer, kernel code
00:48:55 cleanest abstractions anywhere nn
00:49:20 not optmized kernel, optimization
00:49:50 benchmarking
00:58:40 kind of wrong, some kernels bad
01:01:35 how nice is tinygrad
01:08:40 love for python
01:09:30 faster
01:11:20 MadDogNation is gifting 34 subs!
01:12:40 focusing on slowest
01:21:25 multiview shapetracker
01:25:40 lin.upcast
01:27:10 reshaping to real size
01:27:40 very happy with this infrastructure
01:31:35 let's make it faster
01:35:25 metal max fmax
01:39:25 gated loads, finding bugs
01:41:30 108 GFLOPS
01:43:30 we have to start thinking
01:47:30 we build an optimizer
01:49:30 upcast, search and optimization is amazing
01:52:25 out of ideas
01:59:50 that kernel just looks fast it just isn't
02:03:20 that kernel was not the problem
02:04:45 automatic optimizer
02:07:25 10% faster, we learn, knowledge is power
02:08:15 running this anywhere
02:08:50 4416 lines, 1000 lines, new goal 5000 lines
02:09:10 thank you for gifting subs!
02:09:40 on topic questions only for non-subs
02:10:03 up to the front tinygrad, short and generic
02:10:55 every part of a full deep learning stack
02:11:05 resnet 50 pytorch like code
02:11:25 tinygrad is a waste of time?
02:12:00 why is pytorch faster? pytorch less capable library
02:12:20 cudnn hand optimized by 20 nvidia engineers for 10 years
02:13:00 the bitter lesson
02:13:10 can't I just call CUDNN?
02:14:05 generic optimizer so much simpler to reason about
02:14:20 learn a model that look at shapetracker to optimize
02:14:40 recursive self-improvement
02:14:50 AGI inside tinygrad
02:15:20 AGI will run in 4k lines
02:15:40 every piece of tinygrad you can reason about
02:16:15 no more debates, heat death of the universe
02:16:45 AIs will come, monkey civilization
02:17:15 why people think AI is dangerous
6
-
6
-
High quality version uploaded -> youtube.com/watch?v=L_C4zLnekLI | Thank you to comma.ai team for providing high quality video and recordings of this amazing event!
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | If you have skill apply on -> comma.ai/jobs | Join the community on github.com/commaai/openpilot | Check if your car is supported -> comma.ai/vehicles | How to setup your car comma.ai/setup | follow the official youtube.com/commaai for more livestreams | Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop.
Chapters:
00:00:00 where is my intro
00:00:45 black mirror episode with ratings
00:01:30 three stories, past, future, present
00:01:40 the past, cruise automation
00:01:58 cruise RP-1 highway autopilot
00:02:20 giving up on technical challenge
00:02:30 too many cars to support
00:03:10 how to handling failures
00:03:55 fingerprinting
00:04:30 comma was not supposed to exist
00:05:15 what is worth doing in the world
00:06:10 6 generations of hardware
00:07:00 keep iterating the hardware
00:07:25 being real, honesty, communication people
00:08:20 shipping keeps you honest
00:08:30 stated preference, revealed preference
00:09:04 iteration
00:09:25 humility, egohot, big ego
00:10:36 so why haven't we won?
00:10:47 things take a long time, even accounting for things taking a long time
00:11:23 no exit, the system will get better
00:11:49 climbing out of a well
00:12:38 it turned out these companies were scams
00:13:28 bitconnect and luna scam
00:13:55 marginal utility - marginal cost
00:14:10 value destruction, linux fastest growing platform on Azure
00:14:50 markets - rationality, you just have to survive
00:15:05 lifetime revenue
00:16:05 $22.2M on $18.1M raised
00:16:18 twitter followers
00:16:35 future followers
00:16:45 employee retention
00:17:03 come work to comma ai all work by the team not George
00:17:18 keeping idiot's aways from comma
00:17:35 bar very high, you get to ship things that get better in the world for the better place
00:18:18 business world
00:18:30 the future
00:18:44 computing speeds vs human brain computation
00:19:30 forward looking statement
00:19:44 comma con 9 in year 2037
00:20:00 44 stores in malls, top 10 biggest consumer electronics companies, without selling out
00:20:52 hard to go anywhere without finding a comma, cars, bodies, puppies
00:21:20 comma body
00:21:30 comma 7 for $2500
00:21:46 3rd generation comma ASIC, 50000 TOPS
00:22:50 first comma device to boot into tinygrad
00:23:05 trying to build people
00:23:30 trolling Elon, out system is going to so end to end even or driving monitoring will be end to end
00:23:50 comma 7, how many silicon people work a comma
00:24:08 your comma 7 is yours, models we train in the cloud are foundation models
00:24:55 thank you to meta for LLaMA, open source AI
00:25:13 offer pretrained policy models, custom software
00:25:41 comma 7 is like child
00:25:52 will learn behaviors you teach so teach it well, pick up skills as quickly as smart human
00:25:02 comma 8 will be running the company
00:26:12 comma three human head
00:27:00 humans are so good at moving, driving
00:27:35 intelligence
00:27:46 the present COMMA_CON 2
00:28:10 the mall store soon
00:28:30 sitting down for dramatic effect
00:28:55 failure rates for iPhone, failure rate for comma two
00:30:10 comma three failure rate, comma three failure rates by month
00:31:32 dropping the price
00:32:00 comma 3x
00:32:15 much better injected molded case
00:33:00 3X specs, same hardware and software as comma three
00:33:34 upgraded cameras, 140 dB HDR
00:33:55 128GB of storage
00:34:14 CAN-FD integrated
00:34:28 still OLED better supplier, onboard road cameras
00:35:24 stereo in-case sound
00:35:45 upgraded power architecture
00:36:05 about half the components, more reliable
00:36:14 the comma 3X board front, no more GPS connector, better antennas
00:37:15 the comma 3X board back
00:37:45 battery slides in and out, chokes for CAN-FD, e-sim
00:38:10 cameras and right on the boards, failure rate of this device need to be 1 or 2 %
00:38:52 comma three vs comma 3x board complexity
00:39:02 no more NVMe, just on-board flash, QR codes on the board for tracking
00:39:35 is it a devkit? to set expectations
00:40:00 it's time to call it just a comma 3X, mass market
00:40:50 comma 3X just $1250
00:41:20 no more questions that is why it's just one edition
00:42:10 it's just a comma three but better
00:42:30 order now, ships in 1-8 weeks
00:43:10 comma 3X in EV6, no more red panda, no more two harnesses
00:43:20 when did the iPhone get good
00:43:50 the comma team build this not George
00:44:50 value in running language models on the device
00:45:30 marketing guy, testing on commercial vehicles
00:46:55 marketing, the story of openpilot, openpilot ADAS
00:47:40 buy now, 30-day money back guarantee.
00:48:05 comma 3X always compatible with cars
00:48:40 port a new model to comma 3X
00:49:35 mass market, liability air gap with installing custom software
00:50:25 level 2, you must pay attention at all times
00:51:30 large foundation model on device, expansion port
00:53:35 failure rate, should be build like a tank
00:55:00 saying no to things, best highway experience
6
-
6
-
6
-
6
-
6
-
6
-
5
-
5
-
5
-
5
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | Go detect your own hen -> https://tinygrad.org/recognize.html | From $1499 buy comma.ai/shop/comma-three | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:02:20 c backend for $2500
00:06:15 elon and george and twitter
00:10:15 bad user in chat
00:15:30 random extension of power
00:21:30 beautiful c code
00:28:20 ctypes load library
00:29:25 os path file exists python
00:31:25 pain of using C, multithreading
00:38:00 np.ctypeslib.as_array
00:41:25 c99 max function
00:43:30 c backend in 33 lines
00:49:45 indexError: list index out of range
00:52:50 segmentation fault 11
00:54:40 running tinygrad example
00:59:00 CLANG=1
01:00:40 using rust
01:01:40 thank you to everyone who keept the example working
01:06:10 gpu runner, jit_cache
01:08:10 args, buffers, malloc
01:19:10 CLPrograms error
01:20:45 loading the weights, identifying the inputs and outputs
01:23:00 c auto formatter tabs, clang-format
01:25:40 copyright, spreading the word, selling royce rolls
01:26:25 lib jpeg pain, linking to opencv, cheapeth scam
01:27:30 tiny corp is broke
01:27:55 bad chat, who can follow this, talking to Alex
01:29:15 can anyone follow this, explaining to noobs
01:36:00 arg max
01:37:55 c float infinity
01:43:35 language translation features from github
01:49:40 fread
01:56:10 512 bugged
01:58:20 how we are going to get an image?
02:00:30 c array of strings
02:03:20 very upset about reading the image
02:04:45 c example libjpeg
02:05:35 link to libjpeg
02:06:20 libjpeg dev homebrew, libpng
02:07:20 png file format, load png in c header only
02:08:10 c link to opencv
02:08:35 worse part of C
02:08:55 stb_image.h
02:21:45 trying coca cola image
02:24:10 remote control
02:25:50 embedding binary data in c
02:37:10 incompatible integer to pointer conversion
02:40:30 so long, putting it to the pipe
02:44:40 binary data in c header
02:49:35 I love temp, temp my favorite folder on computer
02:52:15 completely portable application that runs classifier
03:00:40 banana, hyperpop song
03:03:10 imagesnap
03:05:25 detecting objects via camera
03:08:40 triceratops
03:09:20 sending people to jail
03:11:25 distributing recognize to tinygrad.org
03:15:30 banana, c time nanoseconds, code html5
03:19:25 finding the right image, c assert, bing chatgpt
03:45:10 letting everyone else to test on their devices to fix any bugs
03:52:55 what can tiny corp do that other companies can't do
03:56:00 can you run this command on your terminal
03:58:00 what we should do next
04:01:40 49% of tiny corp to microsoft for $10B
04:02:10 openai code and models
04:03:25 George trolling the chat, vulcano story
04:04:40 Andrej Karpathy joining openai
04:05:30 what features to remove from c++
04:07:00 refactors to do
04:09:40 responsible ai, ai safety, anti democratic
04:12:30 chatgpt safety filter
04:19:10 bash command line assert output
04:21:45 tinybrowser, nn simple, tensors
04:24:10 compile c to wasm
04:28:30 python3 -m http.server
04:30:25 svelte wasm
04:32:30 javascript fetch
04:46:55 .wat file, wasienv
04:48:35 wasm
05:03:50 fromCharCode
05:14:00 javascript abort
05:16:40 SEEK_CUR
05:19:50 it works
05:26:40 upload file from javascript
05:35:30 doing style
05:51:50 connecting to webcam
05:55:15 fixing img box
06:05:20 camera capture
06:39:05 ios webcam from non https
06:44:20 working on iPhone
06:45:30 HackerNews
06:46:00 real future ideas
06:48:10 Alex on the phone
06:50:20 fixing bugs
06:56:25 governance tokens
06:57:20 happy hacking keyboard
06:57:50 what did we do today
06:58:15 net's in the browser
06:59:00 tiny corp no money
06:59:45 trap drums
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | From $1499 buy comma.ai/shop/comma-three | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:02:00 n02018795 imagenet categories
00:05:30 staying on topic, tinygrad discord
00:06:00 rap battle
00:07:25 benchmark_train_efficientnet.py
00:10:05 what net to use
00:11:00 how to contribute to tinygrad
00:12:55 state of the art for training imagenet
00:13:10 tinygrad discord, CLCACHE
00:14:00 pull requests
00:15:00 no nix, rust, llvmlite, cuda
00:16:30 what paper to implement
00:18:40 resnet paper
00:23:10 CIFAR-10
00:25:20 1080 ti
00:25:30 torchvision installing issue
00:26:30 nix does not solve this
00:27:55 xkcd standards
00:29:30 net ema
00:30:44 maxpool2d
00:36:30 19s runtime in pytorch
00:38:20 tinygrad discord strict rules
00:38:38 hackernews people, maslow's hierarchy of argument, conversation
00:39:10 life after degree, ideas, honest conversations
00:41:10 compassion will hold you back
00:42:55 elon's rules, rule 0 surface complexity, building hardware, vertical integration
00:43:55 no reason for this to be a class
00:51:40 pytorch unsqueeze
00:52:45 beautiful implementation
00:55:35 ResNet whitening
00:59:20 shorter, better than pytorch
01:01:25 dumbest question
01:03:25 fire tweet, ai safety, old people text
01:04:15 fate of the world, twitter saga, internet get things wrong
01:05:40 jai beta
01:07:00 creating an optimizer
01:09:00 broken hoodie
01:09:30 batchsize 512
01:10:40 642 vs 53 lines
01:13:50 make_pair and first attempt at hlb_cifar10
01:15:00 DEBUG=2
01:17:50 more readable code
01:19:00 TEST_AST=1, unable to allocate 144 GiB
01:25:00 gelu is causing nans
01:28:00 64 batch size
01:31:10 nn.CrossEntropyLoss
01:34:30 cifar-10-batches-py/data_batch_1
01:39:20 open-assistant
01:40:40 comma ai safety principles vs waymo safety
01:44:25 bad questions in chat
01:46:00 train_step_jitted
02:05:00 the losses are different with optimizer
02:12:20 the idea of tinygrad performance
02:13:05 printing out the kernels that are running
02:16:20 opencl nvidia tensor core
02:18:40 58 TFLOPS
02:20:15 question in tinygrad discord
02:23:20 why the fusion is broken, tensor cores, float 16 support
02:40:15 17 TFLOPS
02:45:10 arm support for tinygrad
02:46:55 all time spend on gpu
02:50:55 12x off
02:55:20 4x4 matrix
02:58:36 optimizer outputing wrong code
03:05:00 kernels output
03:08:00 -1%10
03:14:10 decode utf-8
03:16:40 fma.rn.f32, opencl cuda enable tensor cores
03:21:40 reviewing symbolic test pull request
03:23:00 sm_86, using triton, kernel search
03:30:50 only 10x off
03:36:00 tinygrad.org, pytorch mps, python requests get
03:43:00 right shapes, lazy.py
03:45:20 tinygrad contributors, over 1000 lines
03:49:45 comparing the cpu backend, pytorch cuda
03:53:35 is tinygrad stupid project
04:01:45 reviewing tqdm pull request
04:08:30 reviewing progress bar pull
04:09:35 highest quality code you ever written in life
04:10:35 housing and food for tinygrad intern (no salary)
04:11:35 tinygrad looking into making money (merch)
04:13:15 comma tiny corp partnership
04:13:45 refactors in tinygrad
04:14:00 employee number 1 pitch
04:21:00 checking food options
04:23:15 make_pair, progress bar pull request
04:34:00 tqdm requests progress bar
04:41:45 bad pull requests
04:46:50 good programmers, money, goat progress bar
04:56:25 new policy chatgpt banned
05:11:50 plan to fix the optimizer, reviewing pastebin code
05:31:30 instagram women dm filtering
05:31:50 testing intervention
05:47:25 NoneType object has no attribute copyin, how much fake speed
05:53:00 the best code is no code
06:12:30 this should only copy the base buffer and retain the shapetracker
06:20:25 aritter2 get's VIP for his good comments
06:28:00 offensive question
06:29:15 commit fix multiple accumulators, ast fuzzer, progress today
06:34:00 interesting code, jeff bezos e/acc, KOPT=2, 6.5x speed up, debug, tinygrad internships
5
-
5
-
5
-
Buy comma three from $1499 https://comma.ai/shop/comma-three | What is comma.ai/openpilot | openpilot documentation docs.comma.ai | What is comma.ai/connect | Join the comma team comma.ai/jobs | Check if your car is supported -> https://comma.ai/vehicles | How to setup your car comma.ai/setup | Read blog https://blog.comma.ai/ready-for-growth/ | Fun timestamp 00:42:55 a wild Jim Keller appears! | Follow the official youtube.com/commaai for more livestreams. Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop.
Chapters:
00:00:00 intro
00:01:00 showing the booth
00:01:35 time at twitter
00:02:00 Adeeb
00:02:45 no partnership
00:02:55 car and driver test drive
00:03:15 no comma four
00:03:50 comma 3 improvements
00:05:00 merge/acquisition, spac
00:06:18 comma works everywhere
00:06:28 Harald
00:06:47 test drives today
00:07:10 comma harness
00:07:38 stop and go at traffic lights
00:08:40 list of supported cars
00:09:55 toyota security
00:10:35 Alex and Shane
00:12:08 announcement of price decrease
00:12:45 comma 3 reliability
00:13:15 sales need to go up
00:13:40 comma 3 life cycle
00:14:15 comma does not work with anyone
00:14:50 marketing intern
00:15:13 best car for comma 3
00:15:33 open source comma3 car mount
00:15:45 how is lower price possible
00:16:33 no trade in for comma 2
00:16:43 raising money
00:17:00 navigate on open pilot in Europe
00:17:10 no refunds for orders placed in last 30 days
00:17:33 made in America, Toyota corolla for taco bell?
00:17:52 manufacturers integrating comma
00:18:15 bosses at comma
00:19:00 price
00:20:25 what would Elon do?
00:20:52 best car for comma 3
00:21:20 is the dev harness the same as the comma serial?
00:21:30 Radar less is ready?
00:22:00 you said you wouldn't lower the price
00:24:10 comma 3 won't work with Tesla (maybe)
00:24:20 check avatar script by George
00:24:30 I have a comma 2. any advantage getting a comma 3?
00:25:00 marketing strategy, bolt euv integration
00:25:15 comma 4 when?
00:25:40 chip in comma 3, c3 on front & rear windows, trolly problem
00:25:55 Red Panda and cables needed
00:26:15 tesla vs comma
00:26:25 comma con
00:26:35 buying with paypal credit
00:26:45 CommaCon confirmed, zoox
00:27:35 works with right side drive cars
00:27:55 how to work at comma
00:28:05 hackathon
00:28:25 comma robotics
00:29:00 love tesla
00:29:10 comma con july 31st this year
00:29:50 partnerships and collaborations
00:30:13 never IPO
00:31:02 experimental mode, nav on openpilot
00:31:20 return rate for comma
00:31:40 lawsuit over
00:31:50 comma 3 work in a boat
00:32:05 showing the comma 3
00:32:55 why newer comma 3 better
00:33:20 high temperature resistance
00:33:55 how many comma 3 sold, failure rate
00:34:15 new SoC vendor
00:34:30 wholesale for car dealers? 10% off
00:34:45 bad and good features
00:35:20 openpilot works in RHD countries
00:35:35 learning on device
00:35:55 stop and go at traffic lights
00:36:30 speed bump detection, making chips
00:37:35 how to contribute to the growth
00:38:00 Shane talking about micd, fork, smart lane switch, support, autotune
00:39:10 how Shane got started with comma
00:39:30 bad experimental mode,
00:39:50 jobs, hiring, car-interface engineer, research, infra
00:41:10 comma connect price drop when?
00:42:02 different storage same driving
00:42:25 remote work
00:42:55 a wild Jim Keller appears!
00:43:55 experimental mode
00:45:35 comma knee
00:45:55 snow, warranty extension
00:47:25 focus on experimental mode, marketing
00:48:38 Aptera, Subaru experimental, support
00:50:55 driver monitoring
00:51:20 Mercedes no support, Tesla DM
00:52:10 openpilot active users, George's drive
00:53:08 lawsuit, comma body, sony ces sponsors
00:54:54 how nav works
00:55:35 Alex last words, scams, remote work
00:57:55 Harald about work in progress
01:01:15 Geohot voice directions, lane change
01:02:20 autoencoder
01:03:05 sentry mode, on device learning
01:05:10 end words
5
-
5
-
5
-
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
https://youtu.be/lnVQsJJFcdg?t=13715 Hiring entire stack for tiny corp join if you are interested | https://youtu.be/lnVQsJJFcdg?t=14190 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:00:55 Plugable BT-KEY3 and Galaxy Z Fold5
00:02:00 enabling ssh server on the phone
00:05:25 5G, eSIM
00:07:43 talking to Alex, minister of AI dubai, visas, Miami, immigration policy
00:12:50 vscode server, android
00:18:00 labs.perplexity.ai mixtral-8x7b-instruct
00:20:18 bionic libstdc++.so, node, vscode server android termux
00:27:50 Donald Trump - Make America Great Again (Country Song)
00:29:40 Vivek Ramaswamy smart
00:30:25 vscode in a browser
00:32:25 reach on twitter, Elon did more for anti woke cultural movement
00:35:00 tmux can't copy paste
00:43:10 python plugin tab completion
00:47:35 IOCTL_KGSL_DEVICE_WAITTIMESTAMP_CTXTID
00:50:20 growth, accelerationism, decels, You may not be interested in politics, but politics is interested in you.
00:53:13 replace opencl with driver that use IOCTLs, hook native function in python
01:04:55 python frida
01:06:05 IOCTL syscall attach to our process
01:08:05 perplexity hallucinating
01:10:30 bionic by google, libc, android
01:18:05 apple, dumb comments
01:19:05 python most beautiful programming language in the world, C
01:21:59 chillin in my rover american empire is over, uni freshman
01:24:20 illegal instruction, arm64 breakpoint instruction
01:29:20 Ennaria - Monstarrr, internet in the 90's
01:33:00 signal devide by zero signal, arm64 disassembler online
01:39:50 aarch64 svc, ctypes callback
01:41:05 j_blow raiding the stream, hooking iocal from python
01:47:30 got plt python, dlsym
01:51:40 symbols dylib, libdl github
02:04:04 dl_iterate_phdr
02:10:15 python dl_phdr_info
02:11:38 beautiful libraries
02:15:15 hooking ioctl universal, best toolset for debugging gpus
02:22:05 python trampoline
02:29:50 libc strstr, libc memmem
02:36:00 restype = ctypes.c_void_p
02:48:25 callback
03:07:25 corrupting the wrong got, hotpatching a function
03:10:10 aarch64 assembler, aarch74 abi
03:20:35 getting callback
03:29:55 segmentation fault, illegal instruction, python arm flush icache
03:32:45 python call ioctl syscall
03:39:40 IOCTL_KGSL_GPUOBJ_SYNC, iowr
03:44:20 libc cache
03:44:45 anyone paying attention? instruction cache?
03:48:48 chat comment trigger, dancing monkey
03:50:10 Logitech C920, python, calling from libc
03:56:30 parsing the structs, ioctl command encoding
03:59:40 z fold 5 kernel source
04:02:00 qualcomm 8550 reference kernel
04:04:00 file in user space in android that is gpu driver
04:04:50 gpuctypes
04:15:15 ctypes pretty print struct
04:18:30 alloc(16) being called, page aligned
04:19:20 so much of modern software development is completely wrong, leaky abstractions
04:21:45 bar for writing software, do you understand what you are doing?
04:23:25 javascript, softmax, jit, javascript engine
04:28:40 communicating in english and communicating in programing are the same thing
04:29:00 syscall, companies need to stop tolerating people who don't understand
04:31:20 american salaries, skill, motivation
04:33:00 people below and above API line
04:34:15 choices, bad, good habits, productivity
04:37:05 dream society, wireheaded
04:37:50 stream overview
04:38:40 making money at expense of other people
04:39:05 frog infestation on island
04:40:30 io_uring
04:42:05 same chip for quest 3 and samsung phone
04:42:50 make progress yourself before getting investors
04:45:30 softskills, social cues
04:46:50 integrating AI in everything
04:48:05 HR, human resources background
04:49:45 american empire, infrastructure, ivy league schools
04:53:50 trump, right wing, left wing
04:55:30 progress, faster plane is better
04:57:40 money, scam, value, perception is not reality
05:01:30 where are the viewers from
05:06:15 france gdp per capita
05:07:25 france vs singapore, china
05:08:50 uk vs us
05:10:45 foreign exchange reserves 100 years
05:13:15 WTF Happened In 1971?
05:14:30 raiding twitch.tv/lana_lux
05:16:15 be better, you canchange your habits and your life philosophy
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
Writeup and show notes: https://www.latent.space/p/geohot
Grateful to Shawn Wang (of Latent Space) for allowing us to upload this video. Follow, Subscribe to Latent Space:
- https://www.latent.space
- https://youtube.com/@LatentSpace-podcast
- https://twitter.com/latentspacepod
- https://twitter.com/swyx (Shawn Wang)
- https://twitter.com/fanahova (Alessio Fanelli)
Source: https://youtu.be/K5iDUZPx60E
Chapters:
00:00:00 intro
00:00:55 open pilot, devkit, gatekeeping
00:01:35 the hero's journey, what was the portal?
00:02:15 sam altman congress, ml compute, nvidia, qualcomm
00:03:24 CISC, Arm, RISC-V
00:04:15 good AMD stack, Google TPU, Google wrote their ML framework
00:06:05 turing completeness, re-order buffer, speculative execution, branch predictions, halting problem
00:07:40 clockless, analog computing, changing cache hierarchy, removing branch predictions, warp schedulers
00:08:20 turing completeness is easy, what is CUDA, TPU, systolic arrays
00:10:05 systolic arrays visualization, TPU closed source, AWS Trainium
00:11:25 tinygrad, lines of code, pytorch, tensorflow code
00:12:34 tinygrad developer experience, ONNX, ONNX runtime, compliance tests, core ML
00:13:25 unnecessary memory operations, pytorch lightning, why pytorch relu a class?
00:16:05 laziness, eager, graph compute model
00:17:30 competing against smart people, less complexity
00:18:15 how does fusing work, lazy.py
00:19:10 GRAPH=1, DEBUG=2, John Carmack
00:21:05 tinygrad right now uncompetitive on nvidia, x86, slower
00:21:32 tinygrad competitive on qualcomm gpu's
00:22:25 tensor core support, AMD bugs, opencl, ml perf
00:23:45 AMD kernel driver, ml framework, user space runtime, cuda_ioctl_sniffer
00:24:30 kernel panic, intel GPUs, AMD Lisa Su, AMD communication people
00:26:35 open source culture, nvidia nickel, nvidia P2P, cuda memcpy
00:28:00 building in public, contributing bug fixes to open source
00:28:32 ggml, M1 pytorch, AMD pytorch
00:30:00 test_ops.py, CI, good tests, mojo, pytorch compatibility
00:31:35 replicating python hard
00:32:08 tiny box red, limited by GPUs, luxury ai computers, fp16 llama
00:33:22 ggml quantization, compressing the weights, memory bandwidth
00:35:32 int8 support, weights in int8, fp16 to int8 to fp16
00:37:45 tiny box challenges, 6 GPUs, blowers or watercooling, pcie 4 extenders, pci redrivers
00:39:10 silent tiny box, 45-50 dB, one outlet of power, limit the power on GPU
00:40:30 AI hub for the home, personal computer cluster, pci bandwidth
00:41:50 training limit on tiny box, 7B, interconnect bandwidth
00:43:05 training longer, making bigger model, training, inference on cloud
00:44:30 on device training, fine-tuning
00:45:25 mining FLOPCoin, how to tell crypto is a scam
00:45:45 how to ensure your data is correct, tiny net
00:46:25 federated training, distributed training
00:47:42 enterprise use, flops per dollar, flops per watt, one person of compute as 20 PFLOPS
00:49:32 one Tampa of compute, GPT 4 mixture model, 16 inferences
00:50:40 secretive companies, hiding something that is not that cool
00:51:10 better training, batch norm, flash attention
00:52:50 Rich Sutton The Bitter Lesson, OpenAI computers you all you need
00:53:40 Hutter Prize, RNN, MDL, what is OpenAI getting wrong? vs working at facebook
00:55:38 how to hire people when computer can do everything
00:56:20 can a model do a simple pull request
00:57:05 unimpressed language models, subpar rap lyrics generation
00:58:04 10 LLMs in a room to discuss the answer, program generation
00:58:45 tiny corp is a remote company, 1000 job applications, programming challenges
00:59:30 tiny grad pull requests, stipend
01:00:45 coding is tool complete (above API line), driving is not tool complete (under API line)
01:01:40 stable diffusion replacing artists, tools getting better
01:02:30 full time at tiny corp, working on bounties, proposing bounties
01:03:16 separation in company
01:04:05 comma body, software problem
01:05:40 large YOLOs, segment anything, talking to LLMs, latency
01:06:12 LLaMA vs ChatGPT
01:06:40 no distinction between computer vision and language
01:07:30 company after tiny corp, AI girlfriend, merging with a machine
01:08:50 brain upload, George's brain already on youtube
01:09:30 living forever, how many weights a human has
01:11:05 the goddess of everything else, AI is not really going to kill us
01:11:35 AI alignment problem, the complexity will continue, paperclipers do not exist
01:12:25 grateful for AI, don't need hard math to understand AI, ML
01:13:54 John Carmack six insights, Elon's methodology
01:14:25 accessibility, tiny corp building computers, luck
01:15:25 why transformers work, semi weight sharing, qualcomm
01:16:25 the weights can change dynamically based on context
01:17:10 attention is all you need
01:17:50 Elon fundamental science physics, George fundamental information theory
01:18:55 e/acc, only the left takes ideology seriously
01:19:45 effective accelerationism, Mark Andreessen
01:20:25 why avatar 2 bad, Jake Sully
01:21:35 ChatGPT level pull request
01:22:00 impact of chat bots, spam bots
01:22:40 go try tinygrad
01:22:55 building chips, building silicone mines, self reproducing robot
All material displayed in this video belongs to their respectable owners. We uploaded this video in good faith to share the work and progress of George Hotz, tiny corp and comma.ai.
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
5
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
Watch youtube.com/watch?v=iFUmWho7fBE - The AI Alignment Debate: Can We Develop Truly Beneficial AI? | Everyone should read how to ask questions the smart way --> www.catb.org/esr/faqs/smart-questions.html
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | tinygrad.org | github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/realGeorgeHotz & twitter.com/__tinygrad__ to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 muted intro
00:00:55 un-muted, touch grass freestyle
00:02:45 conductors, new chair
00:04:20 official tinygrad meeting in tinygrad discord
00:04:45 PS5, 33 years old man, video games
00:05:50 installing sentencepiece, llvmlite, slow load
00:12:10 bitcast op, Alex
00:16:15 different debug levels
00:18:00 thank you to bedeks
00:18:35 ops_gpu not right
00:20:45 code getting unreadable
00:21:55 raw buffer, no copy, it's just a view
00:22:25 on gpu cl buffer and change dtype
00:23:30 this is becoming insane
00:23:45 collapsing under complexity, needs a refactor
00:27:55 bitcast just a flag on cast
00:30:40 making tuple
00:36:00 numpy bitcast
00:42:20 assertionerror
00:46:25 test_shape_change_bitcast
00:47:20 who wrote this crap haha
00:48:00 feeling good about this
00:48:25 test_dtype works with CPU and PyTorch
00:53:40 all this casts a mistake
00:53:50 features make things hard to read
00:54:10 dtype mismatch on realize
00:57:50 useful tracebacks
01:00:30 tuple no attribute name, assert instance dtype
01:02:40 I wrote unreadable code
01:07:10 how is numpy getting the type
01:11:50 I wrote hacks, disktype
01:17:00 we need tests for this, bitcast
01:18:50 Nelly rob the jewelry store and a grill
01:20:20 I'm a 20 year old girl who is looking for a good lay in Palm Coast
01:23:50 chat with 70B good
01:24:30 socialism
01:25:30 getting inappropriate
01:27:00 how to prevent regulatory capture
01:31:30 Llama 2 very inapropriate
01:32:30 how many genders are there
01:33:00 I want to be software
01:34:00 barbie post woke
01:34:40 discord at 5pm welcome to join the meeting
01:35:50 tiny voice channel
01:37:10 tinygrad pull requests
01:40:30 bounties
01:42:25 deceptively easy/hard $400
01:43:20 CIFAR improvement before selling tinybox
01:43:10 tinybox specs vs A100
01:44:50 locking SVC bounty
01:47:00 training multi gpu pull
01:48:00 rewriting AMD backend
01:48:25 gem stuff integration, single GPU faster
01:48:35 something like cuda cache thing
01:50:25 whisper pull
01:55:25 tiny3 back up, tiny5 mechanical box
01:58:50 tracking via pull issues
01:59:35 implementation is all it matters
01:59:45 engineering is all about tradeoffs
02:00:40 meetings not the right place for generic questions
02:01:20 hate meetings every day at 5
02:01:35 tinycorp, hate managing people
02:02:00 don't ask me questions
02:02:10 how to ask questions the smart way
02:02:50 tyranny of remote work
02:03:10 better and better metric that track people
02:03:30 what is a company in the future
02:03:40 async strategy
02:05:00 monday meetings
02:06:20 designer at comma
02:06:35 dealing with people pisses me off, dealing with hardware make me happy
02:07:35 why did I start another company
02:08:25 tiny boxes will print cash
02:08:55 end goal is building a chip
02:10:50 tinygrad has the right ideas
02:12:30 getting better, no regression
02:12:50 rockchip tinygrad backend
02:13:30 windows banned from tinygrad, whinygrad for windows
02:14:00 companies becomes own by bankers
02:14:10 qualcom board of directors
02:15:00 nvidia board of directors
02:15:25 meta board of directors
02:16:30 getting cancelled
02:17:10 defcon vaccine cards
02:18:40 cyber
02:20:40 tiny grad operating system for robots
02:21:05 every robot should use RK3588
02:22:10 why do I even stream
02:22:20 Connor Leahy debate
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
https://tenstorrent.com/cards/ (Grayskull™ e150 - Add to cart) - $799 | https://tenstorrent.com/setup/ | https://github.com/orgs/tenstorrent/repositories | https://github.com/geohot/tt-twitch
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:00:40 tenstorrent e150
00:02:40 AT&T, AMD, Qualcomm
00:04:07 Jim Keller, Tenstorrent, Open source
00:07:13 nvidia orin devkit, TRUFFLE–1
00:10:10 tinybox specs, nvidia 5090 rumor
00:11:05 pushing AMD, embracing open source
00:12:20 unboxing e150
00:13:55 interactions with AMD, lying
00:15:13 tenstorrent mega blower fan
00:23:10 fan loudness check
00:23:45 fanless, thermal protection
00:28:42 tenstorrent.com/setup, NDA
00:29:10 Alex
00:30:22 hardware installation
00:34:10 banjo.canonical.com
00:35:10 firmware installing broken
00:39:55 work life balance, Alex
00:41:15 tt-smi, temperature
00:43:07 tenstorrent bounties
00:47:15 smoke test
00:49:45 buda
00:53:40 cloud is a scam, selling cards
00:55:20 import error
00:57:50 this is why tinygrad is going win
00:58:55 nix
00:59:25 docker, no dependencies
01:02:20 nvidia value, cuda
01:02:55 reproducible builds, getting rid of complexity, it just works
01:05:30 import error
01:06:40 the complexity
01:06:54 redis great software
01:08:22 missing jemalloc
01:11:00 key error BACKEND_ARCH_NAME
01:12:15 mysql, sqlite
01:14:40 docker encouragees
01:15:15 exploring tt-metal
01:17:45 PEP8, bad programmers, details
01:18:25 downloading multiple gigabytes
01:20:35 translating github comments
01:25:00 tt-buda
01:26:04 stop writing code like this
01:32:30 if you thought AMD is bad try tenstorrent card
01:32:55 tt-smi reset options, reset file, reset json
01:35:45 this is the opposite of how you do complexity
01:36:40 ordering food, folding phone
01:42:05 tt-metal
01:46:00 start saying what it is
01:48:30 HSA fail, making everything generic
01:51:20 tt_lib, tt_eager
01:52:00 py-buda, luwen
02:04:00 installing egg
02:10:00 tensor bob
02:13:20 devin
02:15:30 reading docs
02:17:40 pyrsistent
02:19:40 Jim stop, nobody wants 50% PyTorch
02:19:55 be proud of what the chip is and expose the chip for what it is
02:21:19 ttlib docs, chip info
02:23:10 no real company is going to buy this card, dojo, inference price per dollar
02:23:50 tinygrad in for 10 years
02:24:05 PyTorch trigger
02:31:10 what this is?
02:35:45 grayskull e150 number of cores
02:37:20 bad documentation
02:42:44 tenstorrent staff in the stream
03:00:25 not trusting C
03:05:02 Alex closing the door
03:11:50 hate coding in C
03:17:00 bad code trigger, memcpy
03:27:17 SFPU
03:34:15 doing math, 13+7=21
03:43:30 kernel APIs
03:47:50 this better than GPUs
04:02:25 opencl api bad concept
04:06:30 food
04:12:20 life magazine 1970
04:16:10 Alex, Groq $20k card, Groq open source
04:20:05 brain backprop, small work units
04:20:39 nvidia full fabric memory
04:22:26 tenstorrent website bug
04:24:07 keeping up with state of the art ML
04:25:10 AMD price advantage, mi300x
04:28:20 graph compiler should be generic
04:28:35 port tinygrad to tenstorrent
04:33:40 the bitter lesson
04:38:50 break
04:47:12 worry about correctness
04:48:25 thumbs up ai
04:48:50 tinygrad code for metal
05:09:00 host_api
05:11:00 Karen ai, devin, VCs, deep state
05:15:17 unitree humanoid, money printer
05:19:04 1971, gold
05:25:50 clang 14 install, VC investment scam
05:42:50 dispatcher kernel, 1971, zebu ep1
06:03:00 fake queuing, scheduling is the key, big scale
06:11:30 Groq demo, tt hardware, wormhole
06:29:25 software, endorsing tenstorrent, comma body
06:49:50 X9000 G2 Groq demo challenge
4
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | A half noob lesson, introduction to tinygrad. Go read the codebase -> github.com/geohot/tinygrad | Support George by subscribing on twitch.tv/subs/georgehotz | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
From 02:38:20 on George is playing the guitar. George is so good on the guitar that is triggering youtube copyright claims.
Chapters:
00:00:00 intro
00:00:41 drug test, drinks, to shower or not to
00:04:40 sub only chat, tinygrad lazy
00:05:10 tour of tinygrad, Tensor
00:06:20 graph
00:07:30 print underlying buffer if it's realized
00:09:20 lazy mode, two armies of haters
00:10:20 multiplying two matrices and graph
00:11:00 GPU support, dashes - not a contiguous tensor
00:11:35 tinygrad codebase, three backends CPU, GPU and Torch
00:11:50 lazy mode
00:13:00 tinygrad is a tensor library, tensor is a list of numbers
00:14:20 benefit of laziness
00:16:10 stream lag, dirty hoodie
00:16:48 ImageNet inference, stream lag, GPU=1
00:19:47 tinygrad definition, 992 lines
00:20:25 tensor class, karpathy, micrograd,
00:23:15 backends, openpilot model
00:24:05 automatic differentiation engine, bad bug
00:26:05 let's fix that bug
00:29:15 hlops, tests
00:31:50 fun thing in tinygrad
00:32:25 pad2d, matmul
00:35:40 transpose, flatten, sum
00:36:40 softmax, dropout, pools, conv, unary
00:37:05 binary ops
00:38:20 tinygrad is one of the best things I have ever written
00:39:25 div, short break
00:40:30 neural networks ops
00:41:45 linear
00:43:10 sequential, reduce
00:44:40 an instantiation of the function in the context
00:45:45 mlops, cleanup, unary ops
00:47:50 good tests, more cleanup
00:53:20 the derivative of convolutions as convolutions
00:53:50 llops - unary_op, binary_op, movement_op, procesing_op, reduce_op
00:54:20 working on the code
01:00:00 how does it run on the device
01:02:10 numpy, GPU backend, Torch backend
01:02:50 train efficientnet and graph
01:05:00 custom accelerator support, shapetracker
01:05:45 optimizers, BatchNorm, Conv2d
01:06:40 cherry hardware, data movement is expensive
01:08:45 rockchip neural network accelerator
01:09:20 replacing numpy
01:12:45 improving performance by removing GPU cache
01:14:00 equivalent of Numpy.argsort() python
01:16:15 codebase understanding, tests
01:18:00 chain the autograd
01:20:00 nobody liked the tinygrad tour
01:22:15 new readme
01:23:00 tinyvoice, tinygrad will be faster then pytorch
01:24:15 GPU support
01:25:50 simple and memory efficient
01:27:00 tinygrad support
01:28:00 this should be in the readme, everything compiled on the fly
01:28:40 tinygrad++, refactor feels correct
01:30:10 convolution
01:31:30 bugs to fix
01:37:55 benchmarks
01:39:10 Tuple index out of rage, cannot resolve name lazyop, more errors
02:11:40 commit typechecks
02:21:00 why I don't stream tinygrad
02:22:35 torch question, issue
02:26:15 timeout for bad question, more bad questions
02:28:40 cool pull request from smurfd0
02:29:50 sub so you don't get a timeout
02:30:10 no j_blow stream to raid
02:31:20 freedom of speech on twitch, non subs vs subs
02:34:30 giving money to amazon, patreon
02:35:45 prime sub
02:36:25 tinygrad is work of art, whiteboard
02:38:20 tuning the guitar
02:40:00 drake - best i ever had
02:41:00 eagles - hotel california
02:44:00 the front bottoms - twin size mattress
02:46:30 goo goo dolls - iris
02:48:08 american football - never meant
02:48:45 john denver - take me home, country roads
02:51:45 dragonforce - through the fire and flames
02:52:05 the hotelier - an ode to the nite ratz club
02:52:45 led zeppelin - stairway to heaven
02:57:25 thank you for watching
4
-
4
-
4
-
4
-
4
-
4
-
Buy 1 year commacare warranty -> https://comma.ai/shop/commacare | Check if your car is supported -> comma.ai/vehicles and buy -> comma.ai/shop/comma-three from $1499 | blog.comma.ai/dumb-questions | What is comma.ai/openpilot | Join the comma team comma.ai/jobs | How to setup your car comma.ai/setup | Follow the official youtube.com/commaai for more livestreams | Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop.
Chapters:
00:00:00 intro
00:00:13 drink sponsor
00:00:35 office poster
00:00:45 plant automation
00:01:08 comma body
00:02:53 banning user
00:03:35 comma one
00:03:58 COMMA_CON 2023
00:05:33 tickets
00:06:25 new product
00:07:45 visiting Alex
00:08:11 if you want to work at comma
00:08:50 comma three boards
00:09:00 factory machines
00:09:25 Alex
00:10:14 comma three extended warranty
00:11:25 comma care on comma.ai
00:11:50 comma care fine print
00:12:35 you can't make people happy
00:13:00 COMMA_CON for product announcement
00:13:20 $400 commacare extended warranty
00:14:05 raising price for including car harness
00:14:20 comma two failed
00:15:02 no fault warranty
00:15:20 comma three sales
00:16:02 COMMA_CON
00:16:25 profitability
00:16:50 parts changes
00:17:15 additional price drop, margin
00:17:30 software engineer salary at comma
00:18:08 employee buying house of stock
00:18:15 no opportunity, no remote work
00:18:40 device falls of the glass
00:18:58 qualified installers for comma
00:19:25 driver monitoring
00:19:45 cabana comma
00:19:55 comma three life span
00:20:25 driver monitoring training
00:20:40 chinese ev support
00:20:58 did not leave the comma president
00:21:20 marketing intern
00:21:40 navigate on openpilot
00:22:05 heat and comma three
00:22:40 comma con stream
00:22:50 design role in comma
00:23:25 comma three heat
00:24:00 latest build vs tesla fsd
00:24:45 outside of US openpilot
00:25:20 can fd hardware
00:25:39 comma security
00:25:50 new release
00:26:18 longitudinal strategy
00:26:40 following distance
00:27:00 devkit
00:27:55 comma care
00:28:35 parking in sun
00:29:10 selling out, 7 more years to solve self driving cars
00:29:25 99% to get you to destination
00:29:40 opensource vs closed software
00:30:05 boxing match vs Elon
00:31:10 comma three compute
00:31:46 EU roads, india roads
00:32:30 VIP tickets sold out
00:33:55 comma three in rain
00:34:14 what does George do all day
00:34:35 camera see better in rain
00:35:20 simulator, slam
00:36:10 training should not be open source
00:36:35 weights open, training stack $1 milion computer
00:37:30 confidence in open pilot
00:40:30 designer job at comma
00:41:15 comma profitability
00:42:31 Lisa Su is cool, corporate partnership
00:43:05 VIP tickets back
00:43:35 manufacturing warranty
00:44:05 fuses blown, ecu camera broke to 1 person
00:45:10 comma warranty
00:45:50 tinygrad in openpilot
00:46:45 fsd beta
00:47:15 George's car not supported by openpilot
00:47:45 rolls royce ghost, dealership
00:48:05 designer job at comma
00:49:20 hardware team hiring process
00:49:42 comma body
00:49:52 python openpilot code base
00:50:50 speed humps
00:51:10 fully automatic lane changes in supported cars
00:51:40 experimental mode slow
00:52:20 comma body coffee shop
00:53:45 no motorbikes support, cars are safe
00:54:55 head of production job
00:55:18 uploading clips to comma
00:55:39 new product is comma care
00:57:18 not sending product to youtube's
00:58:00 George's salary 0 right now
00:58:40 comma shares
00:59:20 never going public
01:00:00 how hard to fingerprint the car, best toyota, tesla not really supported
01:01:15 gm canceled the bolt, stopping ai, comma care, OEM reach out
01:03:35 aptera, safety, banned for comma network, don't scam comma
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
Watch youtube.com/watch?v=6yQEA18C-XI - George Hotz vs Eliezer Yudkowsky AI Safety Debate | Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/ | Sources for this stream:
- https://twitter.com/realGeorgeHotz/status/1690831164431585280
- https://wtfhappenedin1971.com/
- https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
- http://prize.hutter1.net/hfaq.htm
- https://arxiv.org/pdf/2303.08774.pdf
- https://geohot.github.io/blog/jekyll/update/2023/08/08/a-really-big-computer.html
- https://arxiv.org/pdf/2005.14165.pdf
- https://arxiv.org/pdf/2203.15556.pdf
- https://en.wikipedia.org/wiki/Landauer%27s_principle
- https://en.wikipedia.org/wiki/Why_there_is_anything_at_all
- https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_computer_science
- https://en.wikipedia.org/wiki/Richard_S._Sutton#Career
Chapters:
00:00:00 muted intro
00:01:15 un-muted intro
00:01:38 debating Eliezer Yudkowsky, bad people
00:02:35 Sam Altman
00:04:35 Thermodynamics is to Energy as <???> is to Intelligence
00:05:00 where did smartest people go in 1950
00:05:40 debating Sam Altman, RLHF, truth
00:07:55 universities is where the smartest people go in 1950
00:08:25 where do the smartest people go in 2023
00:08:54 hedge funds, FAANG, startups, power over nature
00:10:00 less theory, more applied, hard doomers (EA), thermodynamic god
00:11:00 why this split exist, we have no theory, arguments for Yudkowsky
00:11:50 intelligence criticality are possible?
00:12:00 1800, small stick boil water calculation
00:15:20 asking ChatGPT grams of wood to boil a gallon of water
00:16:25 how much intelligence do I need to prove fermat's last theorem
00:17:35 thermodynamics, classical physics
00:18:18 no unit for intelligence, unit of energy per second, J/s, W, Horsepower
00:19:35 person-years, bad universities, Carnegie Mellon University
00:20:50 2012, 2014 universities destroyed
00:22:15 getting a drink
00:22:50 wtf happened in 1971
00:24:00 information theory, entropy, thermodynamics for information
00:25:18 real university reach out to me
00:25:40 collage, scam, Scott Aaronson, Robin Hanson
00:26:15 administrative class, professional managerial class
00:26:40 hutter prize, compression is intelligence
00:27:40 gpt-4 technical report
00:28:25 don't hate Open AI, overhyped, person-years
00:29:55 using GPT 4 for field of science, best tutor to ever exist
00:31:00 plot gpt-2 gpt-3 bits per characters on enwik8
00:31:45 Sergey Brin is back at Google, Sundar Pichai
00:32:30 petaflops-days, 1 person = 20 PFLOPS
00:33:40 Paul Christiano Bankless
00:34:20 losing to Connor Leahy = thinking about AI alignment
00:35:39 Joscha Bach latest Lex Fridman
00:35:55 1 person years = 7300 petaflops-days
00:37:20 chinchilla deepmind, bits per character, token
00:38:38 Elon, Twitter/X login pop-up, destroy so much long term value in a company
00:39:40 what unit is the loss in for LLMs
00:40:05 nat (unit), shannon (unit), bits, 1 george of intelligence
00:41:10 beautiful elu, minimal energy requirement
00:42:20 landauer's principle, can reversible computation implement intelligence
00:43:30 why AI doesn't destroy the world, God is real
00:44:15 we are in the video game, highly rated game, we don't die in some stupid way
00:46:10 never use the word AI safety, instead use AI alignment
00:46:49 feminism legal equality of man and women
00:47:05 alignment, good loss functions that do what you want
00:47:32 defcon, Why there is anything at all
00:48:25 there is no difference between agents and tools, Robin Hanson
00:49:00 the tail wags the dog, car is the dominant species on earth
00:51:35 we have no real theory to answer how much intelligence we need for...
00:52:08 list of unsolved problems in computer science
00:52:50 people think George is not serious, care about the search of truth
00:54:15 academic system that is completely destroyed
00:54:45 information based approach is good
00:55:15 university not good for finding a job, education industrial complex
00:56:40 Eliezer Yudkowsky did not attend high school or college
00:58:25 no theory of intelligence, bounds of complexity theory
00:59:35 where is Marcus Hutter, deep mind, Jürgen Schmidhuber, Richard S. Sutton
01:00:40 Sutton became a Canadian citizen in 2015 and renounced his US citizenship in 2017
01:01:09 defcon vaccine card check, defcon crazy people, public apology, san francisco
01:02:35 real science, real questions, can't do science anymore, grant process
01:03:40 professional managerial class + AI = bad
01:04:05 Alex tired of the rants, preparing the debate
01:04:28 open source AI are bad, only for trusted people
01:05:20 if science about intelligence is done somewhere link it in the comments
01:06:05 lot's of questions, no answers, physics off the rails
01:06:30 what happened that stop science
01:06:45 we don't live in a world where standing still is safe
01:07:00 what happened to the internet, consolidate power
01:07:55 thank you for watching, we do not die
01:08:14 are you the problem? if you are think about what can you do to be less of a problem
01:08:45 tinygrad, tinycorp, comma, comma hackathon
01:08:55 deleting twitter, living under the bridge and finding truth
4
-
4
-
4
-
4
-
4
-
Thank you to the comma team for this amazing event and sharing the livestream and high-quality videos. All credits to them. Follow their official youtube.com/commaai for more livestreams.
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | openpilot.comma.ai is the best ADAS system in the world | If you have skill apply on -> comma.ai/jobs | Join the community on github.com/commaai/openpilot & discord.comma.ai | Check if your car is supported -> comma.ai/vehicles | How to setup your car comma.ai/setup | Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop.
Chapters:
00:00 black mirror episode with ratings
00:45 three stories, past, future, present
00:55 the past, cruise automation
01:13 cruise RP-1 highway autopilot
01:35 giving up on technical challenge
01:45 too many cars to support
02:25 how to handling failures
03:10 fingerprinting
03:45 comma was not supposed to exist
04:30 what is worth doing in the world
05:25 6 generations of hardware
06:15 keep iterating the hardware
06:40 being real, honesty, communication people
07:35 shipping keeps you honest
07:45 stated preference, revealed preference
08:19 iteration
08:40 humility, egohot, big ego
09:50 so why haven't we won?
10:00 things take a long time, even accounting for things taking a long time
10:38 no exit, the system will get better
11:04 climbing out of a well
11:53 it turned out these companies were scams, bitconnect and luna scam
13:10 marginal utility - marginal cost
13:25 value destruction, linux fastest growing platform on Azure
14:05 markets - rationality, you just have to survive
14:20 lifetime revenue
15:21 $22.2M on $18.1M raised
15:33 twitter followers
15:50 future followers
16:00 employee retention
16:18 come work to comma ai all work by the team not George
16:33 keeping idiots away from comma
16:50 bar very high, you get to ship things that get better in the world for the better place
17:33 business world
17:45 the future
17:59 computing speeds vs human brain computation
18:45 forward looking statement
18:59 comma con 9 in year 2037
19:15 44 stores in malls, top 10 biggest consumer electronics companies, without selling out
20:07 hard to go anywhere without finding a comma, cars, bodies, puppies
20:35 comma body
20:45 comma 7 for $2500
21:01 3rd generation comma ASIC, 50000 TOPS
21:12 first comma device to boot into tinygrad
22:25 trying to build people
22:50 trolling Elon, our system is going to be so end to end even our driving monitoring will be end to end
23:10 comma 7, how many silicon people work at comma
23:50 your comma 7 is yours, models we train in the cloud are foundation models
24:08 thank you to meta for LLaMA, open source AI
24:32 offer pretrained policy models, custom software
24:56 comma 7 is like child
25:06 will learn behaviors you teach, so teach it well, pick up skills as quickly as smart human
25:16 comma 8 will be running the company
25:26 comma three human head
25:55 intelligence
26:20 humans are so good at moving, driving
26:57 the present COMMA_CON 2
27:22 the mall store soon
27:45 sitting down for dramatic effect
28:12 failure rates for iPhone
28:50 comma two failure rate
29:25 comma three failure rates
30:43 dropping the price
31:15 comma 3x
31:36 much better injected molded case, cameras on board
32:12 3X specs, same hardware and software as comma three
32:49 upgraded cameras, 140 dB HDR
33:14 128GB of storage
33:28 CAN-FD integrated
33:45 still OLED better supplier, onboard road cameras
34:40 stereo in-case sound
35:02 upgraded power architecture
35:22 about half the components, more reliable
35:32 3X board front, no more GPS connector, better antennas
36:30 3X board back
37:01 battery slides in and out, chokes for CAN-FD, e-sim
37:30 cameras are right on the boards, failure rate of this device need to be 1 or 2 %
38:05 comma three vs comma 3x board complexity
38:25 no more NVMe, just on-board flash, QR codes on the board for tracking
38:50 is it a devkit? to set expectations
39:24 it's time to call it just a comma 3X, mass market
40:15 comma 3X just $1250
40:30 no more questions that is why it's just one edition
41:29 it's just a comma three but better
41:48 order now, ships in 1-8 weeks
42:25 comma 3X in EV6, no more red panda, no more two harnesses
42:44 when did the iPhone get good
43:24 the comma team built this not George
44:05 value in running language models on the device
44:50 marketing guy, testing on commercial vehicles
46:13 marketing, the story of openpilot, openpilot ADAS
46:55 buy now, 30-day money back guarantee
47:20 comma 3X always compatible with cars
47:56 port a new model to comma 3X
48:50 mass market, liability air gap with installing custom software
49:45 level 2, you must pay attention at all times
50:50 large foundation model on device, expansion port
52:55 failure rate, should be built like a tank
54:17 saying no to things, best highway experience, first comma device for friends
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
4
-
Go find the bug 02:06:30 | Who is going to be the first to find what was the bug? 02:49:45 | 02:50:20 | https://adventofcode.com/2022 | https://pastebin.com/rx0Yj0H1 | Go buy comma three from $1999 https://comma.ai/shop | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Follow George on twitter.com/realGeorgeHotz & instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog | Support George by subscribing twitch.tv/subs/georgehotz
Chapters:
00:00:00 intro
00:01:33 advent of code in scala
00:02:20 mobile streaming setup
00:04:21 python 3.12 improvements
00:04:35 monitor size 30" at home
00:04:50 why at twitter
00:05:45 vscode scala debug
00:06:35 twitter interview, technical interviews
00:07:55 no build target could be found for the path
00:11:10 elfbar vape hardware, new identity
00:12:00 chatgpt overload
00:12:15 scala debug single file vscode
00:13:20 vscode metals
00:14:40 real tips to get better with using keyboard shortcuts
00:17:45 learn default keys, hate Ghidra default keys
00:18:35 scala build target single file
00:19:45 scala app
00:21:05 scala startup time slow
00:21:50 yesterday stream bad mood, meatballs
00:24:25 scala 3.2.1
00:26:50 scala bloop
00:27:55 java javahome
00:29:55 scala build target, sbt example
00:38:50 aoc day 5
00:41:45 scala slice
00:43:16 scala every 3rd element from list
00:45:00 scala via a launch json config
00:47:50 scala flatmap, scala list method
00:48:40 scala for loop
00:50:20 scala list comprehension
00:52:25 scala list yield
00:56:15 scala list until
00:57:05 scala array api
00:58:58 scala array update
01:02:00 aoc day 5 finished, scala nice
01:04:50 culture at comma, appreciation for beauty
01:05:20 beautiful code, no build tool detected
01:07:38 aoc day 6, appreciate aoc, jack dorsey
01:08:30 halting oracle
01:11:58 only classes can have declared but undefined members
01:13:35 scala array no duplicates
01:14:18 scala loop break
01:14:48 scala print break
01:15:08 scala return
01:15:47 scala find
01:19:04 aoc day 7
01:23:05 so easy in python
01:23:40 making class
01:23:55 scala class, directory
01:25:58 scala substring
01:26:30 scala elif
01:33:25 scala map mutable
01:35:20 object oriented stupid, class gone
01:38:30 scala copy list
01:45:00 yolo, answer wrong
01:52:25 some directory you never ls but understand it exists
01:53:20 adding to this directory
01:55:00 scala set
02:00:30 jonathan blow raid!
02:02:05 scala map getorelse, map get or set
02:02:40 what we are doing today
02:03:08 getorelseupdate scala
02:03:35 how nice is scala
02:05:30 functional programming in scala second edition
02:06:30 trying to submit the same answer, it's wrong
02:07:30 refactoring, substring worry
02:08:15 it will quit telling you after third attempt
02:09:55 directories with same name
02:12:35 it's not falling back to address to hash
02:16:51 slash special directory
02:18:25 bigger than maximum int
02:20:05 two directories with same name
02:22:00 sometimes there is a trick
02:24:18 scala int max size
02:24:59 going full object oriented
02:25:58 scala extends
02:26:20 scala class constructor
02:27:55 scala union type
02:33:40 coding like java programmers
02:35:55 value update s not a member of object file
02:37:10 beautiful object oriented code
02:37:35 scala class, scala abstract base class, trait
02:43:00 scala typeof
02:44:25 scala flatmap
02:47:40 base case, this ::
02:49:30 bigger number is right
02:49:45 what the hell was the bug?
02:50:20 excited to read the youtube comments about what was the bug
02:50:30 part 2 of aoc 7
02:55:50 scala sort
02:57:00 not the right answer
02:57:20 it want's the size not the name
02:57:45 get name not needed
02:58:04 thank you for watching the stream appreciate you all
02:58:20 san francisco
02:58:45 I want to see in youtube comments someone figure out what the bug was
02:59:10 nobody to raid
02:59:35 https://pastebin.com/rx0Yj0H1
03:01:05 every day we get a little bit better, key to success
03:01:15 climbing out of a hole, as long as every day you climb more feet up than you slide down
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
arxiv.org/pdf/2101.03961.pdf | news.ycombinator.com/item?id=38570537 | blog.research.google/2022/11/mixture-of-experts-with-expert-choice.html | together.ai/blog/stripedhyena-7b | github.com/tinygrad/gpuctypes
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 muted intro
00:01:12 un-muted intro
00:01:25 tiny keyboards, feud with oneplus
00:02:38 bought oneplus open via oneplus store
00:03:16 bricked phone, fastboot set active slot, qdl mode, qdl data not available
00:05:14 oneplus 10, 11, open no images to restore, not honoring return policy
00:06:50 oneplus destroyed their brand
00:07:47 chargeback, buy oneplus buy from amazon
00:08:20 comma return policy complete and true
00:09:10 chargeback sending the phone back, apple late delivery experience
00:11:40 help George process emotions, do not buy it from oneplus store
00:13:00 comma does not want you to have a bad time
00:13:27 sending it back
00:13:55 edl, unofficial qualcomm firehose
00:16:50 phone market, why no images
00:17:48 companies don't want you to own anything anymore
00:18:15 Apple is so consumer focused
00:18:35 jailbreaking is not a solution
00:19:05 comma.ai forks of the software and hardware
00:20:35 possible to make phones
00:21:28 don't buy via oneplus store and don't expect anything from oneplus people
00:21:38 buy on amazon, oppositional defiance, great regrets in life
00:23:05 how do we get oneplus to release the qdl files
00:23:40 suing someone different
00:24:55 2% of unreasonable people
00:25:18 George reads all the emails
00:25:33 reading youtube comments
00:25:50 spicy comments, oppressive powers and free market, apple safari
00:27:48 anarcho capitalism
00:28:27 interns pay $1400-$2000 week
00:28:40 trans people, J. K. Rowling, transhumanism
00:30:10 ssh to tinybox 8
00:30:20 mixtral.py
00:32:08 python mmap
00:33:35 only 6 GPUs
00:34:15 bigger font, glasses, astigmatism
00:35:10 torch_load dynamically
00:37:25 Switch Transformers: Scaling to Trillion Parameter Models
00:37:41 Hacker News Mistral "Mixtral" 8x7B 32k model
00:38:53 Mixture-of-Experts with Expert Choice Routing
00:41:30 it's how GPT4 works
00:42:00 StripedHyena-7B, mixtral-7b-8expert
00:44:05 MixtureFeedForward
00:58:25 AT&T feud, billy naming
01:01:35 underlying buffer is not writable
01:04:35 read only memoryview in python
01:05:40 ctypes const pointer
01:07:05 ctypes from buffer
01:08:20 asking Quentin
01:11:19 memoryview readonly
01:14:10 python get address from memoryview
01:15:33 using a read-only buffer with ctypes from_buffer
01:16:30 b partition in honor in oneplus phone
01:16:44 oneplus store worst customer experience
01:17:35 buy cybertruck, oneplus worst company over time
01:19:05 HIP=1 DEBUG=2, realize() struggle
01:24:50 output of a gate
01:29:21 how easy is tinygrad to use (multi GPU usage)
01:30:40 looking at mixtral implementation
01:34:20 you need to run all the experts
01:34:50 stupid questions about buying tinybox
01:36:20 emotional regulation issues, don't be evil, us government
01:38:20 only batch size 1
01:42:30 print top, norm
01:45:20 /opt/rocm/bin/rocm-smi, htop
01:48:50 bug in AMD gpu
01:49:00 multiple $1k bounties in tinygrad dropped
01:51:00 libhsa-runtime64
01:53:20 rude to blame people until you are sure it's their fault
01:55:00 HIP runtime rewrite
01:57:25 first token 330
01:58:55 what is Mistral going to say
02:00:00 GPU usage explained, letting it speak
02:03:10 running on mac impossible, tinybox can run the model
02:04:00 pushing the code to github, refactor needed
02:06:30 letting non-subs talk for some questions
02:06:44 AMD bug
02:07:15 first real use of tinybox
02:07:40 why is mistral getting so much attention
02:08:00 how did we split them
02:08:10 quantization worst performance
02:08:20 Cmulator 50 gifted subs, thank you to subs
02:10:10 bug with errors with the micro engine scheduler
02:10:32 bug hip compilation inexplicably takes 5x longer then CL compilation because of different code path
02:10:40 open source, submiting pull requests and CI
02:10:55 tinygrad beating cuda, mmap nvme copy in to gpu bug
02:12:45 opinion on triton openai, delivering tinyboxes
02:13:50 best Mixtral content so far
02:17:55 cross gpu overhead, copies between gpu
02:22:45 working on synchronize things across gpus
02:23:00 GO, C speed, fixing the algorithms to speed up
02:23:55 non-sub asking for moderator, tinygrad CEO
02:26:18 gpuctypes tinygrad, GPT2 below 2ms per token
02:28:10 three new $1k bounties
02:30:30 never buy anything on oneplus store
02:31:40 Elon thoughts, famous, opinions, Taylor Swift, Sky Bri
02:34:08 small tasks needed in tinygrad
02:34:30 can't use company money to buy conference tickets, networking
02:36:16 never buy via oneplus store, oneplus not open anymore, verified image programming
02:37:00 contribute to tinygrad, where to start
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
Anyone heard of https://hsafoundation.com/ ? 02:50:20
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
https://youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | https://youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:00:20 no warning, linkedin ban, child prodigy
00:02:25 torchrl
00:03:20 tinybox pre-order, AI computer, lambda labs
00:05:20 lambdalabs vs tinybox
00:12:20 7302p vs 7532 epyc
00:15:00 tinybox raspberry pi for ML
00:16:10 not buying Apple Vision Pro
00:16:45 fastdd github, Meta buying H100
00:18:10 twitch removing content warning, linkedin post
00:20:15 linkedin worst dating site
00:20:35 selling rolls royce, money
00:21:25 George not a good fit for twitter, culture war
00:23:00 Peter Thiel dinner party, e/acc
00:25:55 better processor and it got slower
00:27:35 drive faster than dev 0
00:30:00 boost frequency, perplexity
00:33:50 bios, ipmi, epyc boost, not boosting
00:44:40 btop, pcie 4 vs 5
00:46:25 direct democracy
00:47:40 boost speed
00:50:10 hip graph is not fast
00:52:20 ROCm 6.0, Llama-2-70b slow, single thread
00:52:55 single thread, multithread, multiprocess tinygrad
00:53:55 ggml, tinygrad long term goal, universal
00:55:00 event, block slow
00:56:40 GPU queue sync, multiprocess
00:58:35 writing your own GPU driver, userspace
00:58:54 AMD HIP, clone of CUDA
01:00:30 finding HIP graph code
01:02:50 spinlocks, multiprocessing, GPU driver
01:03:25 how do GPUs work?
01:06:00 prebuilding the queues, hip semaphores
01:09:20 rdna 3 instruction set
01:12:40 so much complexity, micro engine scheduler
01:14:40 Alex
01:19:55 reading the code to send packets
01:22:45 hate free stream
01:23:30 amd gpu scheduling
01:32:50 perplexity valuation, how to value a company
01:34:00 HSA queue
01:35:30 perplexity fast, GPT4 slow, anthropic
01:39:00 HSA level 0
01:41:30 HSA runtime book, anna's archive
01:45:30 no copyright infringement intended
01:46:20 AQL packets
01:52:55 piano
01:53:20 AMD is for people who likes to get twice as much GPUs for their money
01:54:50 tinygrad pay per token API
01:58:30 replacing HIP support with HSA support
02:00:20 Nvdia vs AMD datacenter, customer GPU architecture
02:02:30 secret good version of openpilot joke
02:03:40 HIP does not use DMA engine
02:05:00 bit blit
02:14:20 rocm-bandwidth-test
02:17:00 hca kmt api amd
02:17:55 Alex
02:20:08 the hidden song
02:23:30 going on a journey
02:27:55 real completion events
02:31:30 hsa example of kernel dispatch
02:32:00 cool that AMD is so open
02:35:40 just using HSA
02:36:25 HSA rabbit hole, hsa foundation
02:37:50 the chapel language with 0 github stars
02:39:40 HSA Programmer's Reference Manual
02:40:20 linkedin post
02:42:40 the weather people, if you could design a country, deep state
02:45:45 conservatism, progressivism quote
02:49:00 Alex
02:50:00 thinking from first principles, experiments hard
02:50:20 has anyone heard about HSA foundation
02:52:50 scientific computing people, OpenMP, OpenACC
02:55:20 AMD extensions
02:56:15 traveling salesman, 2^n algorithm, scientific computing funding
02:59:10 leslie greengard
02:59:40 deep learning revolution
03:01:00 tinygrad experiment, complexity dysfunction of governance
03:01:38 misunderstanding of how software is developed today
03:02:00 compression is intelligence
03:02:20 complexity management instead complexity reduction
03:02:40 spacex rocket landing controls genius
03:05:10 complex systems, twitter
03:06:10 software 0 cost to replication
03:07:40 twitter acquisition best political dollar ever spend
03:09:30 making the tinybox good
03:09:46 making money off OSS
03:10:10 pre-order tinyboxes
03:11:10 etched.com, tenstorrent.com
03:13:50 tenstorrent offering a card to George
03:14:15 respect to tenstorrent, intel tier
03:15:05 extropic.ai
03:16:40 science grants, fundamental research that needs to be done
03:17:40 bullish on perplexity
03:18:40 atomicsemi.com
03:19:30 ranking startups, tenstorrent open source
03:21:00 tinygrad factorization
03:23:30 hammer.lol, berkshirehathaway.com
03:24:40 stop using javascript
03:27:20 apple.com website, feross.org
03:30:10 lana_lux 5k viewers
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
02:03:50 tinygrad intro | 03:31:25 sending GPUs back to AMD | 03:31:54 it compiles but it does not do anything musical | 03:52:00 tinybox not responding |
rocm.docs.amd.com/projects/HIP/en/latest/user_guide/hip_porting_guide.html#threadfence-system
Threadfence_system makes all device memory writes, all writes to mapped host memory, and all writes to peer memory visible to CPU and other GPU devices. Some implementations can provide this behavior by flushing the GPU L2 cache. HIP/HIP-Clang does not provide this functionality. As a workaround, users can set the environment variable HSA_DISABLE_CACHE=1 to disable the GPU L2 cache. This will affect all accesses and for all kernels and so may have a performance impact.
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
https://youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | https://youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:02:40 HIP P2P copy slow
00:15:20 hipDeviceEnablePeerAccess fast
00:17:30 synchronization mechanism
00:21:50 dumb questions = banned
00:23:20 no idea if this is working
00:25:40 hip c++ 11 atomics
00:29:10 cuda atomic sync
00:31:20 global cache flush hip
00:34:20 glc dlc
00:42:00 no idea if access is correct
00:43:00 hip atomic and system example
00:48:10 deep dive in spinlocks
00:49:00 hip cache flush
00:52:10 global_atomic_and_b32
00:56:30 hipDeviceSetCacheConfig
00:58:10 threadfence system
01:03:00 block
01:04:40 D0, D1
01:07:00 events are slow because syscalls
01:10:40 answering smart questions
01:11:35 bad athomics understanding
01:11:45 x86 vs arm difference
01:12:20 HIP event API slow
01:12:50 tinybox vs 8xA100
01:13:40 radeon pro benefit 48GB RAM
01:14:20 H100 price, MI300
01:15:00 buy gaming PC in $1000 range
01:16:30 6x7900 XTX
01:16:45 nvidia software support
01:17:25 tinybox power efficiency
01:18:20 lamma2 70b unquantized on tinybox
01:19:00 tinybox V2 same case, GPU refresh
01:20:10 6 tinyboxes a day factory
01:20:35 nvidia blackwell release timeline
01:26:10 never shipping anything in your life
01:32:30 removing lines
01:40:30 test_multitensor
01:48:20 deeper problems, good abstractions
01:50:55 does not work and it's slow
01:55:00 Hello nobody is talking
02:01:30 event 4
02:03:15 Lana_Lux raid!
02:03:50 introduction to new people, tinygrad
02:04:15 beautiful_mnist.py
02:05:30 tinygrad operations
02:06:40 matrix multiplication in tinygrad
02:07:10 convolution
02:07:55 pytorch difference
02:08:25 dtypes code
02:08:45 the bitter lesson, search machine
02:09:30 DEBUG=2
02:10:30 convolution custom op
02:10:50 tinygrad long term vision
02:11:10 tiny corp business model
02:11:40 tinybox
02:12:00 getting AMD on MLPerf
02:12:15 synchronization between GPUs
02:12:30 international forwarding
02:13:20 tinybox HIP support, full 16 lane pcie 4 GPU multi GPU connectivity
02:14:40 ocp 3.0 slot
02:15:20 tinybox infiniband
02:16:25 direct AMD support
02:16:50 traning over internet
02:17:05 synchronous gradient descent
02:17:50 best chip nvidia H100
02:18:50 tinybox launch apps, tinygrad bounties
02:20:10 no secret software
02:20:45 tinygrad master under 5k lines of code
02:21:30 9x 140mm noctua fans
02:22:30 custom cables
02:22:50 demos on pc
02:24:30 line count
02:25:15 4090 price tracker
02:25:40 decision transformers
02:28:25 small font
02:31:15 python no C
02:36:50 zyn
02:40:30 semaphores for 6 GPUs
02:41:20 thread_sync_global
02:47:10 writing GPU program
02:51:20 DEBUG=2 synchronization points
02:55:00 tiny9
02:57:00 hip cache flush
03:00:40 hip coherent memory
03:03:30 amd_cocclr_copyBuffer
03:06:00 amdgcn sleep
03:11:40 hipEventRecord fix
03:22:55 recurring expenses and non recurring expenses
03:23:05 we do things because we can
03:26:30 hipSyncAndResolveStream
03:28:15 HSA_DISABLE_CACHE=1
03:31:25 sending GPUs back to AMD
03:31:54 it compiles but it does not do anything piano musical
03:32:20 no boost, no cache
03:33:10 whiteboard idea
03:36:00 memory-fence instructions, no HIP rebuild
03:38:50 cache vs no cache
03:42:30 whiteboard
03:44:25 hsa_disable_cache definition
03:48:30 HSA amd flush l2 cache
03:52:00 tinybox not responding
03:53:20 tinybox piano comeback music, HSA driver, fences
03:56:25 tinybox back
03:59:40 need to write GPU driver, threadfence_system
3
-
3
-
3
-
3
-
3
-
3
-
3
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | tiny corp has a website tinygrad.org. Let your friends know! | tinygrad lazy branch https://github.com/geohot/tinygrad/tree/lazy | Support George by subscribing on twitch.tv/subs/georgehotz | Enjoy quickly made timestamps and suggest better ones in the comments Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro Charron vs. Pat Stay - Battle Rap
00:01:00 reading youtube comments
00:02:13 convolution
00:04:10 new place, openpilot model, ops, data flow
00:05:18 shapetracker
00:06:33 ops_lazy, explaining the graph, nodes
00:08:30 idea, power of tinygrad, reducing the nodes
00:11:20 SHUFFLE_MOVMENT_OPS
00:12:10 REMOVE_MOVMENT_NOPS
00:12:55 MERGE_ELEMENTWISE_INTO_CONV_OUTPUT
00:15:20 plan for today
00:16:30 python typing using class type inside class
00:17:50 real op
00:21:00 thinking, writing the right thing
00:25:25 love types, python the best language
00:26:05 LazyBuffer object has no attribute arg
00:26:30 NoneType object has no attribute name
00:28:00 long variable names, bad behavior editor
00:28:45 why is that wrong, we got that all right on the first try
00:31:15 running in 799.61 ms, what is slow
00:33:25 moving self realize out of toCPU
00:34:39 it should be a lot faster, running in 464 ms
00:35:30 optimization back on and see what breaks, bad investment gbtc
00:36:28 MERGE_MOVMENT_OPS breaking
00:43:40 lazy branch of tinygrad
00:44:20 SHUFFLE_MOVMENT_OPS slow, REMOVE_MOVMENT_NOPS wrong
00:46:20 can anyone fix this bug
00:46:40 mind thinking in currently working projects
00:47:40 optimization without changing the graph
00:52:00 if you have a binary op, can't have larger text
00:56:35 this isn't acually right
01:01:30 python double slash, respect shapetracker slower
01:02:00 fixing bugs
01:09:00 379 ms with shapetracked kernels
01:10:00 unary op shapetracked
01:17:20 tricky, the big gray combiner hard to write code, graph breaking, goal 10 ms
01:23:10 color speed explanation (red, green, gray)
01:30:50 unhashable type
01:40:20 np.int32 really slow (copied to CPU) win for tinygrad
01:41:25 autocorrect extension comment
01:42:00 writing the fix
01:51:18 107 ms big improvement
01:52:00 stupid sort by name, there should be no NumPy
01:58:40 24 ms to run the python, can't beat this, conv.cl
02:01:35 realize explanation
02:03:20 DEBUG=1
02:05:10 making this operations work
02:07:20 the problem was
02:08:00 python the best language
02:08:50 print(len(srcs)
02:10:20 writing some types, writing more code
02:25:30 checking for stupid code, ready to write ast traversal
02:28:00 short break, singing
02:29:00 ast traversal
02:36:00 it's slower and wrong
02:45:00 opencl not type checking anything
02:46:30 working with ast builder commit
02:47:00 let's move this out
02:48:40 65 ms in python brutal, lol slower commit
02:51:50 fix float speedup commit
02:55:10 needs valid commit, unfortunately it's slower
03:01:30 list comprehension, it's wrong
03:06:50 create_built_program_from_source_cached
03:07:20 cache misses
03:09:10 lru_cache 128 default value is a joke, easy fix
03:12:31 most of the time in shapetracker.py
03:20:55 things get slower
03:22:30 cloud scam
03:23:15 all that did nothing, commit not working
03:25:00 local workgroup matters a ton on qualcomm
03:27:55 57 ms commit
03:28:50 convolution that we can use
03:30:40 trying different things
03:37:55 this shouldn't be void
03:42:55 undeclared identifier conv_output
04:01:30 gray one appearing when I disable that,
04:06:40 all waiting for the kernels now
04:14:35 shapetracker needs to be improved
04:19:40 wrong but look at that speed
04:24:30 42 ms is actually fast?
04:30:50 running on batman
04:36:00 George doing an ad
04:37:10 fine we are going to do it
04:57:30 bunch of things we can still do
05:05:20 actually running on the comma three
05:13:30 it worked on the comma three 1.5 s
05:20:35 some questions, comments from non-subs
05:24:05 cheapeth, scams, community, langue de bois, recession
05:27:00 bailing people out, printing money, Kim Dotcom twitter thread US is beyond bankrupt
05:30:20 shut it down party, democracy, government solutions, infrastructure, taxes, state governments
05:36:20 when to move out of US, where to live, comments on a lot of countries
05:40:40 America export the world culture, corrections, infinite jest book, three body problem book
05:42:35 China, chinese content, thank you for watching
3
-
3
-
Watch youtube.com/watch?v=6yQEA18C-XI - George Hotz vs Eliezer Yudkowsky AI Safety Debate | Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/ | Sources for this stream:
- https://twitter.com/realGeorgeHotz/status/1690894647755988993
- https://ourworldindata.org/energy-production-consumption
- https://wimflyc.blogspot.com/2021/01/the-henry-adams-curve-closer-look.html
- https://ourworldindata.org/population-growth
- https://wikispooks.com/wiki/Main_Page
- https://geohot.github.io/blog/jekyll/update/2021/12/18/the-fourth-estate.html
- https://washingtonpost.com/wp-srv/national/longterm/unabomber/manifesto.text.htm
- https://beff.substack.com/p/notes-on-eacc-principles-and-tenets
- https://youtu.be/UT4ApSUkUQM
- https://slatestarcodexabridged.com/Meditations-On-Moloch
- https://geohot.github.io/blog/jekyll/update/2021/07/11/individual-sovereignty.html
- https://youtu.be/6VT8vsiyjTQ
- https://facebook.com/watch/?v=170134181769283
- https://zdnet.com/article/physics-explains-why-there-is-no-information-on-social-media/
- https://en.wikipedia.org/wiki/Fredkin_gate
- https://youtu.be/VdZDVWX6K2Q
To George: Sorry George would be uploaded as one episode, but we saw your comment too late. Also, would make it much harder to do chapters if it would be as one video.
Chapters:
00:00:00 intro
00:00:30 entropics, how much intelligence, compression
00:04:45 industrial, computer revolution, energy usage over time
00:06:42 eniac, intelligence on the planet is about to skyrocket
00:12:00 chess elo over time
00:13:15 how close are we to optimal usage of compute
00:15:25 GPT4 bad programmer, leetcode vs GPT4, project euler
00:23:40 what is overfitting, lossless compression cannot be overfit
00:24:50 college is a scam, stopping progress, henry adams curve
00:26:50 fiat currency, club of rome the limits to growth
00:28:15 birth rates
00:30:32 no understanding how government works
00:32:10 example of scientifically true but socially inconvenient
00:33:25 what happened to universities, Mencius Moldbug
00:34:25 wikispooks, the fourth estate blog
00:35:20 no science that explains this
00:36:20 comma.ai vs ghostautonomy
00:37:08 not understanding, not crazy, unabomber manifesto
00:40:45 is George crazy?, yudkowsky time magazine
00:42:50 Connor Leahy, Terry Davis
00:43:54 manifesto's psychological analysis, HolyC
00:44:25 intelligence explosion won't happen, society is very dishonest
00:50:15 e/acc manifesto, crooks fluctuation theorem, consciousness
00:54:00 Bankless We All Gonna Die Eliezer Yudkowsky, Robin Hanson
00:54:54 how we don't die, don't steal my atoms, buy my atoms
00:57:42 how we die, 1e10x better superintelligence
00:58:15 tinybox will keep me safe
00:58:40 expected many wireheads, psyops, skittles commercial advertising ruins everything
01:00:45 the prime intellect world
01:01:25 hard takeoff in days, society is way more inefficient than we think
01:03:35 moloch slate star codex, prepared arguments
01:05:24 does society make sense for most of you, are you outside the norm
01:06:50 individual sovereignty, Alex
01:08:08 MadDogNation gifted 100 Subs!
01:10:30 George asking Alex if he is crazy
01:12:50 Kanye West
01:13:30 google, marc andreessen, sam altman
01:14:10 shittalking stream
01:15:10 are you the problem, taylor swift
01:17:00 eliezer yudkowsky good faith intellectual, chicken farm man
01:18:22 negentropy, e/acc manifesto, psychohistory
01:20:55 you will never go to space
01:22:10 what is about chat that always derails me
01:22:40 we are not going to day
01:25:00 mind is flat by nick chater
01:27:10 1e20x GG, MadDogNation gifted another 100 Subs!
01:28:10 the hero's journey, real questions
01:30:20 entropics researcher job at tiny corp
01:30:38 how many silicon FLOPS exist in the world
01:31:50 the last question
01:32:45 GPT4 is sick
01:33:45 2.22 zettaFLOPS = 111000 people of compute
01:34:25 taking back every bad thing I said about Sam Altman
01:35:14 how many silicon FLOPS are produced each year
01:36:25 want to make a zettaflop computer
01:37:40 25000*312 teraflops to exaflops, using GPT4 wrong
01:38:20 computer civilization growth
01:39:00 compute, power grows per year
01:41:40 Nick Land
01:42:28 the mind is flat, finite
01:44:00 quantum mechanics
01:45:18 Bojack & Diane do you think I'm a good person
01:47:30 joe pera, physics explains why there is no information on social media
01:48:30 MadDogNation gifted another 100 Subs!
01:49:25 twitter can be run with few people
01:49:50 farcaster, web3, token
01:52:55 david deutsch, the beginning of infinity
01:54:50 GPT4 wow, rapid recursive self improvement
01:57:20 fredkin gate, economy time to double
02:00:30 growth of compute
02:02:40 GPT4 inference on tinybox, George pays for $20 for GPT4
02:03:35 google bard login, respect for GPT4
02:05:10 thank you MadDogNation!
02:06:00 how much compute do you need questions
02:06:35 watch the debate, watch the world
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
3
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
Please go read the bitter lesson -> http://www.incompleteideas.net/IncIdeas/BitterLesson.html | https://openreview.net/pdf?id=9pe38WpsbX (MuDreamer: Learning Predictive World Models without Reconstruction) | NEW research position at tiny corp 02:03:35 02:09:11
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
https://youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | https://youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 muted intro
00:00:20 un-muted
00:00:45 the incident
00:05:10 not recording stream locally
00:05:40 twitch vs twitter revenue
00:07:00 why is RL difficult, data
00:09:25 Richard S. Sutton rl book
00:10:20 every bad thing about data
00:11:35 deepmind soccer robot
00:12:55 neurolink, changing monkeys data
00:13:30 search always works
00:14:20 things to consider when debugging rl pytorch
00:14:50 beta in PPO
00:17:10 cleanup
00:20:50 episodes 40
00:25:10 beta
00:27:00 gamma discount factor
00:28:48 openpilot does not use any RL
00:30:55 committing changes to master
00:31:25 combined experience replay (CER)
00:33:50 go play with the code
00:34:35 debugging RL
00:35:05 don't try stuff and run to see if it works
00:35:40 ML going dark, companies do not share secrets
00:36:35 misguided belief, comma traning, compute
00:37:30 your ability to execute
00:38:10 5 year old reddit post
00:39:20 private data, curating the data
00:40:15 keeping tricks secret
00:40:45 upstream code open source, MIT licence
00:41:30 aixi, bellman equation
00:42:50 bayesian optimization search
00:43:40 ml empirical science
00:44:05 data extraction pipeline, training pipeline
00:44:55 human groundtruth
00:46:20 minimal rl test environments
00:48:00 batch size 256
00:49:00 spam trying programming
00:53:25 why it is so bad
00:54:50 openai spinning up
00:56:20 A Walk in the Park: Learning to Walk in 20 Minutes
00:57:30 machine learning reddit
00:59:20 rlhf
01:05:30 latest from deepmind
01:07:55 transformer reinforcement learning x
01:09:18 Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
01:10:30 comma ai simulator
01:12:10 failed to implement decision transformer
01:13:25 running the nodes
01:14:25 Decision Transformer: Reinforcement Learning via Sequence Modeling
01:15:50 imitation learning better
01:16:30 muzero, initialization
01:17:50 atari 100k benchmark, dreamer v3
01:18:30 tinybox, mastering atari games with limited data
01:19:50 $1000 bounty for solving atari 100k in tinygrad
01:20:35 CARL: Controllable Agent with Reinforcement Learning
01:22:10 dreamer v3 vs efficentzero
01:22:55 MuDreamer
01:23:10 simillar what we are doing at comma
01:23:50 MuDreamer, same problem as at comma
01:26:20 value prediction network
01:28:15 want to solve mario, researcher at tiny corp
01:28:50 wall training time, code short, tinygrad
01:30:25 tinyboxes, pricing
01:32:00 all basics should work on tinybox
01:32:30 just love this stuff, excited
01:32:40 did not delete the stream
01:33:00 through frustration we more forward
01:34:00 paying bounties
01:36:00 not doing tiny problems
01:36:38 hlb cifar
01:38:10 dreamer v3, MuDreamer
01:39:15 Alex
01:41:20 MuDreamer
01:42:05 twitch warning
01:44:00 moving off twitch, onlygeorge.com
01:46:40 instagram reels
01:48:30 twitch front page
01:49:40 Linus Tech Tips reach out, entitled
01:51:10 unroll multiple steps
01:53:00 GRU
01:54:30 i-jepa
01:58:10 curiosity-driven exploration by self-supervised prediction
02:00:00 language model beats diffusion
02:01:20 Finite Scalar Quantization: VQ-VAE Made Simple
02:03:25 MaskGIT: Masked Generative Image Transformer
02:03:35 tinygrad research dream mario 64
02:04:40 high fidelity simulation
02:05:00 copyright game problem mario 64
02:06:00 wayve simulator
02:08:20 the bitter lesson
02:09:11 research position on tiny corp
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
George is live on https://www.twitch.tv/georgehotz (part 2) | https://adventofcode.com/2022 | 00:48:35 !!!WARNING!! loud noise by George blowing into the mic | Go buy comma three from $1999 https://comma.ai/shop | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Follow George on twitter.com/realGeorgeHotz & instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog | Support George by subscribing twitch.tv/subs/georgehotz
Chapters:
00:00:00 intro
00:01:00 bad san francisco
00:01:20 advent of code, hardcore engineer
00:01:55 plans for today
00:02:30 what is scala
00:03:00 scala read file
00:03:10 openai login for chatgpt
00:04:00 how do I read a file in scala?
00:05:15 scala illegal start of toplevel definition
00:06:20 LiveOverflow raid!
00:07:05 sting to int scala
00:07:35 scala partition
00:10:00 Lana_Lux raid!
00:11:30 difference scala array, list
00:12:07 scala calling tip
00:14:55 scala sort
00:15:10 scala list slice
00:16:45 meatballs
00:17:00 aoc day 2
00:21:55 scala map
00:24:00 scala inline if
00:26:40 a lot of scala very nice
00:28:10 what to use instead of scala
00:28:25 kotlin
00:28:45 python slow
00:29:10 food missing
00:36:45 chips
00:44:20 scala string index
00:46:20 indexof scala
00:47:35 scala char
00:48:35 !!!WARNING!! loud noise by George blowing into the mic
00:48:50 free speech
00:49:18 scala char to int
00:53:55 scala for loop
00:54:40 scala zip iter data
00:55:00 how anyone lives in san francisco
00:55:10 scala slice
00:55:40 scala every third element in array
00:57:25 scala mkString
01:00:45 aoc day 4
01:02:20 no food at twitter
01:02:50 vape, chess and twitter
01:04:10 san francisco agitation
01:04:30 twitter relocate to san diego
01:04:45 austin, politics
01:05:30 unhoused persons
01:05:45 twitter files
01:06:00 twitter search ban
01:06:10 work from home
01:06:30 basic corporate behavior broken
01:06:50 what we are building, the public square
01:07:23 google search
01:07:35 this week work, twitter search
01:07:50 small mediocre features
01:08:00 impressive demo that can't be shipped for 2 years
01:08:20 can't do in 12 weeks
01:08:30 good infrastructure
01:08:45 can't stay in this city
01:08:55 twitter is nice, need better food
01:09:25 agitation
01:09:45 talent at twitter
01:10:20 twitter in san diego
01:10:30 incredible ml stuff needs a year of work
01:10:44 your codebase reflects your organization
01:11:08 refactoring or features
01:11:30 slack channels
01:11:40 talent at twitter higher
01:12:20 refactoring improves performance and velocity of new features but takes time
01:12:55 elon on average better decisions
01:13:15 twitter infra
01:13:30 advent of code this year, no green on github
01:13:45 twitter microservices
01:14:10 scala fine programming language
01:15:25 chisel3
01:15:45 need food
01:16:45 opensourcing twitter
01:18:50 scala string split
01:23:00 advent of code complaints, reading comprehension
01:24:55 twitter login pop up
01:26:10 political battle
01:27:00 mastodon, truth social
01:28:30 all cool people on twitter
01:29:00 hoon is a disaster
01:30:25 elon is not a political extremist
01:31:25 trump ban
01:32:35 line should be illegal in usa
01:34:15 what is a country
01:34:50 rules for the town square
01:35:00 moderation is not censorship
01:36:00 brandenburg test wikipedia
01:37:25 clips not allowed, hitler not good
01:38:15 rules average of eu, us
01:38:45 violence not ok
01:39:05 kanye needs to chill
01:40:10 twitter hiring
01:40:35 python 3.12
01:40:45 politics is interested in you
01:41:10 free speech reason why I came to work at twitter, comma.ai record profit
01:42:10 real change coming, comma winning
01:43:34 chatGPT, Text-Davinci-003 demo, meatballs
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
Chapters part 2:
04:19:30 AMD last chance, latest trunk driver, opencl context, irrecoverably crashes
04:24:30 building kernel takes a long time, plum bonito amd
04:28:25 geohot rapping, tom delonge, debunking navy videos on youtube
04:30:30 amdgpu driver walkthrough, make clean
04:36:35 is it even boot, sad stream, if you have to build a custom kernel, trying trunk, I never send stuff back
04:37:35 here is my deal with AMD, I don't want money, release all documention on this GPU and will build a driver
04:38:00 problem with legacy, supporting graphics, amd gpu psp, $1000 cards
04:43:00 naming nvidia code names, sudo make install
04:49:15 computer alive, clinfo, pyopencl for tinygrad, kernel build
04:52:30 if this does not work they are going back to amd
04:55:40 until 25 may to return them
04:57:18 crash again, failed to add hardware queue to MES, it can be the card broken
05:02:00 how long should we give AMD, by 12 may some sign that you care about your drivers
05:07:00 swaping the GPUs, no crash
05:17:35 bandwidth test
05:23:55 it's not the GPU crash reproduced on both
05:26:10 George going to New York, did we give AMD the benefit of the doubt
05:30:10 appreciate open source, vbios 7900xtx, rocm smi reset gpu
05:44:00 kernel that is not that bad, we can reset it, amd appreciate if you cared, does amd want this
05:47:00 does amd want their cards to be beloved by machine learning enthusiasts
05:54:40 AMD MES - MicroEngine Scheduler
05:57:50 virtualisation being enabled in bios issue, can we crash beyond repair
06:03:00 hate sending them back, enable bios, checking for pci errors
06:13:41 hsviewer keyboard not working
06:16:51 vast.ai, buying 7900xtx stupid
06:19:10 checking bios, bios for ants, boot delay, config.sys
06:22:15 wait for bmc disable, can't undo 10 years of crappy drivers in a stream
06:24:10 hire the tiny corp, write a clean driver just for deep learning
06:27:50 61 FLOPS promissed delivered 34 FLOPS
06:28:30 AMD to mlperf, intel arc worse then amd driver
06:30:10 we need a sign from AMD before R_DAY
06:32:20 RDNA3+ canceled
06:33:15 Jensen wins
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
From $1499 buy comma.ai/shop/comma-three | 01:56:30 bounty for $50 via paypal | Read americanaffairsjournal.org/2021/08/the-value-of-nothing-capital-versus-growth/ | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:52 plan for today
00:03:10 preparing whiteboard
00:08:35 using triton
00:09:15 comma contracting tinycorp
00:11:30 qualcomm, tinycorp vs pytorch 2.0
00:12:15 ai chips, triton
00:12:52 jim keller, tenstorrent, shapeways
00:14:15 no ugly kernels, vs nvidia
00:15:05 scaling with gpu's, writing the right code
00:16:40 tinygrad kernel
00:18:35 tensor.py
00:21:00 onnx.py
00:21:20 tinygrad.org
00:22:40 easy to add backend
00:24:35 net.svg
00:27:57 alex
00:28:30 tinycorp needs to make money
00:29:35 tinygrad stable diffusion output
00:30:20 mlops.py
00:30:50 lazy.py
00:31:30 tinygrad vs pytorch 2.0
00:32:50 pytorch 2.0
00:33:10 250 vs 15 primitive operators
00:34:15 can tiny grad autdiff through exp(M)
00:35:50 frame evaluation api
00:36:34 pytorch inductor
00:37:00 zygote ad framework
00:37:45 writing a backend for pytorch
00:38:30 thinking what to do
00:39:50 torch inductor
00:40:35 not logging in to github
00:41:35 hope for tinygrad
00:45:10 trade offs
00:46:00 compile.py
00:47:35 mps support
00:50:25 NVIDIA_TF32_OVERRIDE
00:51:15 gpu vs cuda
00:51:54 3080 ti, reversing the ANE
00:52:13 library should be fast
00:54:40 ssh to ubuntu computer
00:55:08 poll slow kernel debug, python 3.12 upgrade, triton port
00:58:10 slow kernel debug
01:18:00 invalid command queue
01:29:00 copilot not good, thinking what to write
01:35:30 correct and faster
01:39:30 flip extra dimension indexing for speed
01:40:30 reduce error
01:49:15 explaining what group_for_reduce
01:51:10 hard to write
01:53:00 search space, should not write by hand
01:55:40 hard to implement search
01:56:30 bounty for $50 via paypal
01:57:40 kernel search is the way
01:58:40 explaining the search problem
02:07:35 search space
02:12:10 way slower
02:13:40 first restriction
02:18:00 kernel_search.py
02:24:05 ordering food
02:28:00 how will tinycorp make money
02:31:30 minimum contract price for tinycorp
02:32:05 aritter2 good search suggestions
02:34:10 first little attempt at search
02:42:00 optimise every op in isolation, A*
02:43:10 accessible to people
02:48:10 encouraging still take a week, upcast, shape must be ints and can't contain 0
02:51:35 complex google optimizer
02:54:40 why did I not do this sooner
02:58:25 a framework
03:00:35 cleanups
03:16:30 upcast, winning interventions
03:17:40 super promising, thank you to aritter2
03:18:55 programming competitions, aoc, project euler, ctf
03:21:35 computer security, info sec, bug bounty
03:23:04 not everyone is going to like you
03:24:35 nejintejag question shapes network effects function
03:26:20 reading youtube comments
03:27:00 compute for comma four, five, jim keller
03:28:20 can't buy qualcomm, nvidia chips, nvidia orin
03:33:30 amd accelerators, amd mi25, apple M2 chip
03:36:40 reading youtube comments, hard problem of consciousness
03:41:05 excited about search
03:41:55 moving to ukraine, joscha bach, emad mostaque
03:43:45 how I seduce my crush
03:44:10 no drugs, nostr
03:45:25 jesus, christian, catholic
03:46:45 jewish people smart, kanye west
03:47:50 bad culture, bad reach out to work at comma
03:48:50 openai, ai safety, working at comma
03:50:40 woke college culture, non-issues, marriage
03:54:10 people trying to change your culture, johnny hobo new mexico song
03:55:40 not eating meat, plants, math, the freshman requirements, main vs master
04:00:30 stupid topics, hitler lost the war, russia vs china, 10 lane highway
04:03:55 sovereign countries, unified world order, pollution in china
04:05:00 american culture, shapeways stock, spacs, elon WEF, tiktok, google, fb
04:07:20 offer to private equity firm for hostile takeover of shapeways
04:10:15 metaverse is lame, no comma 4, john carmack, AGI, among us vr, stream deck
04:13:25 comma body arm, the value of nothing capital versus growth, china, resurgence in ai, malaysia
04:18:25 leave comments in the video and we will answer them on next stream
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
tiny corp is hiring! -> 3x Software Engineer, 1x Hardware Engineer - https://tinygrad.org/ | It's happening George started a project -> Getting AMD on MLPerf -> https://github.com/geohot?tab=projects | Go help George if you are capable! He will appreciate you! | From $1499 buy comma.ai/shop/comma-three | LLaMA https://arxiv.org/pdf/2302.13971.pdf | github.com/geohot/tinygrad/blob/llama/examples/llama.py | github.com/geohot/tinygrad/tree/llama | tiny corp has a website tinygrad.org | Check out and star github.com/geohot/tinygrad | Support George by subscribing twitch.tv/subs/georgehotz | Follow George on instagram.com/georgehotz & twitch.tv/georgehotz & twitter.com/comma_ai to be up to date | Read George's geohot.github.io/blog/ | Programming archive🡿 youtube.com/playlist?list=PLzFUMGbVxlQs5s-LNAyKgcq5SL28ZLLKC
Chapters:
00:00:00 intro
00:01:30 guitar tuning and playing
00:04:30 giving advice to elon, woke
00:07:25 llama, trip to india, bengaluru, bike to mumbai
00:08:40 when software was cool
00:11:20 llama, privilege
00:13:45 discord ban, pressure, power
00:16:05 prime, blue crown, money
00:18:00 tiny corp, making value, food
00:22:25 capitalism
00:23:50 facebook LLaMA, apply for access, torrent
00:25:10 ai safety, enjoy the decline
00:28:00 deepmind, google, rants
00:28:40 thinking for yourself, jesus, george is christian, god
00:30:30 a progression to christian
00:31:50 big text, copilot
00:35:10 running on gpu m1
00:39:30 using tinygrad tiny ram usage, float16
00:46:00 METAL=1
00:47:25 numpy size of datatype
00:49:12 interesting, np.frombuffer
00:52:10 problem raw buffers to have a datatype?
00:52:55 python read into buffer, not making copies = fast
00:59:40 loading from disk, making it lazy
01:00:50 tinygrad is the future, replacing programing with dsp paradigm
01:02:17 tinygrad has grown
01:03:50 metal buffer, float16 hard
01:05:20 interview question how big is a float
01:06:40 making metal buffer, supporting dtypes
01:09:40 tinygrad recognize llama
01:10:25 facebook llama arxiv, unbiased model
01:13:25 class Attention, FeedForward
01:21:40 washed heads
01:24:00 class RMSNorm
01:30:25 testing inference
01:35:30 error and slow loading because torch
01:39:00 wasting 0.6 seconds
01:40:45 how transformers work, Linear
01:47:20 torch not defined, if you are slow you get undefined
01:48:00 fstrings, shapes matching
01:48:45 writing calls
01:55:00 class Attention
02:02:00 missing position
02:05:10 not matching
02:13:00 why did that happen?
02:18:10 memory should not be 0
02:19:50 just creating the weights is breaking it
02:24:30 why so much memory?
02:25:15 getting dtype in inference editor
02:26:15 WEIGHTS=1
02:29:30 number of ops and memory use
02:31:10 9 seconds = 1 character, tensor cores
02:32:40 fastest metal_matmul.py on M1 = fastest llama on M1
02:36:10 stable diffusion interesting ops
02:37:10 dynamically loading the weights from zip file
02:37:30 biggest llama on M1
02:39:50 full tiny llama grid
02:42:20 looking at the paper, thank you to facebook
02:43:55 dollars are arbitrary
02:44:50 licence for ai safety, mmap
02:47:40 tiny corp no money
02:51:20 making the compiler running less
02:52:50 bought land, rolls-royce and meta stock
02:54:10 sam bankman-fried
02:56:30 little people
02:58:00 andrew tate making money, kiwi farms, coffeezilla
03:00:20 entire economy is a fraudulent scam
03:01:50 AGI
03:06:30 how would George do in prison
03:10:00 un focused today
03:18:00 floats should not be floats
03:22:30 generated c files
03:23:10 openpilot compile
03:32:00 missing 1 required positional argument x
03:34:10 faster
03:36:00 terrible support for halves
03:38:00 numpy bad chat suggestion
03:38:45 bugs in the training
03:42:30 ENABLE_METHOD_CACHE=1 should be default
03:43:30 when you make something 0 page and copy on write, take operating system class
03:45:40 not ready to generate a word
03:46:00 using tinyjit for training
03:47:15 rope is hard
03:49:35 importing SentencePiece
03:51:35 cheapeth, how does anyone take anything seriously
03:55:00 list must contain integers
04:02:10 offensive code because it uses complex numbers
04:08:00 can't trust anybody
04:11:30 macbook pro m1 max ssd read speed
04:14:50 dealing with company problems
04:15:20 not starting a company
04:16:25 complex multiplication, the moment
04:17:20 what we missed off stream
04:18:35 never start a company, llama is working
04:24:30 SentencePieceProcessor
04:32:10 Kristen Rengren
04:34:50 temperature sampling numpy
04:41:25 TechQuest Laser Tag Gun
04:44:20 singularity
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
Buy tickets https://www.eventbrite.com/e/comma-con-tickets-153553216805
Schedule (PDT)
10:00AM-10:05: Opening remarks
Alex (COO) and Chris (Head of Production & Communications) welcome you to COMMA_CON
Talks include 10 minutes of Q&A at the end. There is such a thing as a bad question.
10:05AM-10:50: Willem Melching -- Head of openpilot
openpilot has evolved from a hobby project to legit some of the best software in the automotive world. In the process we’ve written a whole set of robotics software (messaging, kalman filters, localizer) that outperform everything available today.
Come to this talk to learn how openpilot works. It’s quite sophisticated, yet still approachable. Learn the Android of self driving before everyone else.
11:00AM-11:50: Harald Schäfer -- CTO
Get the data, train the models. Easy, right?
Not even close. From the world’s most sophisticated automatic ground truthing stack, to an agent trained in our “Small Offset Simulator”, you’ll learn where the magic comes from, and how the magic will continue all the way through to level 5. Fun and easy!
12:00PM-1:00: Lunch: Burgers
1:00PM-1:50: George Hotz -- President
comma is a profitable company that provides value to 1000s of users. How do you keep focused and build something like that in a world with clown valuations, fake demos, and phony products?
Our mission is “solve self driving cars while delivering shippable intermediaries,” we aren’t stopping at highway autopilot. Come to this talk to learn about what’s next, and how the clowning will continue until the end. You won’t want to miss this.
2:00PM-2:50: Robbe Derks + Hardware Team
Building a car is hard. Working with car companies is impossible. We chose the third option. Working with existing cars. comma did this so casually you’d think it was easy. It wasn’t.
We interface with over 100 different cars. Want to understand how we’ve managed to do that without ballooning complexity? Come to this talk.
3:00PM-3:25: Adeeb Shihadeh-- openpilot Engineer
Come to this talk to see a secret new feature be revealed!
3:30PM-3:55: Greg Hogan -- Head of Infrastructure
We have a 50 kW on premise datacenter with 2,000 cores, 80 GPUs, and 3 PB of storage. Learn how it came together in this talk.
4:00PM-4:25: Weixing Zhang-- Research Engineer
Camera based driver monitoring (DM) is a key part of the magic of openpilot. It’s so good that Tesla copied it 2 years later!
Come learn how it was built into the best DM on the market by one engineer part time. Understand the power of the comma research stack, crowdsourcing, and big data.
4:30PM-5:20: Group Chat / QA
George will get on stage and lead a group conversation about comma and openpilot’s journey from hobby quality crap to the top rated ADAS system.
5:30PM-6:30: Happy Hour + Community Lightning talks (apply to alex@comma.ai)
7:00PM-8:00: Dinner: Flatbreads
7:00PM-8:00: VIP Dinner
Sit-down dinner with the comma team (3rd floor)
8:00PM-10:00: comma Jeopardy
Three teams of four people will compete in two rounds of comma themed questions. Test your knowledge of comma, cars, and computers while drinking free pitchers of beer on stage. VIP tickets get first dibs on teams, the rest of the players will be selected randomly. Score the most points and win a “prize!”
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
ecromata.com |
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:00:10 rack gone, donuts, full time android
00:02:50 a brief history of intelligence
00:04:25 skibidi toilet
00:06:20 hipstreamwritevalue32, hipstreamwaitvalue32
00:10:40 linearizer
00:15:30 optimizations for shapetrackers and group for reduce
00:16:16 graph
00:17:10 ADD and LOAD, topological sort
00:20:30 Alex
00:23:15 linearizer great code
00:26:00 most garbage code in tinygrad
00:27:30 difference, liniar list of code
00:29:20 Alex
00:31:00 the idea of new linearizer
00:31:45 kernels that can store more than one thing
00:32:50 food
00:34:20 buffer loads, shape, buffer iteration
00:36:40 how do compilers work?
00:38:00 chris lattner
00:39:00 color dimensions
00:40:00 tvm, optimism work, thinking about what you want to write
00:41:40 installing tvm
00:46:00 jonathan blow precedence compiler
00:48:20 binding axis to gpu variables
00:50:30 cse_var_2, common subexpression elimination
00:51:10 reading documentation from others
00:52:30 not thinking
00:53:55 tvm_gemm.py
00:55:20 what is tvm
00:55:40 schedule API in tinygrad
00:56:10 linearizer
00:57:10 jira in companies, project management
00:59:40 jira board, complexity, work signaling
01:00:40 tiny corp board
01:01:35 better expression of ideas
01:02:00 Proofs from THE BOOK
01:02:25 scrum, sprints at comma, agile
01:04:00 above the API line
01:04:40 comma is going to beat tesla in FSD, adding value tomorrow, V12
01:07:00 if anyone want's to write tinygrad docs, tutorials like tvm
01:07:40 blame, working with elon, elon's companies structure
01:09:00 the concept of non technical project manager
01:10:10 better programmer than people you manage
01:11:20 senior vs junior engineers
01:14:49 type inference
01:15:45 junior engineers, learn lessons the hard way
01:17:10 not getting more intelligent
01:18:50 competition programming, high schoolers
01:20:20 spam programming vs slow high quality programmer
01:21:55 codegen and scheduler
01:23:20 halide tvm
01:25:10 complex, chris lattner about tinygrad, compilers
01:28:20 model optimization tvm
01:30:00 building tvm, tvm output verbose code
01:33:00 tvm same ideas as tinygrad, tensor accelerator
01:35:00 group_for_reduce, kernels
01:37:00 searching for the right name
01:40:00 graph rewriting
01:40:50 break
01:48:56 hhkb professional hybrid type-s
01:50:15 blueprint name
01:51:10 ajusting new keyboard
01:57:50 vscode spaces
02:01:50 Alex
02:04:30 typing on new keyboard
02:06:30 new linearizer concepts
02:16:50 group
02:20:30 comments overrated, wrong documentation
02:26:00 ecromata.com
02:27:00 hammer = perplexity
02:28:00 land, managing people, plans
02:31:40 starting your own company, selling land
02:33:30 no capital constraints, how do you deploy $500M
02:34:20 delusional founders, what limits the growth, 10 new comma people, integration to the company
02:35:00 working together superlinear effects, scale book
02:36:00 practical factorization of the problem, limiting factor of growth
02:36:15 complex problem, breaking into appropriate parts
02:37:50 Tesla, Elon average decision better
02:39:20 management coefficient, people
02:41:10 cult followers, dedicate your life, hitler, burning man
02:43:20 social media, dating apps, quality
02:44:10 low and high buyin
02:45:30 hippie cults 60s failing, machines, energy
02:46:30 robots, people who want to take way more than taking back
02:48:40 bounty systems, burning man works, people dynamics, social dynamics
02:51:00 what are people, how do they work together, power in system, market dynamics
02:55:30 inadequate equilibria
02:56:00 the bible stories, stoicism, modern scientific people dynamics, racial differences in intelligence
02:57:40 what can you change, what you can't change, how change happens
02:59:20 clones scaling
03:02:30 power on the land, solar panels
03:08:35 projected revenue numbers, fake wineries
03:09:50 horse boarding, we live in capitalism = good
03:11:30 aravind srinivas perplexity, swimming upstream, tinyboxes on land
03:15:10 degree value, poker
03:18:15 land location, road to land
03:21:40 market already priced it in, hard to beat the market
03:22:10 linearizer 2 week project, 5 times more inteligence vs 10 clones, piano
03:25:00 housing expensive, how to make money, fraud, ben shapiro rap
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
github.com/geohot/dumbrl | stable-baselines3.readthedocs.io/en/master/ | youtu.be/FVGQJM2MVNY (Deadliest Journeys - Congo: The Last Train in Katanga) 01:01:11 carefully building infrastructure, CI testing | andyljones.com/posts/rl-debugging.html | spinningup.openai.com/en/latest/ | https://arxiv.org/pdf/1912.02875.pdf (Reinforcement Learning Upside Down)
Bounties for tiny corp / tinygrad -> docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/
https://youtu.be/lnVQsJJFcdg?&t=13715 Hiring entire stack for tiny corp join if you are interested | https://youtu.be/lnVQsJJFcdg?&t=14195 work major source of value in your life
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:01:40 stream disclaimer, twitch ban
00:02:45 only 50% of subscription money
00:03:05 kick.com streaming
00:04:45 kick reach out to George, twitch issues
00:06:20 drugs banner, legal in california
00:08:05 50% money to twitch too much, twitch remove the banner
00:09:20 hyubsama food stream, twitch banned users
00:12:00 streaming on X, negotiating power
00:14:40 stream statistics, streaming schedule
00:16:10 applying for twitch partner
00:17:30 twitch revenue
00:18:30 perplexity best way to get banned on twitch
00:22:50 andrew tate impression
00:23:50 stable baselines 3
00:29:40 np.random.randint
00:32:44 NoneType object does not support item assignment
00:33:00 perplexity
00:35:40 render mode defined human
00:37:20 good play, size=10
00:40:30 stable baselines 3 just works
00:50:00 passed a tuple, array element with a sequence
00:51:15 learning
00:52:50 decision transformer stable baselines 3
00:55:20 github.com/geohot/dumbrl
00:56:30 cartpole, stable baselines decision transformer
00:59:30 Jax, wrapper for vectorized environments
01:00:30 deadliest journeys congo, ancestor pothole
01:01:11 building infrastructure, fixing the road
01:01:20 bugs, carefully building infrastructure, CI testing
01:04:00 README
01:05:40 deleting a lot of tinygrad, focusing on what needs to work well
01:09:55 decision transformer repo
01:13:10 beautiful_cartpole.py
01:20:07 andy jones debugging rl
01:24:00 if you are following along
01:29:00 the problem are bugs
01:32:00 asking perplexity, openai spining up and deep rl
01:36:25 log_softmax
01:39:00 broadcasting bug, 2, 3, 5
01:47:20 no detach(), ppo, exp
02:02:00 why is my ppo not working
02:07:40 fast cartpole
02:11:50 banned user
02:15:50 asking it to learn
02:17:15 hyper parameter land
02:25:45 lucky
02:27:30 !!!LOUD WARNING!!! why it's not solving
02:33:20 3 layer network
02:41:30 value function
02:42:50 writing pytorch
02:48:10 if it works in pytorch shutting down tiny corp
02:52:50 pytorch numeric stability
02:55:10 frustrating, having faith in tiny grad
02:56:00 very easy to make progress in tiny grad
02:57:18 tiny grad more numerically stable
03:04:00 the most dead simple thing
03:13:30 size 2, 3 solving
03:14:35 going even simpler
03:19:00 batch size = 4
03:22:40 reward broken
03:27:10 it becomes like an identity matrix over time
03:28:40 this is fire, the gradient, single weight matrix
03:33:00 so beautiful, love watching deep learning happen
03:44:10 learning rate too high
03:51:00 that one does not learn
03:52:40 dying relu, 0xnan getting VIP
04:00:40 advantage
04:06:40 Alex on the phone
04:08:45 no clips, taking out of context
04:11:00 value funtion all noise
04:18:10 graph go up
04:24:00 messing with hyperparameters randomly
04:26:10 slow graph drawing
04:28:20 sampling bias
04:29:10 lower discount factor, larger replay buffer
04:32:45 no major bugs, ppo major bug
04:36:20 entropy loss
04:38:40 counter intuitive in deep learning, bigger learn better
04:40:40 overheads
04:41:40 one good landing
04:42:55 50, 51
04:44:10 Alex home
04:46:30 send this video to a doomer
04:48:00 good enough landing
04:50:40 expectations too high
04:51:35 twitch won't contact George
04:53:30 hope, upside down rl, juergen schmidhuber
04:54:10 good reliable solution to everything
04:54:40 Alex, no checkpoints
04:55:10 last landing, end of the episode
04:55:30 thank you for watching
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | If you have skill apply on -> comma.ai/jobs | Join the community on github.com/commaai/openpilot | Check if your car is supported -> comma.ai/vehicles | How to setup your car comma.ai/setup | follow the official youtube.com/commaai for more livestreams | Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop.
Some parts of the video muted because copyright music - original video is blocked because of copyright youtube.com/watch?v=Q319h43SHnc
Chapters:
00:00:00 intro
00:01:10 Alex welcome to comma_con
00:02:12 Harald Schäfer, CTO: A Drive to Taco Bell with end-to-end Machine Learning
00:34:50 Harald Schäfer questions
00:41:20 Alex introduces Adeeb Shihadeh
00:42:56 Adeeb Shihadeh, CPO: Shipping openpilot
01:02:55 Adeeb Shihadeh questions
01:24:45 Alex introduces Mitchell Goff
01:25:32 Mitchell Goff, ML Engineer: Navigate on openpilot
01:47:55 Mitchell Goff questions
01:57:42 Alex introduces Weixing Zhang
02:00:04 Weixing Zhang, Research: Human-Machine Cooperation in openpilot - a machine learning approach
02:19:30 Weixing Zhang questions
02:30:10 food break
03:42:28 George Hotz, Three Stories
04:27:00 George Hotz questions
04:39:35 Alex introduces Robbe Derks
04:40:50 Robbe Derks, Hardware Engineer: How it's made
05:15:58 Robbe Derks questions
05:24:32 Alex introduces Vivek Aithal
05:25:25 Vivek Aithal, research: Can you tune my car?
05:47:10 Alex introduces Greg Hogan
05:53:03 Greg Hogan, Head of Infrastructure: comma supercomputer
06:16:52 Greg Hogan questions
06:30:12 Alex introduces Yassine Yousfi
06:31:01 Yassine Yousfi, Research: Learning a Driving Simulator
06:54:50 Yassine Yousfi questions
07:13:50 Alex introduces Jason Young
07:17:10 Jason Young, Community contributor: How to port a car
07:45:50 Alex finishing the live stream
07:46:35 COMMA_CON commies
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
nintendo.com/us/store/products/the-legend-of-zelda-breath-of-the-wild-switch/ | nixos.org | zero-to-nix.com
Pre-order tinybox buy.stripe.com/5kAaGL6lk9uX9nW144 more info on -> tinygrad.org | github.com/tinygrad/tinygrad <- simple powerful deep learning framework
tiny corp is accepting new interns. more info on tinygrad.org and tinygrad discord | comma.ai is accepting interns comma.ai/jobs#open-positions
$1250 buy -> comma 3X comma.ai/shop/comma-3x | best ADAS system in the world openpilot.comma.ai | from $999 comma.ai/shop/body the future of people
Support George by subscribing twitch.tv/subs/georgehotz | Follow George on twitter.com/realGeorgeHotz to be up to date | Read George's geohot.github.io/blog/
Chapters:
00:00:00 intro
00:02:50 oneplus customer support random number generator
00:03:15 legally purchased the game and legally emulating
00:03:55 breath of the wild on M3
00:30:55 just one terminal remaining!
00:39:10 you activated all the terminals!
00:41:45 game over
00:43:10 dueling peaks tower
00:48:22 spirit orb
00:55:10 the water guides, ha dahamar shrine
00:59:25 happy with this game, spirit orb
01:00:20 dueling peaks stable
01:01:50 wild horses
01:11:00 locked mementos
01:15:38 ta'loh naeg shrine
01:23:53 game over
01:25:04 waterblight ganon
01:27:35 what is nix, declarative builds and deployments
01:28:20 Zero to Nix
01:29:37 nix installation
01:31:15 run a program with Nix
01:33:05 downloading a lot of stuff with Nix
01:35:45 Nix flakes
01:36:30 shell environment
01:37:50 lolcat github
01:39:40 search for packages
01:41:45 what is nix, reproducible environments
01:43:00 nix language basics
01:44:50 Terraform
01:45:14 too obscure, embrace nix flakes
01:47:33 last time much worse 19GB download
01:49:00 oneplus refund by calling multiple times
01:51:05 what is nix shopify engineering blog
01:53:55 build a package using Nix
01:56:05 downside to good packaging manager
01:57:02 tiny keyboard
02:00:45 how to brick oneplus open, vip, qdl
02:02:00 apache kudu
02:02:40 wrist wrests
02:05:00 run apache inside nix
02:06:44 trying to do something that is not the tutorial
02:08:48 mediawiki package
02:10:15 riff flake example from DeterminateSystems
02:15:45 nixos immutable, it's how package manager should work
02:20:00 ubuntu core immutable
02:21:25 mediawiki struggle
02:28:20 docker mediawiki
02:30:44 where does the docker put the state
02:36:40 apachectl, who setup a LAMP stack in the last 10 years
02:38:10 apache inside nix struggle
02:41:40 good first time chat, provides information
02:42:25 we don't do it because it's easy, we do it because we thought it would be easy
02:44:50 nginx mediawiki
02:46:00 what is nix flake
02:48:15 arch linux vibes
02:50:40 diselbabyy guide, caddy rage quit
02:52:30 are we trying to do something unreasonable?
02:56:55 building flake.nix
03:04:25 flake-utils utils.lib.mkApp
03:13:49 nix mkDerivation
03:15:30 they know the meta, how did I know pname is the right thing to do there
03:16:25 nix is arch linux docker for haskell fans
03:17:10 nix not for George
03:18:40 never making a nixpkg, better than last time
03:19:30 rust tinygrad rewrite
03:21:25 getting the meta in the games, same skill for computers
03:23:05 slurm workload manager
03:24:20 tinybox gpus
03:25:00 tenstorrent software stack, pivot to RISC-V
03:25:40 this is really bad, you are not ready to sell your thing
03:27:25 tinybox pictures, tinybox PSU
03:30:10 server grade PSU too loud
03:30:40 tinybox price $15k
03:31:45 9 noctua 140 mm fans
03:32:10 90 pounds tinybox
03:32:45 thermodynamics fridge vs tinybox
03:35:20 plan with the self driving thing, selling the box
03:37:50 hacker news people about cheapETH
03:39:55 crypto is only good for regulatory arbitrage, bankless podcast
03:44:20 tiny corp raised $5 mil and sells $5 mil worth of boxes
03:45:10 21 bitcoin computer
03:47:05 regulation does not hold back crypto
03:47:25 triangle, anonymous compute
03:49:25 founders that cashed out, once you delivered the value
03:50:25 moon math
03:51:05 TuSimple scam
03:51:50 ethereum switch from proof of work
03:52:05 money going to the right people
03:55:10 multiculturalism
03:56:45 bitkey
03:59:15 tenstorrent if you can't buy it it's not real, nixos is docker arch linux
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
Sorry for potato quality! | Check if your car is supported -> comma.ai/vehicles and buy -> comma.ai/shop/comma-three from $1499 | blog.comma.ai/dumb-questions | What is comma.ai/openpilot | Join the comma team comma.ai/jobs | How to setup your car comma.ai/setup | Follow the official youtube.com/commaai for more livestreams | Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop.
Chapters:
00:00 intro
00:35 comma body remotely controlled
01:05 comma body hat and training wheels
01:55 comma security telepresence
02:10 comma body control website
02:55 the hat, crown
03:20 the dream, mars rover
03:54 Harold, Adeeb
04:15 hiring
04:25 disengagement metric
04:55 control metric
05:35 web team hiring
06:00 openpilot hiring
06:20 openpilot team
06:35 hangout place
06:44 going to car, demo, ui updates, ford
07:10 pr, marketing
07:25 heat testing setup
07:50 ford bronco sport
08:05 ford bronco sport openpilot support
08:50 ford f150 support
09:35 experimental mode
09:50 stopping for stop sign
10:15 ford platform openpilot 0.9.2
11:20 ford long not shipped yet, radar support
12:05 ui changes
13:04 boot time, new models
13:33 0.9.2 release date
13:50 ford maverick support
14:15 any remarkable improvements coming to Bolt with 0.9.2: below 6 MPH control
14:25 nicki minaj model
15:00 low speed on ford
15:15 turning bug fixed
15:40 hiring bounty comma three + $3000
16:00 releasing taco drives for production
16:20 navigate on openpilot, map
17:00 bounty hiring program
17:45 can I go buy an f150 to add my c3
18:05 complaints about openpilot
18:25 following distance adjustment
18:55 customer support engineer role, but must that be local in San Diego
19:20 thermal improvements, 0.9.3
19:45 ui changes in 0.9.3
20:05 limited steering torque
20:20 civic ping pongs a little on the highway
20:35 experimental for bolt
21:10 bronco smooth steering
21:25 metric, torque controller steering
21:45 0.9.2 no disengage on gas default
22:10 the testing simulation work for testing end to end longitudinal experimental mode behavior
23:10 ram start speed activation to 0
23:15 2023 palisade/telluride joining the official support list
23:40 what's the holdup for nav on openpilot in experimental in master
24:35 ui path changes
25:25 does experimental mode use map data at all for speed limits
26:00 does it try and reach that max speed
26:25 what fords is this going to be available for
26:50 only new ford models will be supported lockout still not hacked
27:05 ford is one of the biggest auto in usa
27:20 ram support
27:35 checksum of hybrids different
28:05 should make it so you can livestream right off of comma 3
28:20 comma body live streaming
28:45 sentry mode, f150 and use plug and play timeline
29:10 thoughts on Tesla going to end to end
30:30 third paradigm use overrides to punish the model during training
31:05 master-ci hasn't failed to build in a long time! great testing
31:55 old website looked more clean and professional
32:05 experimental mode and chill mode should have separate cruise speed settings
32:15 brazil we do not have this bronco, the BSM is sophisticated equal to the KIA EV6
32:35 who is comma brasil
32:55 is there a planned release for navigate on openpilot
33:25 progress to taco, corner radars like EV6
33:40 toyota the biggest auto that's unsupported
34:05 thanks for watching
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1