Comments by "Mikko Rantalainen" (@MikkoRantalainen) on "The Exciting, Perilous Journey Toward AGI | Ilya Sutskever | TED" video.

@kevinjacob3022 We already have Llama and Openchat / ChatBot UI. However, training an AI model requires insane computational efforts. If you wanted to improve even Llama 7B model (which is considered a small model these days), you would need about one million USD worth of computer hardware to have a single try in a year – that is, you have an idea how to improve the training results, write some code and then you start the training process and look at the finished training results about a year later when the computation is ready. If you have lesser hardware, you could do the same thing but one try could take about 100–500 years. This is the biggest roadblock in front of open source AI development right now. Unless somebody figures out some way to train the models with 10–100x less computing resources, hobbyists will have to wait a decade or so until powerful enough hardware is accessible. The estimated cost of training GPT-4 was around 100 million USD. That's the resources needed to do the computation part when you have already collected all the data and have the AI software ready.
1