Comments by "Vitaly L" (@vitalyl1327) on "I Want LLM Agents To Be Good" video.

@RomeTWguy a lot of people build their internal systems this way, myself included. There is no point in making a product out of it, since cloning such a product is trivial. Devin is a scam, it does not add any value because it is so easy to make a better system nearly for free.
2
You have no faintest idea of how to use LLMs. Where is your feedback loop? Where is critic? Where is code sandbox? Why did you fail to use SMT for reasoning?
1
@mortiz20101 if you failed to comprehend what I'm talking about, you probably should not use LLMs. 1) Feedback loop: feed the result of testing the LLM output back into LLM, with all the relevant results (syntax errors, test failures, static code analysis output, etc.) 2) Critic: every time an LLM is producing an output, do it a few times with the same prompt, and then use another LLM prompt to criticise all the outputs and select the best one out of them. 3) Code sandbox: give LLM a tool to run arbitrary code in a safe sandbox. Use inference harnessing to ensure the tool is used immediately as the call appears in the output. 4) SMT, Prolog, etc. - LLMs cannot reason, obviously. But they can translate an informal problem into a formal language. Which can then be processed by an SMT solver, a Prolog interpreter, or whatever else you use as a reasoning tool. You have a lot to learn. Do it. Or stay ignorant.
1
@BHBalast this is exactly what he's doing wrong - using an inferior product, Devin, instead of building a rather trivial system on his own, out of open components, using the well known tools and methods I described. And not just using this inferior system, but also making long reaching conclusions about capacity of AI in general.
1
@yuriy5376 how exactly? I build these systems routinely, so I have all reasons to laugh at those who think Devin is a proper AI coding agent.
1