Comments by "Immudzen" (@Immudzen) on "Continuous Delivery" channel.

  1. What I generally find is that having junior developers test their code is what makes it better code. You have them test the code and they complain about how hard it is to test. I work with them to understand that if it is hard to test it is also hard to work with later and we break up the code into smaller functions, make the output of the function depend on the input where possible, simplify the interfaces, etc. and by the time they are done they end up with code that is much better and also tested. At this point I used tests to determine if code is good or bad. If you have never actually used the code you can't say if the design is good or bad and testing will force you to use it and actually confront what you have done. It is also why developers can't just hand off testing to someone else. They need to feel the pain of the code they have written and fix it to get better.
    169
  2. One of the issues I see is that treating people like this makes it really hard to recruit new people. If you are a top programmer why would you choose to work at twitter considering the amount of random abuse that can just suddenly land on you? There are lots of other places you can work, solve interesting problems, and will treat you much better.
    138
  3. My issue with low code systems is when they fail they fail in opaque ways that are very difficult to understand. I have deal with some low code machine learning systems and when they don't work they just don't work. You can't really look and see where the system went wrong or why. Instead your only real option is to build a new machine learning model yourself and figure out why the basic models don't work and fix them. In the end my experience with low code is that it doesn't save any time. You can get a prototype up quickly, it looks like it works in the beginning, but you find far too many failure points and there is no way to address them so you have to do all the work yourself anyways. Honestly, most of them look like scams designed to impress venture capitalists and hope they don't look behind the curtain.
    59
  4. I recently got my coworkers to do more working meetings. We use vscode and live share so that others can join in and we actually discuss issues while actually looking at the code and also do this for group programming for some problems. It is amazing how much faster you get stuff done when you have a modeler, scientist, and numerics person all working together at the same time looking at the code.
    55
  5. I would love seeing an episode on approval testing. I have been working on a system that was in a pretty bad state and we used a combination of approval testing and testing for some of the modules we could easily isolate as a first pass of tests. We then started refactoring the code but all the new code had tests written to specifically cover it. As we cleaned up the system the test coverage go better and the defect rates dropped. We also follow a boy scout type principle of leaving the code in a better state any time you touch it. So if you are working in a module and you see some bad code clean up a function or two, add some tests, etc. as part of your normal work. We went from merging about once per year into the master branch with numerous bugs to being able to do it several times per day and no failures in many months. Most branches now are very short lived before they merge back in. The situation is still not ideal with a CI/CD standpoint but it sure is a LOT better. I used many things I have learned from your channel in guiding this process.
    31
  6. Developers should unionize. The ONLY way you are going to be treated fairly is by working with others to force companies to treat you fairly.
    24
  7. Thanks for this video. I had looked at this "study" and we had just decided it didn't apply to us. All of my work is scientific software and we don't understand the problem until we start working on it. Sometimes we don't even know exactly what problem we need to solve before we seriously sit down and start working on it. Agile works quite well for us but this idea of knowing everything in advance and having some kind of a timetable is just utterly unrealistic.
    22
  8. One thing that SAFe has been good at is giving me cover to introduce more agile methods to our process. We are definitely not fully there because that is just too much stuff to change quickly but our processes keep getting better over time and our results continue to improve.
    21
  9. Even simple tasks can have massively wrong time estimates. I worked on something recently that I thought would take just a few hours and instead it took 3 days to do. Once I was actually building the system I noticed that the data I was working with had a number of mathematical properties that made the problem much harder to solve. Trying to do that analysis ahead of time to come up with an estimate would have been most of the time to solve the problem. At least in my case, nobody really cared that it took longer they just cared that it was as correct as we could reasonably make it.
    18
  10. I work with Python extensively for my job. We usually reserve comments to explain why something is done not what is done. Especially for scientific and engineering code it can really help to have a comment on where the math in a function comes from or why the math is there. We also heavily use the Python system and to further improve we often use dataclasses for input and output for functions with more complicated inputs and outputs. This is much better than a dictionary and it works as a contract for the function that other systems can rely on. For scientific and engineering code python fits extremely well because there are so many good libraries for Python to solve these types of problems. The libraries also tend to be highly optimized so you pay very little performance penalty for using Python. I especially love libraries like numba because you can compile Python code under certain circumstances to high performance code.
    15
  11. We have also been on the precipice of self-driving cars for more than 10 years and progress has nearly stopped. It was easy to get to about 95% but getting beyond that has proven to be insanely hard. These LLMs are already hitting the same point. In the hands of a skilled coder they can assist but they don't understand. Most of the code they generate is wrong to various degrees and when an LLM is questioned about the code it is pretty clear it has no understanding of what is being asked. Remember LLMs are probability models where they predict the next most likely word. They don't actually understand the question being asked. I think they will get better as assistants but won't be able to replace. There are some companies that already tried to do all the work with LLMs and no coders and they failed. I think they will continue to fail.
    13
  12. Over the last year my experience with TDD has been extremely positive. I stared working on a project that merged about once per year and had many bugs so changes where rare and it was difficult to work on. I started adding tests to the project and slowing fixed it. We are now able to merge multiple times per week and have not had anything major break in over half a year. We have had a few minor problems that where quickly sorted out and new tests added to catch them. I am not saying the code base is completely tested at this point but as we continue to add more tests development just keeps getter faster. Probably the biggest change is that by having to write tests for the code it has forced some of the code to be rewritten to be more functional and easy to test. This also had the side effect of making software composition easier to do and a GREAT deal of code was removed.
    12
  13. Approval testing has been extremely useful for me. We took code that had no testing and added approval tests to the system. Once we did that we refactored piece by piece and added unit tests as we broke things up. As we went we found real bugs and would then discuss fixing them and what impact that would have on usage of the system. We could then fix that bug, update the approval tests, and repeat the process. It worked well and I don't know of any other method that would have worked as well to deal with the spaghetti mess of code.
    10
  14. I work on HPC software and I remember a professor I had said you could use c,c++, or even python for the code but if you wrote the code in java he would throw you out the window. He was also not wrong on that when we looked at the numerical performance. Every version of BLAS I have encountered for Java is much slower than the ones available for c based languages. Often 10s to 100s of times slower. However, Python makes it easy to hand off all that heavy lifting to c and c++ while Java calling to a c based BLAS is also slow. So it is easy to get code working and correct in Python and then update the algorithm to be efficient and then push off the performance to lower level languages by calling existing high performance libraries. That is why we have things like numpy, tensorflow, and torch.
    9
  15. The comments I like the most are ones that tell me why and where something comes from. So if you are implementing a scientific algorithm writing down what paper you are implementing is REALLY important. Another common one is if you had to do external calculations for something. For example if you have done optimization and profiling to come up with some settings for a solver then it is really useful to write down why those solver settings are used and where the testing was done.
    9
  16. I have a strange experience with respect to Python performance. Most of the time in production systems I run into Python is faster than Java or C++ even though those languages are easily faster in microbenchmarks. I suspect the issue is that Python makes it easy and standard to hand off the harder computations to standard libraries like Pandas. Numpy, Scipy, Torch, etc. and also that on real systems data structures are so extremely important and Python makes that easier than Java or C++. I know for sure that I can write C++ code that is much faster than Python code but I also know that I had formal training in doing that. Most people don't have that training and without that they often struggle to make something as fast.
    8
  17. When working with older code the policy we use now is based on boy scout rules. Leave the code cleaner than you found it. So if you go work on some old code add some unit tests to the part you are working oon. if you see some really bad code take the opportunity to clean it up a little, if reasonable, and add tests for it. Take that function that is 500 lines long and break it up into many functions with a much clearer interface for each one and then test them. I have found this actually ends up working quite rapidly to fix and older code base.
    8
  18.  @nickjcresswell  I would agree with this. It seems some of the agile coaches I have run into just want to follow SAFe to the letter and that ends up frustrating people enormously. I love CI/CD, testing, KANBAN, and even the daily standups can be useful. The people we have on our team are all specialists. Usually there is only one person that can do any one item in the backlog so doing some kind of group backlog refinement is not very useful. Prioritizing tasks also tends to not be very useful since you don't do the next higher priority task instead you do the next highest priority task that you are capable of doing. We also have other teams who build things to make the highly technical team more effective. There is no real reason to plan more than a few weeks ahead at most because you don't truly know what is coming up. Instead a KANBAN approach and keeping people updated helps respond quickly. Code quality is really important because technical debt really slows down the process.
    8
  19. We have been using copilot to assist with code writing for a while now. I would also say that you are mostly correct about the long term problem. I find that the people that are pure programmers have a hard time using the AI to assist them effectively and that some of them don't seem to be getting better at writing code over time regardless of the AI. One of the big issues is the junior developers like to write giant functions. What I have found is that people that are engineers or scientists that also write code have been writing MUCH better code using copilot. I has made reviewing their code much better and for some reason they are also improving their quality of code. I don't know why but the engineers and scientists I work with follow the idea of writing much shorter functions that basically do one thing and make sure to test all code paths and their code has gotten better but the people that are pure programmers have not.
    7
  20. I often end up going back and forth between the code for a function and the test for it. I work with a lot of numerical libraries and sometimes I am not sure of what the output for something is that I need. For example what is the shape of an array that will result from a function call and how is the data in it arranged. I often find myself making a tiny function and a minimal test just so I can see the code do something and I can verify if it is called correctly. I then tend to flesh out the test a bit to cover what I think the function should accomplish. I then go to implement that in the function. Sometimes I find that the way I wrote the test has results in a very poor design in the function and that with a few changes to the test I can make the function cleaner. Sometimes I find out I need to split out some parts of the function into multiple functions. I find that tests really help with the design process.
    7
  21. The number one thing workers need is a union. All the other stuff is can be nice but pales in comparison. You don't have any leverage when negotiating with a multibillion dollar company. You only have leverage as a group.
    6
  22. @luke5100 I don't dispute it is a timesaver however at the same time it is also very stupid. It doesn't understand what it is doing and no amount of prompt writing changes that. There are quite a few research papers on this already. You are correct that the more narrowly the scope of something is the more likely it is to succeed but that also very clearly moves it into assistant and not something that can be used without programmers. I would say that on scientfic and engineering code if I have it write a 5-10 loc function it will usually get about 80% or so of it correct but you better know how to write that function yourself or you are never going to be able to correct it.
    6
  23.  @traveller23e  I normally push for only the flexibility actually required. I don't do interfaces for something until I have two use cases so I can make an interface to do that job. Otherwise you are just building an abstraction without a usage case and you don't know exactly where and what kind of flexibility you need. If you make it more flexible than is needed you also make your life and everyone else that uses the code much harder because the code will have a higher conceptual burden.
    6
  24.  @tylerarrigoni7700  Just a few hours ago I worked with a junior developer that was complaining about how hard testing some of the code was. I helped them factor it into more, smaller, methods and they got all the tests done about half an hour later and found some bugs in the process. I am a firm believer in testing because it forces you to make the code better.
    6
  25. For some problems I have even found it effective to have 5 or so people working together on the same piece of code at the same time. It allows me to get the various engineers and scientists together and once while writing particularly difficult pieces of code so we can make sure that code does what it needs to do with everyone involved right there to ask questions. Trying to make all the requirements specific enough to completely code them up without needing those people present turned out to take more time than just having them coding together.
    5
  26.  @blackjackjester  Twitter is a small social media company. More people use reddit than twitter. They are also insignificant compared to youtube, facebook, instagram, tiktok etc. Nothing that happens on it matters. He did not kick over any sandcastles for elite people. He has been treating engineers badly and you would know this if you talked to engineers that know people involved. Most engineers I know already refuse to work for any of Elon's other companies despite attempts to recruit them multiple times because they have a reputation of being very bad places to work.
    5
  27. I almost entirely agree with you. When I first started working I was spending about 95% of my time debugging code that I inherited. However, as I wrote more unit tests and cleaned up the code that has dropped to less than 5% of the time and most of the time is spent on actual development. Features are delivered faster and the bug rate has dropped to almost nothing. The ones area I don't entirely agree with is when doing software based on math modeling it can often be worth it to take some additional time to go over the math design and check all the assumptions, stability, ways of solving it, alternative approaches, etc. I have run into too many cases where someone just started implementing a model with some pretty bad consequences. I like iterative coding and delivery but I really want to at least try and make sure the math we are going to implement is the right math to solve the problem we really have.
    5
  28. We have been requiring it also for engineers and scientists and it has seriously improved the quality of their code. I even ran a tutoring session recently on how to use it more effectively. I focused on having them write functions with specific names that do one thing and use type hints in python for the arguments and return values with an expected function size of about 5-10 lines of code. The weird part is that for people that are just programmers they do worse with it. I still don't have a good idea for why.
    5
  29.  @SPeeSimon  I work on writing scientific models for making medicine. We have people that may know more about the kinetics, others that know more about the equipment, others that know more about the math needed, and others that are better at coding. Sometimes to solve part of the problem it really does work faster to get 4-5 people together and just code together so that all the issues can be addressed. Just having people write out the requirements for each of the parts and letting the programmer figure out how to satisfy all of them has not worked. Trying to just have meeting to discuss things is not specific enough to actually code. So a working coding meeting is more productive.
    4
  30. What we need is the same protections that other engineers have. If I have legal responsibility for the quality of the code I also need the legal authority to be able to do that. It is easy for management to create a system where you can't have good code. They don't support it in any way and they undermine any efforts to make better code and they actively refuse to learn about these issues so that they are not held accountable. They will just claim they didn't know their practices where making bad code. If you try to stand up for this you will be fired and it can be a lot harder to get another job after that. Mostly what I see is management wants the work done right now and they don't care how it is done so long as it gets done and you can't have additional resources.
    4
  31. I find that once you get beyond the really poor programmers I have a hard time rating one programming as better than another. Mostly because the knowledge ends up so specialized that it takes a group of people to do anything. I work on scientific and engineering software which involves a lot of complicated math and the skills needed to solve it, optimize it, improve the math, etc. are just pretty specialized and nobody can do it all by themselves. Who is the better programming between the person that wrote the solver and the person that numerically stabilized the math?
    4
  32. In my org we follow a boyscout principle is leaving the code better than when you started. So if you notice something in the code that needs to be cleaned up a bit and the time commitment is not large then go ahead and clean it up. Over time this has a huge impact on improving code quality.
    4
  33. This is why I require developers to write tests for the code they create. It forces them to use the code and experience the pain of using it. It helps them fix their design problem in order to make the code more testable.
    4
  34. I can completely understand this problem. I have seen some complicated and slower microservices stuff to handle things where it made no sense at all. Like basic HPC calculation stuff.
    4
  35. I work with a lot of science and engineering code where we are building simulations for making medicine. I have been able to get branches down to about a week on average but I found when trying to shorten it from that it caused more problems. I think part of the issue is that many of the people doing development are not primarily developers. They are engineers and scientists that need to get a model built and they need experienced support and someone that can look over their code before it gets merged in. The other part is that when building a model it is often hard to determine what correct is. The branches tend to get more group discussion on if the approach being used to solve the problem is the right approach.
    3
  36. I do not like having Java around at all. I much prefer Python. It is easier to call into c/c++ libraries or write parts of the program in a lower level language if needed. It also has a wider range of high performance libraries for math and science than Java does.
    3
  37. In SAFe training they mostly talked about using agile for everything except code. If they only targeted code they would not sell as many organizations on it.
    3
  38. I like that almost everything I do is HPC type stuff. We just use something like PBS to submit jobs and have them run.
    3
  39. I can certainly identify with this video. I have run into proprietary software that was REALLY hostile towards any kind of automation. It makes it so hard to do testing, integration, etc. I hope that we can get rid of some of it in the future because it makes getting everything else tested and running so much more difficult.
    3
  40.  @JanVerny  I do have oversight and people do expect results but they also know the problems are incredibly hard and want to be kept updated. Imagine you are asked with increasing the yield of something. Before you can figure out what to do you have to first figure out what is impacting the yield now, what models do you already have available, what things can you change to try and increase it. Once you know that then you can set about actually solving a problem.
    3
  41. Thank you for making this video. I am going to share it with others because I have been trying to make the same point. Chatgpt and most of these other AI systems have no concept of truth and it is not something that is easy to add. They can be useful and they can assist in programming but you have to watch them very carefully because they will be wrong. I have done some tests with generating Python code and it does a decent job of generating code to the point where it can do a lot of the basic work and you can focus on the overall solution.
    3
  42. I worked on something recently and we thought it would take about 5 days to get it done. It ended up taking about two months. Fixing one problem uncovered a lot of other issues related to complicated math in the simulation that took quite a while to track down and fix. There was no way to see that problem ahead of time because nobody had any idea the problem was there. At some point it seemed the Agile process we were taught just started to break down for this. The code ran, the tests passed, but the tests also ended up being wrong for some of the same complicated math reasons. For scientific and simulation software at some point I have found you just need a few people that work together on an issue and get rid of all the worrying about sprints, tickets, scrum, predictions on when it will be done etc. stuff. There was no point in adding more people to the team to speed things up because there was really not much they could do to help. It was better to have them work on other projects. Especially when a combination of programming and graduate level math skills where needed.
    3
  43. Our team has been working the exact opposite way we build the science/engineering models and make them work, then define a programming interface for them with dataclasses, then start designing some of the visualizations and then we work with the UI team to create a UI for it. Having the UI team work first just wouldn't work because they would have no idea what features the UI would need to control the model and until we build the model we don't know what it will need either.
    3
  44. I am a chemical engineer and the agile approach does seem like an engineering approach to making software. I just see it as still in an early stage of development. Most of our practices and calculations we use for engineering where created through trial and error iteration and improvement and anything that is cutting edge still tends to get developed that way. Sure we don't do that for bridges anymore but we also learned how to build bridges a long time ago and have mostly been slowly refining them.
    3
  45. We have been doing some group coding at work recently using the vscode live share feature. Sometimes it is one person doing the coding while the others are giving input on how to do something but others times we split up to tackle a problem together. For instance if someone is working on a feature and we realize a new simple function is needed something else that would make it easier to implement it then someone will split off to quickly code up and test that function. Especially when dealing with more complicated math problems this has really helped.
    3
  46. Crowdstrike seems to have lower quality standards that my group does writing simulation software. What Microsoft needs to do is much more heavily restrict kernel space and stuff like crowdstrike should run in user space.
    3
  47.  @kiyasuihito  I work in a highly regulated field. The focus is on doing stuff right and making sure it works reliably. Simulations must be traceable and reproducible. I teach junior devs I would prefer they take longer and make sure it is right and tested.
    3
  48. The EU has already placed restrictions on AI. If you AI directly impacts a human the decisions made by the AI must be explainable and accountable. You can't just hand off a job to an AI to hire or fire workers and absolve yourself of responsibility. They have actually done quite a good job on these laws because they specifically left out things like regression networks and surrogate modeling where those things don't impact humans. So if I design a neural network to approximate a partial differential equation I am clear.
    2
  49.  @mastermati773  We have been using it to help get newer people up to speed and spread better coding practices. I have also used it to teach people how to get to a better state of code since often it is hard to understand if you only see the start and end point. For instance to refactor some code first we pulled some methods out and made the output only depend on the input. We did this with a bunch of method that will depend on an interface instead of being part of a complicated object. The next step is to refactor the pieces into a designed result object and then theyw ill use that as in interface. if you only see the end result you won't understand how to do that yourself.
    2
  50. I think my normal approach to design is to look at what people need to accomplish, what they would like it to accomplish, and what features they would like to add to it over the next year or so. From that I try to break it down into some high level ideas based on the math that has to sit behind those features. I look at what math is in common and for that part I try to figure out what I will actually need because often that ends up determining how other parts of the system will need to function and it also doesn't take more than a few hours usually. After that about a day is spent on trying to break the system down into some high level modular components and we try to work through the design quickly to see if it can cover all the usage cases. Once we are to that point we can start coding. In general the classes all have to be small, we try to use pure functions where possible, and use dataclasses for interfaces. This makes writing the testing much easier.
    2