Comments by "Andrea Laforgia" (@andrealaforgia) on "Continuous Delivery" channel.

  1.  @KaratePath  No, your view is NOT the mainstream view. Go read one of the books already mentioned. You couldn’t be more wrong. Stop embarrassing yourself.
    1
  2. Code reviews are highly overrated. Have your team work as a real team via pair/mob programming and you won't need code reviewa and PRs. PRs were not meant to be used by teams of colocated, trusted collaborators.
    1
  3.  @michaelrstover  I don't throw out anyone. I try to make processes inclusive for everyone. Being capable is not enough, though. You must also be able to collaborate. Collaboration is a fundamental requirement in software engineering.
    1
  4.  @carveratutube  Before dismissing what Dave says, I encourage you to try to apply what he suggests. What Dave is suggesting is the best way of developing software today: TDD, trunk-based development + feature toggles. And it's not just his opinion. That companies perform much better by adopting those techniques is supported by facts (see the rationale behind DORA surveys and the book "Accelerate"). The idea that features must not be "not really nice or complex" because someone is suggesting working in small batches using TDD and committing often is fundamentally flawed. That is THE way you should be developing software today, especially in a large organization, where many developers participate in working on "nice and complex" features. Committing WIP in master is absolutely fine if you use feature toggles. TDD is not essential for CI, but it's a big enabler of CI, therefore no reason for not using it. There are billion reasons why you should be using TDD that are not striclty related to CI, so I suggest that you watch a video about it. There's plenty on YT.
    1
  5.  @BigCarso  No, that's exactly what is is: 10-15 mins max.
    1
  6. I can totally confute your point about CD being not realistic given that I have worked for many companies that implement it, and I know many others who do. Continuous Delivery is a well-established practice in many large companies. What is "daily CD"? Continuous Delivery does not dictate that anything should happen "daily". I suggest that you read Dave's book "Continuous Delivery". The book "Accelerate" is another good resource. Merging to trunk every day is another well-established practice that has been around for many years, and it's adopted by companies like Google or Facebook (plus some I've worked for), in a scaled version, where thousands of developers collaborate on billion lines of code. >What if I need to work on a module that takes me 40 hours of coding? Then Test-Code-Refactor-Push, Test-Code-Refactor-Push, Test-Code-Refactor-Push, for 40 hours. This is not theory. It's reality that's been going on for many years.
    1
  7. PRs are not strictly necessary for automation. The automated steps you attach to a PR can live in the build and deployment pipelines. What stops bad code from being merged is automated tests and code review, not Pull Requests. Code reviews can be performed in many different ways, for example continuously, when adopting social programming approaches. The more you dig down, the more you discover that no, there isn't plenty of reasons to use them. PRs were never meant to be used by durable product teams.
    1
  8.  @nextlifeonearth  PRs are not lights.
    1
  9. With CI / trunk-based development, that’s totally doable via feature toggles. You can logically separate those features within the same code base.
    1
  10. The aim should be to transform repetitive, tedious, deterministic manual tests into automated tests and reserve human brain capacity for what it can do best: exploratory testing. Not sure what you mean by "dirty branches". If you mean potential throw-away, experimental work, you can still have those branches, as long as they don't participate in your continuous development strategy. The problem with branches is always when you try to re-integrate them into the mainline of development after days or even weeks have passed by and main/master had diverged significantly. With regard to code reviews, you can have those with trunk-based development, either with short-lived branches or with something even better: pair/mob programming.
    1
  11. That's great news!
    1
  12. Yeah but this idea that a team is a group of people where there are untrusted members supervised by trusted member is really bad. It encourages ivory towers. A team should be a group of people working together: get the senior one join the junior one so the latter soon become trusted. Teams are there to unite people, not to segregate them.
    1
  13. Scenario 1: You can indeed have only 1 function in your code that will implement one behaviour or the other based on the version (for example, you could use the Strategy pattern there). No need to keep your code physically separated when you can have logical separation.

You don’t need to duplicate any other version of the code that the two libraries have in common. No need to fix bugs in that part of the code multiple times.

In general, you can use feature toggles to logically separate “branches” of logic in the same code base. It works really well. 
In some languages (e.g. Java + Spring Boot), leveraging dependency injection, there are some neat ways of injecting behaviours based on conditions that make the above very easy to do (look at how ff4j works, for example). 

Scenario 2: the longer code is parked on a feature branch, the higher the risk of diversion from master and merge conflicts. The best way to develop code we know so far is to continuously integrate into a shared mainline multiple times a day (e.g. Continuous Integration). See above: using feature toggles, you can delay releasing a specific feature as long as you want.

The big misunderstanding here is that each push to master will release to production. One thing is pushing to master, another thing is releasing to production. You can have that with Continuous Deployment. With Continuous Delivery, your code is always releasable but not always released. The key point here is that you must be able to release whenever you want with no fear.

 The reason why trunk-based development seems dangerous to you is because you’ve never used it. Companies like Google or Facebook (but I could also mention some smaller companies I’ve worked for) use scaled trunk-based development on extremely large projects with thousand developers on them. 
 What you describe in your last paragraph is not continuous integration.

Continuous Integration is defined as: “a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.By integrating regularly, you can detect errors quickly, and locate them more easily.” (ThoughtWorks)
    1
  14.  @markar34  Of course I am aware of semver, it's a de facto standard. Semver is irrelevant though. You can use whatever versioning mechanism. I totally understand why you need to keep two separate versions of your library. That's a common scenario in any software, especially when you deal with "dinosaur" companies that move at a very slow pace and are reluctant to upgrade even when they would need to (e.g. big financial institutions). I don't think you understood my comment though. I am not questioning the logical separation of your versions; I am questioning the physical separation. Believe it or not, you don't need long-lived branches to maintain that separation. You can have a logical separation of versions within the same single branch. You can deploy from the shared mainline using different configurations for each client and each version. As I said before, if you have hundred or thousand clients, you cannot possibly afford long-lived branches for all of them. The ability to keep developing your software would grind to a halt. I mentioned dependency injection with regard to how to use feature toggles properly with some languages/frameworks, just to avoid the mess in the code that you were mentioning. Your code can be architected in a way such that the usage of feature toggles has a very minimal impact on it. Unfortunately, it would take time and a real example to show you how that works. The ff4j library is very powerful from this point of view, leveraging proxies and DI. >The fact that we share say, the develop branch, and our CI builds it, and runs automated tests, that's enough to say it's continuous integration. I'm not sure what is your definition of continuous integration. Nope. Not if you use long-lived branches. Continuous Integration, as defined by ThoughtWorks, is: "Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. By integrating regularly, you can detect errors quickly, and locate them more easily." If your development happens on long-lived feature branches, it's not CI. CI doesn't just mean having a CI build. It's a development practice.
    1
  15.  @markar34  The definition of CI is not subject to opinions. It's clear. The word "repository" refers to the shared mainline (master/main/trunk/etc..). I can mention many other definitions that all lead to the same thing. For example, Google Martin Fowler's one. Continuous Integration wouldn't be such if there were multiple branches involved. It would just be a contradiction. The important bit is *continuous*. >Integration can happen in any other branches. I don't see why you can't see that as integration. I said that is not *continuous integration*, not that is not some form of integration. As Altassian correctly says: "In the trunk-based development model, there is a repository with a steady stream of commits flowing into the main branch. [...]trunk-based development is a requirement of continuous integration. With continuous integration, developers perform trunk-based development in conjunction with automated tests that run after each committee to a trunk. This ensures the project works at all times.""
    1
  16.  @markar34  >so what happens if you made a commit on master branch that fails the build? You validate the change before committing to master. Dude, go read something about trunk-based development. >With branching, you can prevent that Again: branching is not Continuous Integration. >What about code reviews? You can't force pair programming. You don't need to force any practice when the benefits are obvious. Pair programming or mob programming are excellent ways of working that emerge naturally in the right environments, with good engineers and the right work culture. They imply code reviews. Continuous code reviews are the best way to review code.
    1
  17. ​ @markar34  You did not understand my example. I wasn't talking about code conflicts which can be easily detected by the code hosting system; I was talking about *feature conflicts*. The feature developed by B would clash with the feature developed by A and tests would fail in master even though they pass on each individual branch. It's totally possible and it happens with a branching model like the one you propose. I never said I can make any organization adopt anything immediately. Pair programming and mob programming face a lot of resistance because of the way many companies normally work, which is absolutely antithetical to CI and CD. If an environment allows to adopt those techniques, the benefits are immediately visible.
    1
  18. Hummm... that “coordination in daily scrums” makes me curious. First because that’s not what daily scrums are for and secondly because it makes me suspect that there is some conversation along the lines of “you go first, I go next, etc...”, which means sequential work, which means defeating continuous integration. Three days can be a huge time if your team works at a very fast pace. There are always ways to make integration/acceptance testing faster.
    1
  19.  @aurelienrb  the benefit is that, in the way I suggest, you are doing proper Continuous Integration. As said many times, rebasing master daily doesn't solve any problem because 1) if no one has ever merged anything in the meantime, it's completely useless 2) if only big chunks of work are merged into master, rebasing will cause big merge conflicts anyway 3) if someone has significantly refactored parts of the code you have in your branch, they are completely invalidating your code 4) no one is receiving your changes on that branch for a very long time hence other huge merge conflicts will arise. I could continue. The approach to follow is continuous integration: work should be granularized and integrated into master multiple times a day. Do not ever think that conflicts are always easily resolvable: if you don't know anything about other people's changes, you're gonna mess up sooner or later. Do I consider merging stuff into master and then removing it a problem? Absolutely not. Master is a workbench just like every other branch. With feature toggles, you can do whatever you want.
    1
  20. >In real life you often have to undo days of work, refactor days of code, incorporate feedback from other stakeholders and so on. People keep mentioning this "real life" like they are the only ones living in the real world and people like Dave Farley come from Mars. Please appreciate the fact that we all belong to the same reality. It's just that we have different work experience. The way of working that Dave is advocating is used effectively by many companies, some of which very large, with extremely large projects, and a huge number of developers. >people will start using your unfinished code and start editing files you are working on. Long-lived feature branches only make things worse in that case. TBD is much better, especially when heavy refactoring is involved. Imagine what would happen if we work in the same team, I branch from master, you branch from master, and start refactoring a number of classes you are working on, maybe removing some code that you are changing. I finish and merge into master. At the moment you need to merge, you will end up in merge hell. It will be impossible for you to understand how to fix the merge conflicts cause you don't know my changes. You will need me to help out, which basically means having you redo the work of days on a completely changed codebase.
    1
  21. That's a non-sequitur. Continuously integrating code and releasing features are two different concepts. You can integrate your code multiple times a day even though your feature will be released in 3 months time. Code landing into master/main doesn't necessarily mean that the feature implemented by that code will be released to production in the hands of customers straight away. You can adopt feature toggles to hide any incomplete or non-yet-validated changes. If you can use feature branches, then you can use a single branch with feature toggles. It's exactly the same thing. In the former case, the separation of your features is physical; in the latter case it's logical.
    1
  22.  @ShroySha  Ah the good ol' "I am not gonna give you any evidence, look for it". Very solid argument dude :)
    1
  23. Wow... where do you work, in a rat cage? :D Mob programming works in environments that embrace mob programming :)
    1
  24.  @Thorax232  Jesus Christ... the way you guys trivialize valuable practices is incredible: "3 people chirping in my ear". It must be such a pleasure working with you.
    1
  25.  @cyrusol  It's fine. You can disagree. You will agree, however, on the fact that what is important for you might not be important for other teams. Git logs are there to track a chronological sequence of commits to a branch. That's it. The concept of "clean" associated to that list is always subjective. Many teams might not care at all about keeping that list clean. They might have a policy of not reverting any commits and just rolling forward. They might be inserting an id in each commit message that easily correlated all commits for a single feature. And so on. They just don't need any concept of "clean".
    1
  26. I totally disagree. Tests are indeed a way of documenting the code, but they have to be written the right way. Keep your test methods short, with just three phases: given/when/then or arrange/act/assert sections, and what the code does will be obvious. When reviewing someone else's code, the first place to go to is tests. When someone asks me about a certain behaviour of the system is correct, I go look at the tests, not the code, to see what it does. You write down intent by making it clear what the expectations are.
    1
  27.  @RFalhar  >I would be interested in seeing Allan talk about research on surrogacy, instead of using his personal opinion and experiences, which can easily be biased. Why? We all refer to our professional experience all the time. It's the data we have. When our data reflect the research on surrogacy, I don't see the problem.
    1
  28.  @RFalhar  Research around metrics and surrogacy prove him right.
    1
  29. ​ @tonytheprogrammer1716  >The abuse generally comes about due to some ignorance regarding what is the important underlying phenomena we are trying to observe. That's a trivialization of the problem. It's not just ignorance. Weaponization of metrics happens all the time in business, even in good faith. It's just a dangerous consequence of using a dangerous tool. Are metrics useful? Absolutely yes. Are they easily weaponizable? Absolutely yes. Are they weaponized all the time? Absolutely yes.
    1
  30. Automated tests.
    1
  31.  @silentwater79  The fact that someone else, other than the developer, has to test the code is not true in general. If you adopt a test-driven/behaviour-driven approach, that gives you a black-box perspective. The work of a human QA is not necessarily required (I know many large projects that don’t use humans for testing other than exploratory). In any case, no, even if a feature required human validation and that would take time, you could still merge your code into master and release it to production, as long as feature toggles were used to hide it. One thing is the ability to continuously integrate the developer’s code into a shared mainline multiple times a day, one thing is the validation of the features implemented for releasing to production. Two different concepts.
    1
  32.  @jeremykothe2847  I think that short-lived feature branches can be even worse than long-lived ones, in some environments. The shorter the life of your branch, the more frequent you need reviews, so reviewers in your team will be context-switching all the time. The obvious alternative is that they will take their time to review your changes, totally invalidating the short life of your branch and impeding your progress. The only feasible option is really getting people work together (pair/mob programming) and do continuous review. If you want to adopt trunk-based development, I suspect you'll need to carry out a lot of groundwork to shorten your feedback cycle. Having only nightly tests is not acceptable with tbd. At least you should have a suite of smoke tests that a dev can run before pushing their changes.
    1
  33.  @jeremykothe2847  >the only technical difference is where the tests run? The difference is Continuous Integration.
    1
  34.  @jeremykothe2847  your local 'master' branch is just a technicality dictated by git being a DVCS. The difference is that your local master is constantly sync-ed with the remote master by pushing your changes frequently during the day. The branch associated to a PR isn't.
    1
  35.  @jeremykothe2847  PR tests run only when you create the PR, the branch associated with the PR is not continuously kept in sync with master. Please read the basics about continuous integration.
    1
  36. Test coverage thresholds make little sense. Remember Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure"
    1
  37. I think you'd be interested in this video by Jez Humble (who co-wrote Continuous Delivery with Dave Farley), who explains how they adopted continuous integration at HP for the firmware of their LaserJets: https://www.youtube.com/watch?v=mBUJ-fg4EKA&t=1395
    1
  38. >to be able to think through the problem steadily and come up with an accurate solution, especially for more complicated problems, and I don't think its possible to do this deep thinking whilst in a mob. Yes, it is possible, but it's a skill that needs to be learned and perfected. Problems must be solved collectively, not alone. The solution is always better when multiple brains share ideas/approaches. You can still spend some time in isolation, if you need, but check back with the team frequently. The risk with thinking alone for too long is that we don't find flaws in our reasoning early, and may go off a tangent.
    1
  39.  @donka86  my question is why set up that cannot work together due to different time zones?
    1
  40.  @donka86  Distributed teams are common but you were mentioning distributed teams *across multiple timezones*. Placing people from distant timezones in the same team is a recipe for disaster. distributed teams across multiple timezones are notoriously difficult to manage. I would never set up a team with someone in the US, someone else in UK and someone else in India, for example, a practice that I've seen in a few places. You can totally pair/mob program remotely, via the various tools available (Code With Me, LiveCode, etc...). If your work hours largely overlap those of your colleagues, you can pair program.
    1
  41. I’ve been applying that idea for a very long time now. It is a very natural thing to do once you have some experience. I tend never to mix behavioural changes to structural changes. If I noticed, for example, a typo, or I spot a function that needs implementation (not interface) refactoring, I do that in a following PR (to be done soon after the one with behavioural changes I’m working on). PRs, for me, must have a very limited scope.
    1
  42. Mob programming has been my favorite way of working for quite a long time now. Whenever I used it, we ended up being way more productive than working independently, with the nice side effect of sharing knowledge across the team, increasing confidence in answering questions about the system, increasing job satisfaction, reducing burnout (deadline was 5pm — no work after that time). I remember joining a team where 4 out of 5 people were new. There was only 1 person who'd been in the company for a few years, who knew the system well and could answer questions. He ended up spending most of his day assisting others. I proposed moving to a mob programming model. It started as an experiment but it became our standard way of working. For me, the crucial moment was when, jokingly, someone said, well "C. is the expert in our team!", to which C. smiled and replied "not anymore!!". Everybody had become an expert. I think there is an upper limit, though, and that's 4-5 people (which, to me, should be the size of a team). I'm aware, however, of a few contexts where large mobs worked really well. Austin Chadwick and Chirs Lucian (Mob Mentality Show, here on YouTube and LinkedIn) interviewed an engineer from Tesla some time ago, if you are interested.
    1
  43. There is a well-known phenomenon called "surrogation". Psychologists have been writing about it for years.
    1
  44.  @miletacekovic  >Of course, you can delay branch creation till bug is found in it, and create a branch from tag when the bug is found. But once branch is created to fix a bug, you need CI pipeline attached to it. It's not a CI pipeline, it's a build pipeline. It's different. CI means something specific: Continuous Integration. You don't do Continuous Integration on the releasae branches, you keep them for hotfixes. In general, however, keeping a release branch for every customer, assuming that you have hundreds of customers, is suicidal, a good recipe for disaster. You cannot really expect to have to hotfix a bug on hundreds of branches. You will need to make those customers converge into a new release at some point. >Verifying if a bug fix did not break anything on developer workstation is little scary for medium to large systems . What developer workstation? Who has ever talking about developer workstations? Developers' workstations are temporary workbenchs. CI is about integrating developers' work into a shared mainline multiple times a day. Tests run on the mainline.
    1
  45.  @miletacekovic  You are not doing continuous integration on the release branch. Therefore you cannot call the build for that release branch a "CI pipeline". You are fixing bugs on that release branch, you are not continuously integrating new development. That bug-fixing activity causes frustration among your developers, rest assured, given that they have to apply the same fixes in multiple places, with all the problems that that practice entails. If you have several bugs, discovered for multiple client's version, you need to multiply that bug-fixing activity for all those branches, increasing frustration and fear of mistakes. The idea that you can keep release branches open indefinitely is not a sustainable model. It doesn't really work anywhere. You will need, at some point, to make your release branch converge into master again or you are doomed to eternal sadness. Stop calling it "CI pipeline". CI happens ONLY on the shared mainline of development, nowhere else. You are talking about separate builds that happen on the CI server. It's not a "CI pipeline".
    1
  46.  @nzalex1  you are vaguely referring to "some situations where it might be better". There is no evidence from the industry that having multiple sources of truth is convenient at times. Multiple sources for truth cannot exist because they would only create confusion (it's ingrained in the concept of "source of truth" that thee should be only one). On the othe hand there is a lot of evidence from the industry that continuous integration/trunk-based development is beneficial to increase the throughput of software teams. I am not creating any "limitation". Feature branching is a practice incompatible with continuous integration. That's it.
    1
  47. Every developer working on a different part of the system is not a team. It's a group of individuals each working in their own silo.
    1
  48. ​ @Oliver Thane No, B would not be able to see any of A's changes because A has not merged any changes into master yet. The point of continuous integration is in that "continuous". If you and I branch from master today and keep working for the next three days on our branches, rebasing frequently for the next three days would bring no benefits to any of us. Not sure why we should resign with "it's just life" when there is a better way of working, which is actually much more practical and solves the integration problem?
    1
  49.  @INFOXlive  I’m amused by your comment 😄. The first principle in the Agile Manifesto states: “our highest priority is to satisfy our customers via early and continuous delivery of valuable software”. The fact that software must be delivered continuously is specifically stated in that foundational principle of Agile, so I don’t understand where you see the falsity in my statement. The Agile Manifesto contains a set of values and principles that foster agility: the ability to respond to change quickly. I was asking you a specific question: do you know a better way of working? That’s the opposite of being dogmatic. I’m all ears. I’m open to learning a better way from you. How do you think software should be engineered if not following the points that Dave is mentioning in this video? You say that they are meaningless but you offer no alternative. What do you think is an advisable way to engineer software so to meet targets more quickly, effectively and with higher quality standards?
    1
  50.  @DanielSanchez_dsanc89  mob programming.
    1