Comments by "Daniel Sandberg" (@ddanielsandberg) on "Git Flow Is A Bad Idea" video.

The problem with getting stuck in FB/PR thinking (in an enterprise setting) is that it optimizes for all developers working alone on "tasks" instead of as a team. And that there need to be "good developers" keeping tabs on "bad developers" and that many commits would eat all of the "good developers" time. It optimizes he entire organization for "the lowest common denominator" and holds everyone back; just in case there happens to be a "bad developer" on the team. It's all just band-aids on top of low-trust organizations. The job of a senior developer is to create more senior developers. That means most of that persons time will be spent NOT CODING ALONE, but instead coaching, pair-programming, teaching, sharing knowledge, etc. I would recommend to watch Dave's other video called "A Guide To Managing Technical Teams".
2
@thenewms5333 You expressed the need to "mitigate bad code coming from bad developers" and I think that is a real problem. Dave calls it "programming by remote control". My point was not that there are not bad developers. My point was that letting "the presence of bad developers" determine ones processes is what leads the organization to optimize for the lowest common denominator. By lowest I mean *bad* - force everyone to adhere to a bunch of processes based on fear and low trust. It becomes a crutch that hinders growth, much like helicopter parents are setting their kids up for failure in real life.
2
I must agree. Every single "git tutorial" starts with "branching" and then too many developers think that that is the goal - to follow some branching strategy perfectly - missing the point of version control and software to begin with. The number of times I've been told I'm unprofessional because I avoid FB/PR and code-reviews and prefer CI, TDD, pair-programming and good tooling that tells me if I'm particularly dumb today. I can't even change jobs anymore because "FB/PR/CR" is mandatory everywhere and I've even been to interviews where they never heard the term "Continuous Integration". I have seen similar effects with for example agile; "we are doing burn-up, burn-down, user stories, story points, sprints and planning poker; therefore we are agile" - and then people end up fighting over "who is doing story points right". They don't understand agile. Oh, and don't get me started on "DevOps Engineer" - if your company have DevOps Engineers and DevOps Teams you don't understand DevOps either, but I digress. I had another experience where the developers wanted to go nuts with versioning; alpha, beta, rc1, rc2, final, gold and when I asked why the answer was basically "it's what the big companies are doing" - so essentially they wanted a complex release process because it "sounded professional", not understanding that all that complexity is most likely a symptom of bad software engineering. Complexaholics all around. Bring the pain! Now I need my benzos again.
2
CI is a human practice. An "automated build" is just one of the steps within CI.
2
By following the practice of CI. 1. Pull and local build (make sure it works before starting new work) 2. Do work and local build (does my stuff work) 3. Pull and local build (is it broken after pull? me? them?) 4. Push 5. Watch the build in the build-system, don't walk away 6. If it breaks, you have 10 minutes to fix it (and promise to not break it again) 7. If you can't easily fix it, revert 8. If you skip some steps, broke it and went home, you buy the team pizza for lunch the next day Practices, principles and discipline.
2
@liquidcode1704 This is the exact situation we want to avoid. If you need to work this way it usually means that things are broken all the time, that testing is done after in some kind test phase, probably also have a "stabilization sprint" and work is thrown over to wall to some "deployment/ops team". By definition you cannot call it CI/CD if you have dev branches, test branches and merges once per week.
2
@OzoneGrif Why is it unacceptable? I don't mean to sound antagonistic but you need to think outside the box of release-trains, Gantt charts, milestone-driven release schedules, and manual test-and-stabilization phases. First of all, all code shall be tested whether they are "feature complete or not", and then you have an evolution: 1. Early in development the features may be excluded/included from the build by using build profiles. This allows local development and testing as the team fleshes out the basics of the feature. 2. Later on features may be included but be disabled by default and enabled by a deployment flag in different environments. Great for feedback and early testing. See "branch-by-abstraction pattern" for one example of implementation details. 3. Once we reach the "this is a nice feature, polish it"-stage we may choose to convert the flag to a runtime property or keep it as a deployment flag. Context and technology matters. 4. When we finally are ready to release the feature to the world we can turn it into a user preference (Opt-in/beta users for the new feature), or even using Geo-location or target groups of who will see it. 5. This allows the business to then decide if the feature shall be available for all users and we can remove all the flags and gunk around it. Or we may decide that it's an "enterprise feature" and customers have to pay for the feature to be enabled. The point is - you can have a lot of unfinished changes in-flight, in a single repo/branch and still always be able to deploy a hotfix. This is kind of how facebook implemented their chat-system. It was running (by bots in your browser) in production, in secret, behind users timelines, and only facebook employees could actually use it. It had been running in production for over six months before normal users could access it and then they incrementally rolled it out to users - without needing to build/deploy/release a special version.
2
1. Continuous Delivery is not Continuous Deployment. 2. Merge to main is not a deployment to prod. 3. If you want a pair of eyes on every change - then pair program.
2
Frequent rebasing from master/main does not make any difference if everyone's changes lives in their own branch. You will end up with lots of changes that are unintegrated for some period of time since there is nothing new to rebase with. Once a complete feature is merged into main (with possibly 100 files and 1000's of lines changed) every branch risks conflicts or breakage, especially if there has been any refactoring done (which there should be). And it usually ends up with a culture of fear - "only change the minimum, don't refactor, don't improve the design...". Redefining "whatever we're doing now to mean CI/CD" does not make it so.
1
With CI/CD we don't attach meaning to branches to keep track of the state of the system, so no!
1
Wall of text, partially off-topic warning! Let's take this from the start - releases In the olden days we used to have something called RTM (release to manufacturing) - basically meaning (after months of testing and polishing) sending the software to a factory that put it on floppies, CD-ROMs, etc. and shipped it out to stores in pretty boxes. There was a really high cost of delivering and patching software back then. Then came tools like Maven that took the old concept of "cutting a release" and made it mean "make a final (re)build and stamp a version number and publish it for deployment". Later CI/CD/DevOps came into the picture and things started to change; we came up with terms like "separate deployment from release". What is meant by that is that deployment comes before release. Enabling new functionality or changing old becomes some kind of configuration change instead of a "all hand on deck deployment 3 AM" thing. This also enables A/B-testing, user opt-in and deploying many many small changes all the time - thus reducing the blast-radius if something goes wrong as well as knowing that every change actually works in real life. This doesn't mean that every change gets a deployment, it's just that deployments becomes a choice by need instead of a planned date/deadline. How does it (ideally) works with CI/CD? With CI/CD we instead try to subscribe to the (impossible) ideal of Toyota "one-piece flow". Instead of keeping track of the state of the software by using branches/merges (with all the management overhead) and thinking in terms of "master represents production" or "let's build a release and a release branch" (a bit like the previously mentioned "cutting a release") - we commit to main, we build it once , we record the version/tag/commit/buildnumber/digest of that build; we test and validate it, and then pass that immutable piece from environment to environment, from validation to deployment and finally release. It's a survival of the fittest scenario to try to prove that a build does not meet the criteria for deployment. Extrapolating and deeper context As with everything there are contexts and compromises. But if we default to use branches for change control, separate departments for frontend, backend and testing, huge ops-teams getting software thrown over the wall from developers (we made a release, your turn) because it feels easier and it worked 20 year ago we are not making any progress or improvements. According to some of the "old IT/tech people" the amount of programmers doubles every 5 years. About 10 years ago everyone went nuts with FB/PR/GitFlow. So we can extrapolate that 75% of all programmers have never done anything but, and so have no idea about Continuous Integration. I'm really passionate about this because I see the same thing repeating in every org I've been in. As long as our thinking doesn't change it doesn't matter how much technology, tools, languages, frameworks and processes we throw at software development - nothing changes.
1
A pipeline does not replace the steps and tasks in a build script (npm, gulp, gradle, maven, etc). This is a common mistake I see, where teams replaces everything they had in their scripts with a pipeline YAML that can only run on some build management system. When practicing CI you should be able to run the majority of the build steps including unit test, some component and functional testing locally. Whole system acceptance tests is difficult and often too slow to run locally. It also serves as feedback; if it is difficult to run, build, test, lint, you probably need to look into the structure and design of both tests and code. If it is very hard or slow to build/test locally you may need to look into the the technology choices made (like frameworks, etc). It is an optimization technique where you should be able to compile, package and run 10000s of tests locally in minutes. Enough that it gives the developers a high degree of confidence that it will work before pushing. The build management system will of course run the commit build as well, plus a bunch of slow wider system tests, performance, etc. This also has the added benefit that if GitHub is down (like it is now as I write this) you can have a "break glass" process and build an emergency patch on a (monitored) local computer and push that to production. While the build system runs the commit build you are not allowed to go away, lunch, or home. If it breaks you have 10 minutes to fix it or revert the change. If it's still broken the team swarm around the problem and get it back to green.
1
 @rafaeltab What do you mean by "not ready for production" and "not able to adhere to once per day"? Why would that be true and why wouldn't you be able to have those things?
1
@rafaeltab This is why I asked my question the way I did. This is a common argument. The problem is that the assumption is that "the code must be broken to make the change". The cardinal rule in CI/CD (and TBD) is that you should merge to main many times per day and you are never allowed to break the code (mistakes do happen but should be rare). You must find a way of working where you can introduce changes to the code without breaking it. There are many strategies and patterns out there, but good unit tests are almost a requirement. More process oriented strategies like compile/build options, branch by abstraction, config flags, feature toggles, dark launching, etc. are common. FB enthusiasts views the ability to isolate changes and "screw around" in a branch as something good. CI enthusiasts views this isolation as a flaw in the system (process, engineering, design, architecture, culture).
1
I assume that you mean to work by a pre-defined release plan where you have Gantt-chart like planning that "next month it's A, B", the month after that it's C and D, and by summer it will be E and F? In CI/TBD/CD we do things differently. We associate the readiness of the a release with a version instead of associate it with where it lives in version control. We also try to separate "code fixes, improvements, etc" from "feature releases", meaning that we can get any change out at any time, in hours, irregardless of plan, without hot-fix branches and special processes. We then promote versions between environments. Having separate testing teams, hand-offs, approvals, etc is pure death to CI/CD. Even if the case is that we plan to release features at specific intervals we can handle that using feature flags at multiple levels. In CI/TBD/CD we don't treat a release as a deployment in the same way. Instead deployments comes *before* release, meaning that unfinished features are deployed, but disabled/hidden until ready. This is how Facebook release their chat way back in pre-historic times. They had "javascript bots" running behind the scenes on a subset of users timelines; testing, trying, measuring and logging stability and security. Later on it was rolled out to a subset of users for trial, that subset then grew and after a few iterations (and performance fixes) it was rolled out to 1 billion users. Note that they did not do a "all hands on deck and deploy the universe and hope it can handle 1 billion users" on release day - it had already been running in production for over 6 months.
1
It's more complicated than that. Humans tend to gravitate towards easy solutions to complex problems, while the best solutions tend to be counter-intuitive. That "lure of easy" often leads to local optimization, isolation and tribalism. Sometimes it goes so far that the system becomes optimized for developers sitting alone in a corner, with headphones, typing code and occasionally popping their heads up exclaiming "I'm done, sent it to QA" and then keep typing. Meanwhile managers starts measuring "performance" based on number of tickets closed, PRs merged, bugs found, story points delivered, etc. This is a death spiral where the best, most senior people leaves and to compensate management hires more people, and the only way to get any order or structure is to impose even more rules, process and managers. Because so many of us grew up with programming as a hobby, often alone, late at night, we also never learned how to collaborate. We mistake "cooperate by divide and conquer" for collaboration and when we then end up in an organization where we have to interact with other fickle people all the time we build walls (branches, functional teams, layers of management, rules and processes) to protect ourselves from those interactions. It is similar to the reasons why so many developers are "afraid" of TDD, pair programming, etc. We have to *change and unlearn* old behaviours and accept that we will be crap and slow at the new thing for a while and that is hard to get past. My point is that XP/TBD/CI/CD is hard indeed. But instead of making the case for isolation by branch-driven development, PR/Jira-driven communication, etc. as the norm; what if people actually learned code hygiene, single-branch development, TDD, pair programming, etc., and got good at it first, and then turned to branches and more complex processes as the exceptions, when needed, based on context, instead of as the default strategy "because it's easier"? We also have to take into account that we have 2-3 generations of programmers that has grown up with GitHub, FB/PR, lone programmer, etc and has never seen or tried TBD/CI. Where it has always been that way and branches has always represented environments and development has always been individual programmers communicating using tickets. Fear, cognitive biases and personal incredulity makes these opposing practices feel offensive, wrong and unprofessional.
1
"feature branching is needed to have code reviews"... That is the only way? The best way? The optimal way? Forever, in all contexts? WTH!?
1
You're right, this is pure Business Sense. :D
1
"Just keep a local copy of master and merge when you are done" does not mean "two weeks of work". It means something like committing/pushing every hour or so. That means committing incomplete/unfinished code, that still has to work, but that remains unused. It requires a different way of working, thinking, designing, testing, building, communicating.
1
@KaratePath You just made the same argument that managers have been having against pair programming for 30 years now. "Don't be silly! Paying two people to type on one keyboard. We pay you to type, not talk and think." Individual programmer productivity is a management pipe-dream; which us programmers use as an excuse not having to talk to other people so we can sit alone on our corner getting paid to do our hobby. You have to look at it holistically over the *lifetime of the product* . How mistakes are caught early, as they are made. How at-least two people were part of the code's creation. Usually the code becomes simpler and clearer and easier to maintain when more than one person has worked on it. Etc, etc.
1
If you mean "Early integration of all types of changes, fast feedback and early detection of problems with the ability to always deploy without tons of meetings, reverts, planning and delays; which funnily enough speeds up development and reduces risk." then yes. :)
1
You'll get better at writing software by practicing CI/TBD first and then only use branches when there is no other choice. Your typical Ad Hominem attack: "learn how to write software" really adds to the conversation. Good work. And by the way; how is enforced FB/PR/GitFlow on everybody regardless of team, context or experience "not prescriptive corporate agile"?
1
What kind of alternative are you looking for? He did mention it; he proposes to practice CI/TBD with pair programming (or similar) and TDD (or at least automated tests) and some kind of automated SDLC pipeline to check the production readiness of the current version of the product.
1
@primevalpursuits Ok. I understand. In the olden days we used to have something called RTM (release to manufacturing) - basically meaning (after months of testing and polishing) sending the software to a factory that put is on floppies, CD-ROMs, etc. and shipped it out to stores in pretty boxes. There was a really high cost of delivering and patching software back then. Then came tools like Maven that took the old concept of "cutting a release" and made it mean "make a final (re)build and stamp a version number and publish it for deployment". This kind of clashes with the notion of a release in DevOps circles in which a release comes after a deployment. What we usually try to do in CI/CD is to stamp *every build* (deployable artifact) that comes out of the commit build (on push to main) with a unique version number (build number, etc) and make it immutable. We then move that reference through the pipeline. Preferably we should treat deployment scripts, and application configuration the same if possible. There is an article on Hackernoon: "A Guide to Git with Trunk Based Development" that has some nice solutions. It does require us to invent some kind of manifest and control repo and tooling to handle it. If you're using Kubernetes then Flux CD is basically using the same solution, but based on some kind of GitOps branching pattern. Now, I'm not using K8S (we're using Fargate and Terraform+GitHub Actions) so I can't really speak for Flux. I recommend to read "Investments Unlimited: A novel about DevOps..." - it's written in the same form as The Goal and The Phoenix Project but is based around a financial institution and how they get a year to fix things or the government are coming with the hammer. Now they chose FB/PR (which I dislike), and many other solutions they bring up are great.
1
All of this has been said and answered a hundred times before. It's just argument from incredulity. - You can easily develop multiple features at the same time in main. The team will need to approach development differently and learn other ways of working. - With CI/CD we don't attach meaning to branches - i.e merge to main does not equal a deployment to production and we avoid environment branches. - There are other ways of working that does not involve branches. Using branches in some "perfect little branch-driven process" does not make it the best way, especially when taking different contexts into account. - Using the argument that it should be so because "it works for everyone" misses the point of effective software engineering and essentially locks the team at the level of the lowest common denominator, i.e. treating everyone as untrusted advanced beginners will hold everyone's learning back. And that has a much larger impact on the long term outcome than anything else. - How about if people would just stop for a second and ask themselves "where is this person coming from and what must be true for that person for them to believe this?" instead of waving them off by implying that "it won't work in the real world" and "Dave must not have any real development experience" in some kind of standard Ad hominem that people keep on doing here.
1
Let's say you have 30 developers, each with their own feature branch. Even if all of them rebases from master every hour - how many changes are integrated to the branches? None! They stay continuously isolated and you won't know which branches that conflicts which each other until one of them is merged into main. CI/TBD is about exposing conflicts early by continuously integrating into mainline. It's nothing strange. What is strange is the obsession with branches to isolate changes and hide systemic issues.
1
@cheebadigga4092 You still won't know that there is a conflict UNTIL some branch is merged into main and the other branches are rebased. That's it.
1
CD does not mean publish/release every commit! It means publish the latest version/build that is regarded as "good" at your choosing (by your context/definition). As the saying goes "If you can't deploy right now, it isn't Continuous Delivery". Rebasing does not work if everyone's changes lives in their own branch. You will end up with lots of changes that are unintegrated since there is nothing to rebase with.
1
By "Never" he means - don't branch for development purposes. There are still business-cases where we for example might want to support multiple versions of the software (for example if it is customer installed applications), but at the speed software moves/changes today we might just as well create a fork of the application and manually backport bugfixes. There is a reason that companies charges a lot of money for subscription licenses - maintaining old versions is expensive. To answer your question: Branch by abstraction and a feature toggle. That is; create a "variation" of the pages/auth/config/whatever and have a flag that can be turned on/off if the new auth system should be used or not. This way we can turn on the new auth in some test environments but have it off in production and we can test and perform audit/penetration tests until happy. We can even just have the auth flag turned on a subset of pages to begin with and expand from there. Later on we can remove the flag if we want so that the app/site always requires auth. In CI/TBD/CD we don't equate a release with merge and deployment in the traditional sense. Deployments comes *before* release, meaning that unfinished features are deployed, but disabled/hidden until ready. This is how Facebook released their chat way back in pre-historic times. They had "javascript bots" running behind the scenes on a subset of users timelines; testing, measuring and logging stability and security. Later on it was rolled out to a subset of users for trial, that subset then grew and after a few iterations (and performance fixes) it was rolled out to 1 billion users. Note that they did not do a "all hands on deck and merge and deploy the universe and hope it can securely handle 1 billion users" on release day - it had already been running in production for over 6 months.
1
@OzoneGrif A few short points in response: 1. "Pushing untested features to master prevents you from publishing hotfixes" - No, it doesn't. CI/CD enables you to deploy to production at any time, even in the middle of development of 23 unfinished features. 2. There is no "delay for next release". In CI/CD we can deploy at any time (if we deem it necessary); there is no difference between a normal deployment or an emergency hotfix - it's the same process. As the saying goes "If you can't deploy right now, it's not CD". 3. In traditional SDLC/gitflow the granularity of a "release" is "a feature". In CD it is "the current version"; which may be once per week or 60,000 times per day (that's what Amazon is doing).
1
Instead of associating a branch with an environment we deploy a selected version (versioned binary or specific commit) to out environment of choice. Maybe the confusion is related to the thinking that branches/tags are viewed as directories on a file server and somehow represent the "readiness" of the code. The state of a particular commit/build/version IS the "readiness", not "where" it is in version control. In short: *a branch is NOT an environment* and the term "master represents production" is a misnomer and completely misunderstood.
1
Is there anything in particular you have questions about?
1
Copy-pasting my usual long rant regarding the belief that "we must have reviews to stop bad code from other bad programmers getting into the code base". I think there is this idea that we programmers only get one chance to implement something. Everything is a one-off "task/project" and then we move on to the next thing. Nothing is ever perfect, learn to accept that and then do something about it. Life is messy, sh*t happens, deal with it and move on. Committing unperfect code, unperfect design, janky solutions (that nevertheless works) is actually OK. What it requires is that people actually have to learn to refactor, redesign, and do continuous improvement *all* the time. It's never perfect, it's never done, so get into the mindset of always fixing and improving things that looks "off". It is a good idea for the team to have recurring reviews of the current state of the software and always improve issues as they are found. Keeping the code clean and working is more important than adding more features, or following a project plan or some rules put down by people not doing the job, all separated by 5 levels of management. The second problem is that every programmer appears to believe that programming is a solemn activity. That is wrong; it's a social activity. In a company setting I see FB/PR as a symptom of missing teamwork and/or bad organizational leadership. If every developer wants to work alone in their corner with headphones and only intermittently "communicate" and "integrate" using PRs and "do their own stuff" you do not have a team. Don't you talk? And yes, we love to use dumb excuses like "but I'm an introvert", etc which is just BS to avoid having to talk with other people while getting paid to do our hobby.
1
Unless it's merged, built, tested and verified on *mainline* - it's not CI - by definition. CI is not "a server that does stuff with code", it's a practice. Maintaining multiple versions (for bug-fixes) can be done easily with release branches where changes are either merged in or backported from master. Having a separate testing department is death to CI/TBD.
1
I promise you there is no regulation that says that you have to use feature branches, pull request and that it must be reviews by a senior dev. It is the organization that has interpreted the compliance requirements [developers shall not be able to make and deploy changes without proper attestations] that way because it's easy for the people in charge to just make blanket edict, clap their hands and declare "we are compliant". It's lazy and damaging to the quality and performance of the organization. Not that they would ever understand, since they are not the one's doing the job. Dave just posted a video about alternatives.
1
@etherweb6796 I do not understand this answer! I was merely pointing out common misconceptions around CI/CD and TBD. Develop can't be your source of truth *unless* you are building immutable production artifacts from "develop" and testing *that* in lower environments. But then you can just skip it and call it "main". Anyway...
1
Was this supposed to be an answer to cugamer8862's comment? I agree anyway. :)
1