Comments by "Daniel Sandberg" (@ddanielsandberg) on "Why CI is BETTER Than Feature Branching" video.

In OSS there may be 1000s of untrusted contributors all over the world and only a few trusted committers. This is where feature branching and PRs shines, this is what feature branching/pull requests were made for. Doing feature branching when co-located, pairing, having whiteboards, talking, discussing, working with each other for years and a shared vision is in most cases just bureaucratic silliness! Unless of-course the "team" consists of individual "pros" that thinks that working in IT is sitting alone in their corner, wearing headphones, getting paid to do their hobby. Want to bet that most programmers with less than 5 years of experience has never seen anything but FB/PR? Ever wondered what we did before that (and git) was around? It has become dogma, almost in a biblical sense. This is the way, the only way, you heathen! So us old developers that are doing TBD are viewed as heretical and unprofessional hobby programmers by the new guys. Kinda funny actually. Context is everything and most developers has no such understanding, nor perspective.
4
I know it feels like the right thing to do. To create processes and branches and whatnot to protect from sloppy practices. But here's the thing, adding these protective measure only ensures one thing; allowing those bad practices to endure. When there is no longer any systemic pressure to "get better" we instead just reinforce bad behavior and never learn any better. While it might be a good idea to create an environment where it is safe to fail for new members, this will inevitable just hold the rest of the organization back. We have effectively created a system where the best we can do is the lowest common denominator. And now we are doing the same mistake (to ourselves) that managers have done to development teams for decades - looking for the silver bullet (low trust) process where everyone does the same thing the same way - holding everyone back.
3
First of all, don't get hung up on the "18 months" thing. That was just an example how bad it can get, i.e a slippery-slope scenario. The first thing (that is painfully obvious in this comment section) is that apparently everyone are working alone, everyone is focusing on finishing "their" tasks, their changes. This is so wrong it hurts. The second worst scenario you can have in an IT organization is a bunch of developers working alone in their corners with headphone on, thinking they are a team. The absolute worst scenario you can have is a lead/manager assigning tasks to individual developers and keeping some kind of tabs/score and handing out gold stars like in first grade. Combining these two scenarios will just create a culture where Bob won't give a sh*t if Anna is in trouble and need help, because he "got to get his tasks done on time or the manager will get angry at him". And then if there is some kind of (merge) conflict it will escalate into a personal conflict. Why not also start paying testers for the number of bugs found, and developers for the number of lines of code written? Wait, we tired that in the 80's - disaster! As a developer you are supposed to refactor. If you are making changes to a module anyway and spot something that's a bit off, just fix it, now! It's there, right in-front of you, you know how it should be, pair up and just do it. This is a really good habit. Look up the term "refactor mercilessly". Regarding OSS: Feature branching and pull-requests were invented for and by the open source community. A situation where you have maybe 1000s of untrusted contributers and just a few trusted committers. This is where FB/PR shines. If you do not find yourself in that situation, there are better ways, like TBD and CI.
2
Yes. I think it's the 50 FPS codec that's messing things up. :/
2
This assumes that the only work happening is the feature, that we would know where conflicts are likely to happen. The thing about TBD and CI is that it has really nothing to do with "optimal isolation of features and workflow for individual developers to avoid conflicts". It is the opposite. There is this thing/strategy called "artificial constraints" which basically sets out guardrails and signs to push peoples behavior in a certain direction. This can be used both in a limiting sense (like feature branching which generally arises due to some kind of culture/trust issues and some need for illusion of control), but it can also be used to change behavior and level-up the entire organisation (like when your personal trainer pushes you and says "only 3 more minutes to go" even though "but it was only 3 more minutes - 15 minutes ago!!!"). Imagine the following scenario: You are working on some feature in your branch, you get a bit stuck, it won't work and you can't figure out why. You start to follow the code-path, you rip your hair and after two hours you figure that there is a function elsewhere that your code is calling and your change causes this function to be called twice. It is named getTicket(username) and it obviously gives you a ticket, right, RIGHT? What you didn't know is that this function actually creates a new ticket every time and that made you waste 2 hours. Now the question becomes - what do you do? Do you just fix your feature code based on this new understanding, commit, create a PR, move on to the next feature? What if you instead fixed the name of the function to its proper name createNewTicket() and also moved it to a different file/class/module because you realized it was in the wrong place. Or do you think "not in scope, feature first, don't touch it, too scary"? Think about it, you just spent 2 hours staring at this thing before you figured out that you were misled by the function name. You could fix it now while the understanding is fresh in your head and save everyone 2 hours in the future, every time they use the function. What if everyone did that all the time; made some little change that improved the life of everyone, forever? How much better would everyone and everything have become after a year, after three years? Now for the kicker; if there is a whole bunch of feature branches and pull requests waiting to get merged and people are doing this opportunistic refactoring that continuously improves the code base for everyone. How long does it takes before people starts getting angry due to merge conflicts and then the culture becomes "Don't refactor! Don't improve!". Feature Branching, when used dogmatically (often due to culture and trust issues) without understanding the long term impact and taking a more holistic systemic view on quality and culture, will actually inhibit learning . There is a difference between learning and being taught. And that is the nuance that is missed when all everyone is talking about is "optimal isolation of features and workflow for individual developers".
1
Why would main/master be broken all the time?
1
Not sure if I understand your question correctly. I will assume that you are talking about how downstream stages/jobs would not having the code constantly changing under them. This kind of gets to the core of how developers tend to misunderstand version control. Version control is not a file-server and a branch is not a directory . Version control should be viewed as a versioned acyclic graph database. This mean one cannot really "build a branch" or "deploy a branch". One would in most cases produce a build artifact (package, zip, pip, npm, etc) from a specific commit and place it in a binary repository of some kind. But even in those other cases when a deployment would mean that we are simply "copying a branch" to a webserver we would never copy HEAD , we always copy the commit/version/tag we have tested even if there are 100 new commits in trunk. This is probably another reason why people get hung up on "we must use feature branches because master must be pristine and represent production" and then fights breaks out because we talk past each other (which is evident in this comment section). For each job in the pipeline we have the following information provided to downstream jobs: - repo - branch - commit - generated build number and version - a copy of the workspace with produced artifacts OR a reference to an artifact repository - other metadata as needed So to answer your question (as I understood it); even if trunk is constantly changing we use the same "version" we started with in all downstream jobs . The other changes that gets queued up are separate instances/runs of the entire pipeline. When it comes to dependencies between projects/repos/components it always gets a bit messy, but it's solvable. For example, a downstream project could have a script that updates the list of dependencies (like a package.json lock-file) based on the properties passed down. I.e. "as project B gets triggered it detects it got triggered because dependency A has a new build/version, try to build B using the new version of A and if the build is successful commit the updated lock-file and save the build, and so on". In a general sense, as people commit their changes to trunk a new "commit-build" is kicked off and if it is successful then the downstream jobs get triggered in turn. But what if the downstream job takes one hour and people are committing to trunk 10 times per hour? The long-running job still only runs twice - triggered once by the first commit and then again after the 10th commit - i.e it gets batched up. Every build becomes a "release candidate" that may make it into production at a later stage. There are exceptions of course depending on context. Summary: Essentially we can say that the "unit of work" that flows through the pipeline is "a commit" and not "a branch" (remember, its not a directory).
1
@IonMudreac CD makes this simpler. Since we are using the same versioned artifact thoughtout the pipeline, every time we pass a gate we can record it however we want - tags, register in external system, etc. Maybe the confusion is related to the thinking that branches/tags represent the "readiness" of the code. This is the "wrong way(tm)" of thinking. The state of a particular commit/build/version IS the "readiness", not "where" it is in version control. How we would keep track of that state is not really important in this discussion as that is usually dependent on the CI/CD tool, compliance, and company policy. In CD we are essentially codifying the development process - from check-in to production - so the automation is inherent in the stages and steps as we promote the "version" between stages and environments. Does that makes sense?
1
Never really had that problem myself. But then I have always been vocal when I see conflicts and hero-culture and try to break out of it as soon as I see the signs. There is a saying in the DevOps world - "a bug, is a bug, is a bug". Meaning that bad code, bad process, bad security, bad compliance, low trust, bad logging, metrics and audit are all bugs. But we tend to pretend they are someone else's problem because "I'm just trying to get my task done". As long as everyone has the mindset that it's a contest, that developers shall sit alone in their corners with their headphones on, typing code all day. Instead of working as a team (across roles), have good practices, a good modular design (so that there is less chance of stepping on each other), working in pairs and sometimes entire groups (see mob programming) we tend to default to defensive behaviour and then there will be friction, frustration and no joy. And then we try to smooth over the cultural issues with processes, Gitflow and feature branches and silos and handoffs with 27 new roles. A team should be measured based on result and outcomes - not individual contributions. My point? Any mindset or culture that focuses on individual "performance" (stack ranking, ego, loudest-guy-opinion, "winners-and-losers") and cop-outs like "but I'm an introvert" is A BUG! The issue with seeing ourselves as heroic loners is indeed the biggest issue in SWE. Since in your scenario both of you apparently were working in related parts of the codebase - I think you should have taken a step back, sat down with each other, talk, plan, pair-program and collaborate on both of the changes instead if sitting there frustrated that "the other guy screwed up my changes again and now I'm getting behind and losing the scrum story-point race". Maybe I'm taking this over the top, but only because I've seen it happen to other teams so many times. Learned helplessness is the worst place to be, because it absolutely drains all energy and joy of going to work and life in general.
1
@derekcarter4157 I didn't mean to imply that people are bugs. :( I meant it more like the system is buggy if it enables and incentivises certain negative behaviours and discourages other positive behaviours. The software industry has spent the past 40 years hiring and attracting certain personality types and I think it has had a detrimental effect. I know how it feels some days, I'm thinking of leaving the software industry and become a carpenter or painter or something because this has given me CPPS. “A bad system will beat a good person every time.”
1
@aprilmintacpineda2713 It is not your job as a tech lead to decide what should be worked on, and it is not your job to decide how the work should be done. It is your job to make sure that the team knows how to do their job properly, to coach them, to share your knowledge and thinking so that they can make good decisions without you in the room. Regarding your point about "why you haven't seen TBD in the wild?" is because, as I have stated before, it has become the silver bullet solution to everything. And software people, devs and managers alike, tend to jump on every bandwagon that will save them. We tend to forget what came before and going around in this 10 year cycle where everything old is new again. Then every job ad is filled with all these things and no-one knows why but we got to have "full stack engineers" and "15 years of Angular experience" and "working with GitFlow". Unfortunately it is all based on local optimization instead of looking at the whole system of work. Like putting on a band-aid instead of doing the hard work of actually changing how we work and think. This is further exacerbated when the incentives, trust and power structures does not change in any meaningful way, and we just end up hiding the real issues behind processes in some kind of "change theater". I have no problem with branches per se - when they serve a purpose. When they are used as an excuse to "be sloppy" or a "mechanism to avoid trusting people", or just implemented by rote - then they are just harmful as they often tend to hide systemic issues within the organization. The example I gave was NOT a description of my situation. It was a projection of what I could see happening at your place based on your original reply. So I'll ask the same question back - if you feel the need to make decisions for the team, to instruct them how and if to do things, to control when and what they refactor, maybe the real problem is that you don't trust the team and should ask yourself why you hired them in the first place. If trusting the team to make the right decisions and do a good job without you looking over their shoulders sounds scary, you have to ask yourself if it's the team that is the real problem. Why put refactoring tasks in a ticketing system? The only time I would do that is if it's something more general like "make order processing simpler in the order module" and then every time someone is in that module, or has some time over they help to move us closer to that goal. Refactoring is not a task/shore, it's a means to an end. And if everyone did that all the time, everything would get better and better.
1
Right, the world is not black and white. Although, programmers tend to take everything as such. There is a difference between "don't branch for development" and "branch for experiments". However I would go so far as to say that if you're doing experiments, trying out new ideas, some kind of redesign, then sure go ahead and branch - but *never merge* those experiments. Instead start over and treat it a normal CI/TBD development cycle and write it from scratch with all the knowledge you just gathered. Large "refactorings" (which often turn out to be more of a redesigns) could easily be turned into small steps, especially if you just tried it out as an experiments and know where you're going.
1
@vyli1 I have a question. What does it matter that trunk is broken just once-in-a-while (meaning seldom)? If it's broken so that unit tests, functional tests or acceptance tests fails it will be discovered (and fixed) quickly. If it's broken in such a way that it's only found during manual testing, you would probably have that happen anyway, and sometimes it's broken without us knowing - we call that a production bug. There is this misunderstanding that "always releasable" means "mainline never broken, always pristine, represents production, etc". "Always" is not black and white! While we should strive to minimize the time mainline is broken (because it blocks everyone else) the goal is not to eradicate all mistakes (impossible). The purpose of CI, CD and TBD is fast feedback IF we broke something. It's okay if it was due to an honest mistake, if it was due to sloppiness then the team have some difficult learning to do. In both cases we have learned something we can act on. To para-quote Jez Humble: "Every build is a release candidate. The goal of the CI/CD pipeline is not to prove that it works, it is to prove that it does NOT work. If you can't prove that it doesn't work you should be able to deploy that version (if you choose to)." . As I have written elsewhere, you don't release a branch, you don't deploy a branch, you deploy a good known commit/version of your code. If the mainline is broken the same hour that we need to deploy to production - it doesn't matter - because we always use/deploy the latest known working commit/version. Version control is not a file server and a branch is not a directory, it is a versioned graph of snapshots in time so we can use whichever one we'd like.
1
Not looking for an argument here, just wanting to point some things out. CI and TBD is designed to "solve" the problems you describe by exposing them instead of trying to work around them with band aids, processes, phases, hand-offs, etc. I would go so far as to say that the whole point/secret to (agile) software development has been forgotten in-place of process-itis and navel gazing. Just my opinion. Here is a great quote about agile "Agile's biggest strength lies not in solving problems, per se, but rather in exposing your buried problems and making them visible so you can deal with them. For a team switching to agile it can feel like things are getting worse, not better, but that’s often because they’re being made aware of pre-existing problems they didn’t know they had. It may not seem like it but this is actually a good thing." - Someone over 10 years ago. Or in the words of Jez Humble (co-author of continuous delivery) "If it hurts, do it more frequently, and bring the pain forward." Instead of getting bogged down in the thinking that "we can't" or "we must", what if you asked yourself a different question? "What would need to change for master/main to no longer be blocked/deployable for new features, fixes, etc; to be able to do any of these things as needed, when needed, and never be more than a few minutes from being able to deploy to production?" Problem-solving is more about asking the right questions than providing solutions. Adding complicated processes to hide the systemic issues is seldom the answer.
1
Why would it take "some time to get the code into a compilable state"? Just throwing it out there; what would need to change for things to always be in a working state?
1
Yes. Version control is not our personal backup system. So if you can't finish before going home and you somehow lost it? Who cares? I can say by experience that starting over in the morning after a good nights sleep makes for much better code/solution than trying to pick up half-broken stuff I did at the end of day yesterday. So what is more important; "saving" the hour from yesterday, or better solutions? There are lots of people in the industry that thinks that every feature/solution should be implemented twice. By the time we write it the second time our understanding has increased and our ignorance has been reduced. There are companies out there that have scripts that deletes all local clones at the end of the day so everyone has to check out from scratch every morning. And just wait until you hear about the Test-Commit-Revert workflow. :)
1
@Omego2K That wasn't my point and had nothing to with CI directly. It was just an example in trying to empathize that it's NEVER about individual programmer productivity. Just that.
1
What does it matter if the thing you did this morning is rewritten/removed a few hours or a day later?
1