Comments by "Daniel Sandberg" (@ddanielsandberg) on "Continuous Delivery"
channel.
-
1
-
1
-
Not sure if I understand your question correctly. I will assume that you are talking about how downstream stages/jobs would not having the code constantly changing under them.
This kind of gets to the core of how developers tend to misunderstand version control. Version control is not a file-server and a branch is not a directory . Version control should be viewed as a versioned acyclic graph database. This mean one cannot really "build a branch" or "deploy a branch". One would in most cases produce a build artifact (package, zip, pip, npm, etc) from a specific commit and place it in a binary repository of some kind. But even in those other cases when a deployment would mean that we are simply "copying a branch" to a webserver we would never copy HEAD , we always copy the commit/version/tag we have tested even if there are 100 new commits in trunk. This is probably another reason why people get hung up on "we must use feature branches because master must be pristine and represent production" and then fights breaks out because we talk past each other (which is evident in this comment section).
For each job in the pipeline we have the following information provided to downstream jobs:
- repo
- branch
- commit
- generated build number and version
- a copy of the workspace with produced artifacts OR a reference to an artifact repository
- other metadata as needed
So to answer your question (as I understood it); even if trunk is constantly changing we use the same "version" we started with in all downstream jobs . The other changes that gets queued up are separate instances/runs of the entire pipeline. When it comes to dependencies between projects/repos/components it always gets a bit messy, but it's solvable. For example, a downstream project could have a script that updates the list of dependencies (like a package.json lock-file) based on the properties passed down. I.e. "as project B gets triggered it detects it got triggered because dependency A has a new build/version, try to build B using the new version of A and if the build is successful commit the updated lock-file and save the build, and so on".
In a general sense, as people commit their changes to trunk a new "commit-build" is kicked off and if it is successful then the downstream jobs get triggered in turn. But what if the downstream job takes one hour and people are committing to trunk 10 times per hour? The long-running job still only runs twice - triggered once by the first commit and then again after the 10th commit - i.e it gets batched up. Every build becomes a "release candidate" that may make it into production at a later stage. There are exceptions of course depending on context.
Summary: Essentially we can say that the "unit of work" that flows through the pipeline is "a commit" and not "a branch" (remember, its not a directory).
1
-
1
-
He is as veteran as they come. If I remember correctly he's one of the original members of Thoughtworks together with Jez Humble, Dan North, Martin Flowler, et. al. In the 2000th they codified the practices of Continuous Delivery, which can be viewed as the amalgamation of agile and extreme programming with everything turned up to 11.
Continuous Delivery is not their invention but more of a parallel evolution of software engineering, coming to the same conclusion and model as many others did at the same time (waterfall doesn't work anymore, agile alone is not enough, siloed teams with 2 year plans are just death marches, devs need to know ops, ops people need to know programming, etc). They just gave it a name, scienced the hell out of it and wrote The Book on the subject.
There are 100s of books, videos and entire conferences around this concept. I suggest reading Continuous Delivery by Dave Farley (the guy in this video) and Jez Humble. But also The Phoenix Project followed by The Unicorn Project. And then if you want to go past enthusiast for professional software engineer, follow that up with The DevOps Handbook, Lean Startup, Lean Enterprise, Domain Driven Design, etc.
1
-
1
-
1
-
Never really had that problem myself. But then I have always been vocal when I see conflicts and hero-culture and try to break out of it as soon as I see the signs.
There is a saying in the DevOps world - "a bug, is a bug, is a bug". Meaning that bad code, bad process, bad security, bad compliance, low trust, bad logging, metrics and audit are all bugs. But we tend to pretend they are someone else's problem because "I'm just trying to get my task done".
As long as everyone has the mindset that it's a contest, that developers shall sit alone in their corners with their headphones on, typing code all day. Instead of working as a team (across roles), have good practices, a good modular design (so that there is less chance of stepping on each other), working in pairs and sometimes entire groups (see mob programming) we tend to default to defensive behaviour and then there will be friction, frustration and no joy. And then we try to smooth over the cultural issues with processes, Gitflow and feature branches and silos and handoffs with 27 new roles. A team should be measured based on result and outcomes - not individual contributions.
My point? Any mindset or culture that focuses on individual "performance" (stack ranking, ego, loudest-guy-opinion, "winners-and-losers") and cop-outs like "but I'm an introvert" is A BUG! The issue with seeing ourselves as heroic loners is indeed the biggest issue in SWE.
Since in your scenario both of you apparently were working in related parts of the codebase - I think you should have taken a step back, sat down with each other, talk, plan, pair-program and collaborate on both of the changes instead if sitting there frustrated that "the other guy screwed up my changes again and now I'm getting behind and losing the scrum story-point race". Maybe I'm taking this over the top, but only because I've seen it happen to other teams so many times. Learned helplessness is the worst place to be, because it absolutely drains all energy and joy of going to work and life in general.
1
-
1
-
@aprilmintacpineda2713 It is not your job as a tech lead to decide what should be worked on, and it is not your job to decide how the work should be done. It is your job to make sure that the team knows how to do their job properly, to coach them, to share your knowledge and thinking so that they can make good decisions without you in the room.
Regarding your point about "why you haven't seen TBD in the wild?" is because, as I have stated before, it has become the silver bullet solution to everything. And software people, devs and managers alike, tend to jump on every bandwagon that will save them. We tend to forget what came before and going around in this 10 year cycle where everything old is new again. Then every job ad is filled with all these things and no-one knows why but we got to have "full stack engineers" and "15 years of Angular experience" and "working with GitFlow". Unfortunately it is all based on local optimization instead of looking at the whole system of work. Like putting on a band-aid instead of doing the hard work of actually changing how we work and think. This is further exacerbated when the incentives, trust and power structures does not change in any meaningful way, and we just end up hiding the real issues behind processes in some kind of "change theater". I have no problem with branches per se - when they serve a purpose. When they are used as an excuse to "be sloppy" or a "mechanism to avoid trusting people", or just implemented by rote - then they are just harmful as they often tend to hide systemic issues within the organization.
The example I gave was NOT a description of my situation. It was a projection of what I could see happening at your place based on your original reply. So I'll ask the same question back - if you feel the need to make decisions for the team, to instruct them how and if to do things, to control when and what they refactor, maybe the real problem is that you don't trust the team and should ask yourself why you hired them in the first place. If trusting the team to make the right decisions and do a good job without you looking over their shoulders sounds scary, you have to ask yourself if it's the team that is the real problem.
Why put refactoring tasks in a ticketing system? The only time I would do that is if it's something more general like "make order processing simpler in the order module" and then every time someone is in that module, or has some time over they help to move us closer to that goal. Refactoring is not a task/shore, it's a means to an end. And if everyone did that all the time, everything would get better and better.
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
@rafaeltab This is why I asked my question the way I did.
This is a common argument. The problem is that the assumption is that "the code must be broken to make the change". The cardinal rule in CI/CD (and TBD) is that you should merge to main many times per day and you are never allowed to break the code (mistakes do happen but should be rare).
You must find a way of working where you can introduce changes to the code without breaking it. There are many strategies and patterns out there, but good unit tests are almost a requirement. More process oriented strategies like compile/build options, branch by abstraction, config flags, feature toggles, dark launching, etc. are common.
FB enthusiasts views the ability to isolate changes and "screw around" in a branch as something good. CI enthusiasts views this isolation as a flaw in the system (process, engineering, design, architecture, culture).
1
-
I assume that you mean to work by a pre-defined release plan where you have Gantt-chart like planning that "next month it's A, B", the month after that it's C and D, and by summer it will be E and F?
In CI/TBD/CD we do things differently. We associate the readiness of the a release with a version instead of associate it with where it lives in version control. We also try to separate "code fixes, improvements, etc" from "feature releases", meaning that we can get any change out at any time, in hours, irregardless of plan, without hot-fix branches and special processes.
We then promote versions between environments. Having separate testing teams, hand-offs, approvals, etc is pure death to CI/CD.
Even if the case is that we plan to release features at specific intervals we can handle that using feature flags at multiple levels.
In CI/TBD/CD we don't treat a release as a deployment in the same way. Instead deployments comes *before* release, meaning that unfinished features are deployed, but disabled/hidden until ready. This is how Facebook release their chat way back in pre-historic times. They had "javascript bots" running behind the scenes on a subset of users timelines; testing, trying, measuring and logging stability and security. Later on it was rolled out to a subset of users for trial, that subset then grew and after a few iterations (and performance fixes) it was rolled out to 1 billion users. Note that they did not do a "all hands on deck and deploy the universe and hope it can handle 1 billion users" on release day - it had already been running in production for over 6 months.
1
-
1
-
It's more complicated than that. Humans tend to gravitate towards easy solutions to complex problems, while the best solutions tend to be counter-intuitive. That "lure of easy" often leads to local optimization, isolation and tribalism. Sometimes it goes so far that the system becomes optimized for developers sitting alone in a corner, with headphones, typing code and occasionally popping their heads up exclaiming "I'm done, sent it to QA" and then keep typing. Meanwhile managers starts measuring "performance" based on number of tickets closed, PRs merged, bugs found, story points delivered, etc. This is a death spiral where the best, most senior people leaves and to compensate management hires more people, and the only way to get any order or structure is to impose even more rules, process and managers.
Because so many of us grew up with programming as a hobby, often alone, late at night, we also never learned how to collaborate. We mistake "cooperate by divide and conquer" for collaboration and when we then end up in an organization where we have to interact with other fickle people all the time we build walls (branches, functional teams, layers of management, rules and processes) to protect ourselves from those interactions. It is similar to the reasons why so many developers are "afraid" of TDD, pair programming, etc. We have to *change and unlearn* old behaviours and accept that we will be crap and slow at the new thing for a while and that is hard to get past.
My point is that XP/TBD/CI/CD is hard indeed. But instead of making the case for isolation by branch-driven development, PR/Jira-driven communication, etc. as the norm; what if people actually learned code hygiene, single-branch development, TDD, pair programming, etc., and got good at it first, and then turned to branches and more complex processes as the exceptions, when needed, based on context, instead of as the default strategy "because it's easier"?
We also have to take into account that we have 2-3 generations of programmers that has grown up with GitHub, FB/PR, lone programmer, etc and has never seen or tried TBD/CI. Where it has always been that way and branches has always represented environments and development has always been individual programmers communicating using tickets. Fear, cognitive biases and personal incredulity makes these opposing practices feel offensive, wrong and unprofessional.
1
-
1
-
1
-
First of all, testers job should not be to "break the system". Testers primary job should be to provide a different perspective while developing each feature. Build quality in so that the bug/misunderstanding never made it into the product in the first place. Yes, of course we will find bugs, but those should not be 95% "oops", they should be "huh, how come this only happens when there is a full moon and no-one thought about that?".
Or in other words: "Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product."
At the risk of pissing people off. There is no such thing as "QA", at least not in the way people think. Testers are NOT QA. In fact, QA is not a department, not a team, not a role, not a phase and not a task. Testing is about "product introspection" while QA is about "process introspection". What this means is that QA is part of everything everyone does, all the time; every decision, act, solution, etc. will affect the quality of the product. QA is a process of listening to and adapting to feedback. Meaning that all the processes, management, design, coding and testing itself are all interconnected in a holistic systemic feedback-loop called QA.
1
-
1
-
1
-
Yes. Three things.
Assumption: We are talking about software that is run as a service, provided by your company (user installed software is a different beast).
1. There is no such thing as a major refactor that's broken for days. No wonder managers get heartburn when developers mentions the word "refactor", and then the XP people comes along and say "refactor mercilessly" and the managers dies of heart attacks.
2. When doing a big re-write in a messy, badly tested codebase a common solution is to implement the new version of the module parallel with the old module directly on master branch. This is called "branch by abstraction" and has nothing to do with version control branches. Perhaps we put the code under a new namespace/package, and then have a configuration flag so that the new code only runs locally when developing. Once the new module is "complete" we can expose the configuration flag so that the build and test environments can verify both implementations. Further, when we later deploy to production this flag can be used to turn on/off the new implementation at will. If things go badly - just turn it off and the old code is running instead. When the new implementation is deemed "good" it should become the default; we remove the configuration flags, the old implementation and possibly any other "complications" left in the code. This is properly managing risk by providing an escape hatch without needing to revert, rebuild, retest, redeploy anything.
3. Another way to do it is of course to make the changes in small increments and use automated tests and verification every step of the way. Jez stated it wonderfully: "...we have to work in very small batches. This is antithetical to the way lots of developers like to work: sitting off on their own going down a coding rabbit hole for days before re-emerging. The elevation of "flow" (by which I mean individual flow, not lean/team flow, which is actually inhibited by this behavior.) [is much at fault here]. Trunk-based development is about putting the needs of the team above the needs of individual. The premise of CI and trunk-based development is that coding is fundamentally a social, team activity. This presents a challenge to the mythos of the developer-as-hero which is still pervasive in our industry."
https://twitter.com/jezhumble/status/982988370942025728
1
-
1
-
1
-
1
-
1
-
1
-
1
-
@prdoyle Wall of text incoming - duck and cover!
1. Sure, but it is not integrated until it's ON main.
2. Even if you rebase your branch, you still only check integration with what has made it into main this far. Everything else that is not on main is an unknown.
As an example: If everyone "merge into main" at the end of the day or end of sprint, you won't know if everything actually integrates until that time. That is way to intermittent and is similar to old school waterfall "integrate and test at the end", although at least it's a more frequent than once per year. If everyone merges their code into main several times per day - fine - but what do you need FB/PR for in the first place if it only lasts a couple of hours anyway? Seems like a lot of ceremony.
I have actually been there, twice, kind of - in a highly compliant industry. We had armies of testers, reviewers, configuration management people, merge bosses, process and branches, and yet we had very expensive production issues all the time. Then I took over as tech lead for the most certification-critical components. The only changes we made was that we skipped all the ceremony/merges/handoffs and started doing proper XP/CI/TBD and our response time became 10x better and our production issues went down by 2 orders of magnitude. That alone saved the company $2M per year. A few years later a bunch of managers decided that "everyone must do FB/PR because we have to be compliant (BS, yadda, blah) and we read on the internet that that is everyone else is doing so we should too" and it became dogma and enforced by people not doing the actual work. After that we had slowed down and were right back in the tank and half of us had left within a year.
*We didn't need FB/PR - we were doing better without them already!*
Because so many are used to treat version control as a kind of personal junk-drawer where we throw things into a branch, fiddling around for a while, have it being broken for hours/days, flaky and unstable and then pick up the pieces into a whole at the end, then squash merge and PR; they get stuck in that cycle.
1. This is often a sign of bad design or issues with the culture.
2. There is a world of difference between "half baked", "incomplete" and "messy crap" - in XP/CI the software should still always work properly.
3. Code reviews are primarily there for learning and feedback. Async blocking reviews actually slows the organization down and have questionable effect on the quality. And even so not all changes requires review and not all reviews have to be blocking - see Martin Fowlers 3 types of reviews "ship-show-ask". FB/PR proponents are stuck 100% on "ask". You can always refactor and improve things after the fact, after it's on main, and even after it's gone live. Something people seem terrified of.
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1