Comments by "Daniel Sandberg" (@ddanielsandberg) on "Continuous Delivery"
channel.
-
36
-
24
-
I think that a lot of gamers and game devs misses the point here a little bit. I have heard this many times from colleagues: "If management/designers could just make up their minds and leave us alone for 3 months...". Sigh. It's supposed to change!
The entire purpose of agile, scrum, etc. was to provide consistency and enable the business to adapt and change as new information, ideas, market issues arises. Continuous Delivery takes the premise from agile and the technical practices from extreme programming and turns it all up to 11. Neither agile nor CD is simply a "developers issue", it's an challenge for the entire organization. Everyone must change, from the CIO/CDO to the people translating the text, to the people making the website and promotions.
If your way of working does not allow for change, including changing dates and/or dropping other features, the result will not be pretty and it will hurt. If the processes, the technical people and management are all pulling in different directions; developers afraid of speaking up, the organization being siloed (it's not my job to test, I just write the code attitude), and managers being incapable to actually manage, it doesn't matter what you do or what you call it.
So no "their constant changes" does not waste a lot of time. Crappy development and management practices does.
23
-
14
-
14
-
13
-
A lot of the issue in University settings is that you often get a "problem of the week" to solve, then the next week there is a new problem.
You mostly work alone and you seldom have to live with the solution you made in the past.
I wish that the majority of tasks was a two person tasks and that most built on top of each other. That would teach the students code hygiene, refactoring, unit testing, documentation, version control and collaboration.
At the time of dinosaurs, when I was in University, we only had two courses in year 4 that could constitute a "real project". One was building a controller and data collector for hardware SPI laboratory equipment. And the other was building a microcomputer simulator in Java (registers, memory/page handling, interrupts, and all), an microkernel operating system to run on said microcomputer and a compiler/linker to go with it. That was really messy since we never had properly learned to be software engineers and work as a team.
10
-
9
-
This is kind of what is called "staying in the problem-space before going to the solution-space". Focus on the problem, the underlying assumptions, the whom and why, not the what or how.
One of the biggest problems with us programmers is that we tend to jump directly to the solution-space. Some business person starts talking about a problem and 15 seconds later our brains have stopped listening to the problem and is instead filled with arrows, boxes, patterns and whatever technology we know. Five minutes later we don't even remember what the problem was, just that we get to write a bunch of code using nodejs, k8s, docker and NoSQL (because I don't have NoSQL on my resumé yet). What if the correct solution is to remove a feature, delete 10,000 lines of code and add a new step to an existing process/feature?
Code is not an asset. Every line of code is a cost, a liability. It has to be built, tested, maintained, deployed, configured and operated. Programmers and code do not exist for the sake of their own existence even if we like to believe that we are. We are not getting paid to do our hobby, sitting in a corner with our headphones on, listening to "programmer focus synthwave mixes".
Sorry.
8
-
7
-
7
-
7
-
6
-
5
-
5
-
4
-
4
-
In OSS there may be 1000s of untrusted contributors all over the world and only a few trusted committers. This is where feature branching and PRs shines, this is what feature branching/pull requests were made for.
Doing feature branching when co-located, pairing, having whiteboards, talking, discussing, working with each other for years and a shared vision is in most cases just bureaucratic silliness! Unless of-course the "team" consists of individual "pros" that thinks that working in IT is sitting alone in their corner, wearing headphones, getting paid to do their hobby.
Want to bet that most programmers with less than 5 years of experience has never seen anything but FB/PR? Ever wondered what we did before that (and git) was around? It has become dogma, almost in a biblical sense. This is the way, the only way, you heathen! So us old developers that are doing TBD are viewed as heretical and unprofessional hobby programmers by the new guys. Kinda funny actually.
Context is everything and most developers has no such understanding, nor perspective.
4
-
4
-
Do you mean "trying TBD" just like organizations "tries Scrum" by replacing requirement with user-story, milestone with sprint, deadline with release-trains, keep assigning devs by "percentage utilization" and doing annual budgets, writing 500 pages of "design" for a year before starting, developers communicating using Jira-tickets instead of talking, and nothing really changes, culturally or behavioral, et. al?
You can't just decide to "go CI/TBD" unless you have the practices and tools that makes it possible. Like pairing, TDD, good fast build system, being able to make changes by many-many small steps without ripping 23 modules apart for 3 days and a culture that supports it.
CI/TBD is a skill, and just like everything else it takes time to get there. I would suggest to start by setting up some things:
1. Make sure there are high-level tests covering the most important happy-paths.
2. Make these tests fast, unfragile, and make them/the system runnable on developers machines.
3. No changes gets committed without some kind of test covering the code being changed.
4. No feature branch lasts for more than 1 day.
5. Get a build system that tests main at every merge/commit.
6. If the build turn red - you have 10 minutes to fix it, or revert it.
7. If you can't get the build back to green EVERYBODY in the team stops what they are doing and do anything they can to get it back to green. There is no situation where a failing build is OK to just leave and keep doing other work (which usually happens with branches - habits matters).
Getting these things into place, and making them a habit will go a long way.
Next we turn the heat up:
8. Pair program as much as possible. Not every change requires a review (and some changes can be reviewed after the fact. See "Ship / Show / Ask" by martin fowler).
9. Try to slice changes into small parts that takes a couple of hours or less.
10. No feature branch lasts longer than a couple of hours.
11. TDD and refactor mercilessly. Make changes easy.
12. Make it hot: Any changes that has made it into main may be deployed 30 minutes after you commit, when you're on lunch. Now code-hygiene, care and practices really starts to matter.
13. Start doing some changes without a feature branch/PR. Evaluate, practice, habit.
14. Remove organizational/cultural issues: The entire team succeeds or fails together. As long as everything is driven by "individual tasks" (and associated rewards), stupid "sprint commitments", and "done" that means - "I coded it, it's done, right? Then a separate QA-department is trying to inspect quality in after the fact and it's someone elses problem after that" - nothing you do will matter. Culture eats strategy and intentions for breakfast.
15. Expect things to go wrong, learn to deal with it and improve.
It takes years for a team to get good at CI/TBD. It takes even more years for an organization to get good at CD.
3
-
3
-
3
-
@OlleHellman The reason I'm asking is that basically everyone in this comment section seems to think that CI is a tool, running "checks" on a build server against your feature branches.
That is not CI, by definition.
You don't "run through CI".
You don't "have CI".
You don't "use CI".
You either do CI or you don't!
CI is a practice and a social contract *not* just a build server even though tool-vendors would like us to think so.
Everyone integrates their changes into the *mainline* , at least once per day (preferably more).
Unless it's merged into main and automatically verified, it's not integrated.
If you can't run a majority of these checks on you local environment before committing, you *can't do* CI.
Running checks against a feature branch is not CI.
Sorry, but I'm adamant about this because our industry has gotten into this bad habit of redefining things to mean "whatever we are doing right now".
- We installed Jira and have burnups, burndowns and user stories - we're agile.
- We created a new team called "DevOps" and hired "DevOps engineers" - we're DevOps.
- One write code, then someone else looks at the code, then we change the code, then we PR the code - we're pair programming.
- We are running Jenkins against "whatever we are doing with branches right now" - we're doing CI.
3
-
3
-
3
-
3
-
3
-
3
-
In part. Continuous Integration is a human practice where it is expected that the developers pulls the latest changes and run the unit tests, lint, checkstyle, findbugs, (and component integration tests) before pushing. Those same checks are then run at push, which is called a "commit build" and should give you a high confidence that everything works. In CD there are other kinds of tests and checks downstream that takes different aspects of the entire system into account.
As I understand it, the embargo shown here is seldom used. Again, CI is a human practice and a social contract: if the build is not fixed within 10 minutes, anyone can revert the commit that broke it. The key here is to be disciplined and don't break things all the time, and the few times it breaks we do whatever we can to get back to a working state. Shit happens, learn from it and move on.
Part of the issue with feature branches is that it optimizes for individual developers working alone in their corner with headphones on. This is a bad thing. Sure, you can use short-lived feature branches if you must; where (in my view) short means a couple of hours. But if most only live 30 minutes to a few hours what's the point of branching in the first place?
Look, I'm just an opinionated a-hole that's tired of arguing with people and also thinks that part of the popularity around FB/PR is an amazing feat of social engineering by the tool-vendors that has even the oil and tobacco companies in awe. So I'm just going to give up and drop this Twitter thread by Jez Humble: https://twitter.com/jezhumble/status/982988370942025728
3
-
3
-
3
-
3
-
I hope Dave doesn't mind me plugging one of Dan Norths presentations "Why agile doesn't scale and what you can do about it".
It is primarily about how "agilists" and business talk past each other and how developers need to stop focusing on technical things and "standups, burndowns, burnups, userstories, storypoints and ..." and taking a more holistic view and understanding of the business.
Organizations tend to end up in a vicious cycle where they restructure the organization and hire a new CEO/CIO/CTO every three years and start over, while all the senior developers, PMs, etc just gives up, sighs and leaves for "something better" and all the lessons learned are forgotten. Agile (and CD) affects the entire organization, from the sales people to CEO/CTO/CIO to developers, HR and training. Without this understanding, no process or methodology will make much of a difference in the end.
https://www.youtube.com/watch?v=F4b_MckXea0
https://www.youtube.com/watch?v=1MedZStiAPg (better audio)
3
-
3
-
2
-
2
-
2
-
2
-
I must agree. Every single "git tutorial" starts with "branching" and then too many developers think that that is the goal - to follow some branching strategy perfectly - missing the point of version control and software to begin with. The number of times I've been told I'm unprofessional because I avoid FB/PR and code-reviews and prefer CI, TDD, pair-programming and good tooling that tells me if I'm particularly dumb today. I can't even change jobs anymore because "FB/PR/CR" is mandatory everywhere and I've even been to interviews where they never heard the term "Continuous Integration".
I have seen similar effects with for example agile; "we are doing burn-up, burn-down, user stories, story points, sprints and planning poker; therefore we are agile" - and then people end up fighting over "who is doing story points right". They don't understand agile. Oh, and don't get me started on "DevOps Engineer" - if your company have DevOps Engineers and DevOps Teams you don't understand DevOps either, but I digress. I had another experience where the developers wanted to go nuts with versioning; alpha, beta, rc1, rc2, final, gold and when I asked why the answer was basically "it's what the big companies are doing" - so essentially they wanted a complex release process because it "sounded professional", not understanding that all that complexity is most likely a symptom of bad software engineering. Complexaholics all around. Bring the pain!
Now I need my benzos again.
2
-
You state that CI is based on knowing the right solution the first time. It's the opposite!
CI and most "agile" practices recognizes that we can't know the correct solution nor the right implementation, or even properly understand the problem to begin with, and optimizes for being able to turn around and adapt over time.
FB/PR on the other hand optimizes for doing it once, one chance, most of the time with an "all or nothing" mentality. Get it right the first time by working on it and polishing it for some time, merge it, then jump to the next feature and don't look back. With PB/PR the granularity of a "commit/push" is "a task".
With CI the granularity of a commit/push is "small change that works and takes us closer to the goal, but sometimes we have to turn around and change it, even when it has reached production" (it still works, it's not broken sh*t code, it's just not quite right yet).
This way of working with CI practices scares people that have never tried it and that often are measured and rewarded based on some kind of productivity metric or risk ending up on some kind of PIP.
2
-
2
-
2
-
2
-
2
-
2
-
2
-
First of all, don't get hung up on the "18 months" thing. That was just an example how bad it can get, i.e a slippery-slope scenario.
The first thing (that is painfully obvious in this comment section) is that apparently everyone are working alone, everyone is focusing on finishing "their" tasks, their changes. This is so wrong it hurts. The second worst scenario you can have in an IT organization is a bunch of developers working alone in their corners with headphone on, thinking they are a team. The absolute worst scenario you can have is a lead/manager assigning tasks to individual developers and keeping some kind of tabs/score and handing out gold stars like in first grade. Combining these two scenarios will just create a culture where Bob won't give a sh*t if Anna is in trouble and need help, because he "got to get his tasks done on time or the manager will get angry at him". And then if there is some kind of (merge) conflict it will escalate into a personal conflict. Why not also start paying testers for the number of bugs found, and developers for the number of lines of code written? Wait, we tired that in the 80's - disaster!
As a developer you are supposed to refactor. If you are making changes to a module anyway and spot something that's a bit off, just fix it, now! It's there, right in-front of you, you know how it should be, pair up and just do it. This is a really good habit. Look up the term "refactor mercilessly".
Regarding OSS: Feature branching and pull-requests were invented for and by the open source community. A situation where you have maybe 1000s of untrusted contributers and just a few trusted committers. This is where FB/PR shines. If you do not find yourself in that situation, there are better ways, like TBD and CI.
2
-
2
-
2
-
2
-
2
-
2
-
2
-
What you just described is an argument *for* CI. CIs main purpose is to *expose* all these faults (code, technical, culture) and by extension: XP's and agile's purpose is to expose all the *systemic* issues in the organization and give everyone a health dose of reality. If it's broken all the time and some project manager still thinks "it will be delivered on time, work properly and be of good quality" and developers just keep adding more code to a broken system, someone is lying to themselves.
Version control is not your personal backup-system, it's an auditable, versioned, *publication* mechanism. As a developer your job is not to just implement some tasks, push to git when you go home for the day and the rest is "someone else's problem" or "other developers fault". Either the entire team wins, or the entire team fails.
1. You make sure to merge other peoples changes from master to your local clone many times per day, and run the build, unit-tests, linting, and functional tests *before* you push to master. That way you know if there is a conflict before you push. This is the prime directive of CI - check your sh*t before you push. And if it still breaks on the build servers, stop working, drop everything and fix it! Did it break because you made a change or did someone else make a change that conflicted with yours? Doesn't matter, say "huh, that's odd?" and then talk to your team members and fix it.
2. Someone deleted a database record/table/whatever in a test-environment and broke something? Well, then everyone stops, figure out what and why it happened and then *talk* and solve the real problem (people, process and culture), not just the symptoms and complaining "Aaah, it's broken again!!!".
3. "Algorithm works here but not there"? Not sure how that could happen. But here is a tip: build it **once**, deploy the same code/binary/package in the same way, by the same mechanism, to every environment, every time, and 95% of all the "works here, but not there problems" goes away.
4. Acceptance tests break but you haven't bothered to run it? How do I even respond to that!?
"If it hurts, do it more, and bring the pain forward." - Jez and Dave
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
@OzoneGrif Why is it unacceptable? I don't mean to sound antagonistic but you need to think outside the box of release-trains, Gantt charts, milestone-driven release schedules, and manual test-and-stabilization phases.
First of all, all code shall be tested whether they are "feature complete or not", and then you have an evolution:
1. Early in development the features may be excluded/included from the build by using build profiles. This allows local development and testing as the team fleshes out the basics of the feature.
2. Later on features may be included but be disabled by default and enabled by a deployment flag in different environments. Great for feedback and early testing. See "branch-by-abstraction pattern" for one example of implementation details.
3. Once we reach the "this is a nice feature, polish it"-stage we may choose to convert the flag to a runtime property or keep it as a deployment flag. Context and technology matters.
4. When we finally are ready to release the feature to the world we can turn it into a user preference (Opt-in/beta users for the new feature), or even using Geo-location or target groups of who will see it.
5. This allows the business to then decide if the feature shall be available for all users and we can remove all the flags and gunk around it. Or we may decide that it's an "enterprise feature" and customers have to pay for the feature to be enabled.
The point is - you can have a lot of unfinished changes in-flight, in a single repo/branch and still always be able to deploy a hotfix.
This is kind of how facebook implemented their chat-system. It was running (by bots in your browser) in production, in secret, behind users timelines, and only facebook employees could actually use it. It had been running in production for over six months before normal users could access it and then they incrementally rolled it out to users - without needing to build/deploy/release a special version.
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
2
-
1
-
1
-
1
-
1
-
@buddysnackit1758 Oh, I read it all right. Trying to disprove CI by arguing that at the extreme it isn't a millisecond contiguous synced workflow is just a strawman argument. It's like saying that science is false because we can never prove anything with absolute certainty. So, no I didn't address it - it's just nonsense.
Second point. With CI we collaborate with other team members, pair program, even pair with testers and SMEe, instead of sitting alone in a corner with headphones for days on end only to share the result at the end (which is all too common). So the amount of "adding wrong code" isn't 50% to begin with. And even if it was, that is not the big deal you make it out to be since the code still have to work , even if it's not quite right. XP/CI embraces that we are not perfect, but instead of isolating work and people with gate-keeping we optimize it around learning, feedback, outcomes and result.
1
-
Short answer:
1. You can commit incomplete code as long as it works and doesn't break anything else. If it's complex, very WIP and "not yet usable" then disable/hide it somehow (usually a build/compile flags). Improve and refactor as you go.
2. You don't "check if it works at the end of each day", you check it ALL the time, you make a full local build every 5-15 minutes and then check-in the code and let the Build Management System validate you didn't screw up. That's CI!
3. It works just as well for changing existing code as well as adding new code (that will need to be integrated with the old code anyway). It will often be faster to deliver *because* you know it always works (whatever definition if working you're using). No "test phase", no "integration phase", no "long arguments if we can merge it yet" because it's already integrated, finished or not.
Nothing is free and everything we know how to do takes time to learn. Meaning, going from "sit alone in a corner, coding for days on a personal branch" to "everything always works and check-in the code many many times per day" is hard. Working with TBD and CI practices takes practice, discipline, and code hygiene.
1
-
1
-
Wall of text, partially off-topic warning!
Let's take this from the start - releases
In the olden days we used to have something called RTM (release to manufacturing) - basically meaning (after months of testing and polishing) sending the software to a factory that put it on floppies, CD-ROMs, etc. and shipped it out to stores in pretty boxes. There was a really high cost of delivering and patching software back then. Then came tools like Maven that took the old concept of "cutting a release" and made it mean "make a final (re)build and stamp a version number and publish it for deployment". Later CI/CD/DevOps came into the picture and things started to change; we came up with terms like "separate deployment from release". What is meant by that is that deployment comes before release. Enabling new functionality or changing old becomes some kind of configuration change instead of a "all hand on deck deployment 3 AM" thing. This also enables A/B-testing, user opt-in and deploying many many small changes all the time - thus reducing the blast-radius if something goes wrong as well as knowing that every change actually works in real life. This doesn't mean that every change gets a deployment, it's just that deployments becomes a choice by need instead of a planned date/deadline.
How does it (ideally) works with CI/CD?
With CI/CD we instead try to subscribe to the (impossible) ideal of Toyota "one-piece flow". Instead of keeping track of the state of the software by using branches/merges (with all the management overhead) and thinking in terms of "master represents production" or "let's build a release and a release branch" (a bit like the previously mentioned "cutting a release") - we commit to main, we build it once , we record the version/tag/commit/buildnumber/digest of that build; we test and validate it, and then pass that immutable piece from environment to environment, from validation to deployment and finally release. It's a survival of the fittest scenario to try to prove that a build does not meet the criteria for deployment.
Extrapolating and deeper context
As with everything there are contexts and compromises. But if we default to use branches for change control, separate departments for frontend, backend and testing, huge ops-teams getting software thrown over the wall from developers (we made a release, your turn) because it feels easier and it worked 20 year ago we are not making any progress or improvements. According to some of the "old IT/tech people" the amount of programmers doubles every 5 years. About 10 years ago everyone went nuts with FB/PR/GitFlow. So we can extrapolate that 75% of all programmers have never done anything but, and so have no idea about Continuous Integration.
I'm really passionate about this because I see the same thing repeating in every org I've been in. As long as our thinking doesn't change it doesn't matter how much technology, tools, languages, frameworks and processes we throw at software development - nothing changes.
1
-
1
-
99% of all IT companies wouldn't know agile if it hit them in the face. Most companies thinks it's just a process and takes all the "easy parts" from scrum, skips the good parts from structured SDLC and ends up with the worst of two worlds. Thinking that burnups, burndowns, boards, jira, standups, story points, user stories, sprints, etc. makes them agile. They ignore all the technical practices from XP, CD, et al. (that is required to make it work) and then wonder why it didn't.
Then people come out on the other side saying "I hate agile". Like my grandpa - he refused to try pizza his whole life. A few weeks before he died we asked him why he never tried it - his answer? "It's just a pancake with ketchup and cheese!".
1
-
This assumes that the only work happening is the feature, that we would know where conflicts are likely to happen. The thing about TBD and CI is that it has really nothing to do with "optimal isolation of features and workflow for individual developers to avoid conflicts". It is the opposite.
There is this thing/strategy called "artificial constraints" which basically sets out guardrails and signs to push peoples behavior in a certain direction. This can be used both in a limiting sense (like feature branching which generally arises due to some kind of culture/trust issues and some need for illusion of control), but it can also be used to change behavior and level-up the entire organisation (like when your personal trainer pushes you and says "only 3 more minutes to go" even though "but it was only 3 more minutes - 15 minutes ago!!!").
Imagine the following scenario: You are working on some feature in your branch, you get a bit stuck, it won't work and you can't figure out why. You start to follow the code-path, you rip your hair and after two hours you figure that there is a function elsewhere that your code is calling and your change causes this function to be called twice. It is named getTicket(username) and it obviously gives you a ticket, right, RIGHT? What you didn't know is that this function actually creates a new ticket every time and that made you waste 2 hours.
Now the question becomes - what do you do? Do you just fix your feature code based on this new understanding, commit, create a PR, move on to the next feature? What if you instead fixed the name of the function to its proper name createNewTicket() and also moved it to a different file/class/module because you realized it was in the wrong place. Or do you think "not in scope, feature first, don't touch it, too scary"?
Think about it, you just spent 2 hours staring at this thing before you figured out that you were misled by the function name. You could fix it now while the understanding is fresh in your head and save everyone 2 hours in the future, every time they use the function. What if everyone did that all the time; made some little change that improved the life of everyone, forever? How much better would everyone and everything have become after a year, after three years?
Now for the kicker; if there is a whole bunch of feature branches and pull requests waiting to get merged and people are doing this opportunistic refactoring that continuously improves the code base for everyone. How long does it takes before people starts getting angry due to merge conflicts and then the culture becomes "Don't refactor! Don't improve!".
Feature Branching, when used dogmatically (often due to culture and trust issues) without understanding the long term impact and taking a more holistic systemic view on quality and culture, will actually inhibit learning . There is a difference between learning and being taught. And that is the nuance that is missed when all everyone is talking about is "optimal isolation of features and workflow for individual developers".
1
-
1
-
1
-
A pipeline does not replace the steps and tasks in a build script (npm, gulp, gradle, maven, etc). This is a common mistake I see, where teams replaces everything they had in their scripts with a pipeline YAML that can only run on some build management system.
When practicing CI you should be able to run the majority of the build steps including unit test, some component and functional testing locally. Whole system acceptance tests is difficult and often too slow to run locally. It also serves as feedback; if it is difficult to run, build, test, lint, you probably need to look into the structure and design of both tests and code. If it is very hard or slow to build/test locally you may need to look into the the technology choices made (like frameworks, etc).
It is an optimization technique where you should be able to compile, package and run 10000s of tests locally in minutes. Enough that it gives the developers a high degree of confidence that it will work before pushing. The build management system will of course run the commit build as well, plus a bunch of slow wider system tests, performance, etc. This also has the added benefit that if GitHub is down (like it is now as I write this) you can have a "break glass" process and build an emergency patch on a (monitored) local computer and push that to production.
While the build system runs the commit build you are not allowed to go away, lunch, or home. If it breaks you have 10 minutes to fix it or revert the change. If it's still broken the team swarm around the problem and get it back to green.
1
-
1
-
1
-
1
-
1
-
While I will try to answer this without purposely being offending, you may feel that way anyway.
Your job as a senior/lead is not to gatekeep, that is, to check everyone's work and approve of it.
You can't behave like a curling/helicopter parent and protect juniors from themselves all the time. The only thing you'll end up doing is holding back their learning and growth, and also teaching them to not rely on themselves.
You need to find a way of working where they can contribute, learn and make mistakes, without you handholding them, nor taking the company down in the process. I don't care what that is, as long as it's a temporary measure and not you gatekeeping everything (you might as well do all the work yourself in that case).
You should be able to find a way of working where in the end "the others" would make similar (or better) decisions as you would, even when you aren't in the room. After all, the job of a senior engineer is to create more senior engineers, not dissuade people and having them leave within a year.
Dave has another video here "A Guide To Managing Technical Teams" in which he calls your style "programming by remote control".
1
-
1
-
1
-
1
-
1
-
Not sure if I understand your question correctly. I will assume that you are talking about how downstream stages/jobs would not having the code constantly changing under them.
This kind of gets to the core of how developers tend to misunderstand version control. Version control is not a file-server and a branch is not a directory . Version control should be viewed as a versioned acyclic graph database. This mean one cannot really "build a branch" or "deploy a branch". One would in most cases produce a build artifact (package, zip, pip, npm, etc) from a specific commit and place it in a binary repository of some kind. But even in those other cases when a deployment would mean that we are simply "copying a branch" to a webserver we would never copy HEAD , we always copy the commit/version/tag we have tested even if there are 100 new commits in trunk. This is probably another reason why people get hung up on "we must use feature branches because master must be pristine and represent production" and then fights breaks out because we talk past each other (which is evident in this comment section).
For each job in the pipeline we have the following information provided to downstream jobs:
- repo
- branch
- commit
- generated build number and version
- a copy of the workspace with produced artifacts OR a reference to an artifact repository
- other metadata as needed
So to answer your question (as I understood it); even if trunk is constantly changing we use the same "version" we started with in all downstream jobs . The other changes that gets queued up are separate instances/runs of the entire pipeline. When it comes to dependencies between projects/repos/components it always gets a bit messy, but it's solvable. For example, a downstream project could have a script that updates the list of dependencies (like a package.json lock-file) based on the properties passed down. I.e. "as project B gets triggered it detects it got triggered because dependency A has a new build/version, try to build B using the new version of A and if the build is successful commit the updated lock-file and save the build, and so on".
In a general sense, as people commit their changes to trunk a new "commit-build" is kicked off and if it is successful then the downstream jobs get triggered in turn. But what if the downstream job takes one hour and people are committing to trunk 10 times per hour? The long-running job still only runs twice - triggered once by the first commit and then again after the 10th commit - i.e it gets batched up. Every build becomes a "release candidate" that may make it into production at a later stage. There are exceptions of course depending on context.
Summary: Essentially we can say that the "unit of work" that flows through the pipeline is "a commit" and not "a branch" (remember, its not a directory).
1
-
1
-
He is as veteran as they come. If I remember correctly he's one of the original members of Thoughtworks together with Jez Humble, Dan North, Martin Flowler, et. al. In the 2000th they codified the practices of Continuous Delivery, which can be viewed as the amalgamation of agile and extreme programming with everything turned up to 11.
Continuous Delivery is not their invention but more of a parallel evolution of software engineering, coming to the same conclusion and model as many others did at the same time (waterfall doesn't work anymore, agile alone is not enough, siloed teams with 2 year plans are just death marches, devs need to know ops, ops people need to know programming, etc). They just gave it a name, scienced the hell out of it and wrote The Book on the subject.
There are 100s of books, videos and entire conferences around this concept. I suggest reading Continuous Delivery by Dave Farley (the guy in this video) and Jez Humble. But also The Phoenix Project followed by The Unicorn Project. And then if you want to go past enthusiast for professional software engineer, follow that up with The DevOps Handbook, Lean Startup, Lean Enterprise, Domain Driven Design, etc.
1
-
1
-
1
-
Never really had that problem myself. But then I have always been vocal when I see conflicts and hero-culture and try to break out of it as soon as I see the signs.
There is a saying in the DevOps world - "a bug, is a bug, is a bug". Meaning that bad code, bad process, bad security, bad compliance, low trust, bad logging, metrics and audit are all bugs. But we tend to pretend they are someone else's problem because "I'm just trying to get my task done".
As long as everyone has the mindset that it's a contest, that developers shall sit alone in their corners with their headphones on, typing code all day. Instead of working as a team (across roles), have good practices, a good modular design (so that there is less chance of stepping on each other), working in pairs and sometimes entire groups (see mob programming) we tend to default to defensive behaviour and then there will be friction, frustration and no joy. And then we try to smooth over the cultural issues with processes, Gitflow and feature branches and silos and handoffs with 27 new roles. A team should be measured based on result and outcomes - not individual contributions.
My point? Any mindset or culture that focuses on individual "performance" (stack ranking, ego, loudest-guy-opinion, "winners-and-losers") and cop-outs like "but I'm an introvert" is A BUG! The issue with seeing ourselves as heroic loners is indeed the biggest issue in SWE.
Since in your scenario both of you apparently were working in related parts of the codebase - I think you should have taken a step back, sat down with each other, talk, plan, pair-program and collaborate on both of the changes instead if sitting there frustrated that "the other guy screwed up my changes again and now I'm getting behind and losing the scrum story-point race". Maybe I'm taking this over the top, but only because I've seen it happen to other teams so many times. Learned helplessness is the worst place to be, because it absolutely drains all energy and joy of going to work and life in general.
1
-
1
-
@aprilmintacpineda2713 It is not your job as a tech lead to decide what should be worked on, and it is not your job to decide how the work should be done. It is your job to make sure that the team knows how to do their job properly, to coach them, to share your knowledge and thinking so that they can make good decisions without you in the room.
Regarding your point about "why you haven't seen TBD in the wild?" is because, as I have stated before, it has become the silver bullet solution to everything. And software people, devs and managers alike, tend to jump on every bandwagon that will save them. We tend to forget what came before and going around in this 10 year cycle where everything old is new again. Then every job ad is filled with all these things and no-one knows why but we got to have "full stack engineers" and "15 years of Angular experience" and "working with GitFlow". Unfortunately it is all based on local optimization instead of looking at the whole system of work. Like putting on a band-aid instead of doing the hard work of actually changing how we work and think. This is further exacerbated when the incentives, trust and power structures does not change in any meaningful way, and we just end up hiding the real issues behind processes in some kind of "change theater". I have no problem with branches per se - when they serve a purpose. When they are used as an excuse to "be sloppy" or a "mechanism to avoid trusting people", or just implemented by rote - then they are just harmful as they often tend to hide systemic issues within the organization.
The example I gave was NOT a description of my situation. It was a projection of what I could see happening at your place based on your original reply. So I'll ask the same question back - if you feel the need to make decisions for the team, to instruct them how and if to do things, to control when and what they refactor, maybe the real problem is that you don't trust the team and should ask yourself why you hired them in the first place. If trusting the team to make the right decisions and do a good job without you looking over their shoulders sounds scary, you have to ask yourself if it's the team that is the real problem.
Why put refactoring tasks in a ticketing system? The only time I would do that is if it's something more general like "make order processing simpler in the order module" and then every time someone is in that module, or has some time over they help to move us closer to that goal. Refactoring is not a task/shore, it's a means to an end. And if everyone did that all the time, everything would get better and better.
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
@rafaeltab This is why I asked my question the way I did.
This is a common argument. The problem is that the assumption is that "the code must be broken to make the change". The cardinal rule in CI/CD (and TBD) is that you should merge to main many times per day and you are never allowed to break the code (mistakes do happen but should be rare).
You must find a way of working where you can introduce changes to the code without breaking it. There are many strategies and patterns out there, but good unit tests are almost a requirement. More process oriented strategies like compile/build options, branch by abstraction, config flags, feature toggles, dark launching, etc. are common.
FB enthusiasts views the ability to isolate changes and "screw around" in a branch as something good. CI enthusiasts views this isolation as a flaw in the system (process, engineering, design, architecture, culture).
1
-
I assume that you mean to work by a pre-defined release plan where you have Gantt-chart like planning that "next month it's A, B", the month after that it's C and D, and by summer it will be E and F?
In CI/TBD/CD we do things differently. We associate the readiness of the a release with a version instead of associate it with where it lives in version control. We also try to separate "code fixes, improvements, etc" from "feature releases", meaning that we can get any change out at any time, in hours, irregardless of plan, without hot-fix branches and special processes.
We then promote versions between environments. Having separate testing teams, hand-offs, approvals, etc is pure death to CI/CD.
Even if the case is that we plan to release features at specific intervals we can handle that using feature flags at multiple levels.
In CI/TBD/CD we don't treat a release as a deployment in the same way. Instead deployments comes *before* release, meaning that unfinished features are deployed, but disabled/hidden until ready. This is how Facebook release their chat way back in pre-historic times. They had "javascript bots" running behind the scenes on a subset of users timelines; testing, trying, measuring and logging stability and security. Later on it was rolled out to a subset of users for trial, that subset then grew and after a few iterations (and performance fixes) it was rolled out to 1 billion users. Note that they did not do a "all hands on deck and deploy the universe and hope it can handle 1 billion users" on release day - it had already been running in production for over 6 months.
1
-
1
-
It's more complicated than that. Humans tend to gravitate towards easy solutions to complex problems, while the best solutions tend to be counter-intuitive. That "lure of easy" often leads to local optimization, isolation and tribalism. Sometimes it goes so far that the system becomes optimized for developers sitting alone in a corner, with headphones, typing code and occasionally popping their heads up exclaiming "I'm done, sent it to QA" and then keep typing. Meanwhile managers starts measuring "performance" based on number of tickets closed, PRs merged, bugs found, story points delivered, etc. This is a death spiral where the best, most senior people leaves and to compensate management hires more people, and the only way to get any order or structure is to impose even more rules, process and managers.
Because so many of us grew up with programming as a hobby, often alone, late at night, we also never learned how to collaborate. We mistake "cooperate by divide and conquer" for collaboration and when we then end up in an organization where we have to interact with other fickle people all the time we build walls (branches, functional teams, layers of management, rules and processes) to protect ourselves from those interactions. It is similar to the reasons why so many developers are "afraid" of TDD, pair programming, etc. We have to *change and unlearn* old behaviours and accept that we will be crap and slow at the new thing for a while and that is hard to get past.
My point is that XP/TBD/CI/CD is hard indeed. But instead of making the case for isolation by branch-driven development, PR/Jira-driven communication, etc. as the norm; what if people actually learned code hygiene, single-branch development, TDD, pair programming, etc., and got good at it first, and then turned to branches and more complex processes as the exceptions, when needed, based on context, instead of as the default strategy "because it's easier"?
We also have to take into account that we have 2-3 generations of programmers that has grown up with GitHub, FB/PR, lone programmer, etc and has never seen or tried TBD/CI. Where it has always been that way and branches has always represented environments and development has always been individual programmers communicating using tickets. Fear, cognitive biases and personal incredulity makes these opposing practices feel offensive, wrong and unprofessional.
1
-
1
-
1
-
First of all, testers job should not be to "break the system". Testers primary job should be to provide a different perspective while developing each feature. Build quality in so that the bug/misunderstanding never made it into the product in the first place. Yes, of course we will find bugs, but those should not be 95% "oops", they should be "huh, how come this only happens when there is a full moon and no-one thought about that?".
Or in other words: "Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product."
At the risk of pissing people off. There is no such thing as "QA", at least not in the way people think. Testers are NOT QA. In fact, QA is not a department, not a team, not a role, not a phase and not a task. Testing is about "product introspection" while QA is about "process introspection". What this means is that QA is part of everything everyone does, all the time; every decision, act, solution, etc. will affect the quality of the product. QA is a process of listening to and adapting to feedback. Meaning that all the processes, management, design, coding and testing itself are all interconnected in a holistic systemic feedback-loop called QA.
1
-
1
-
1
-
Yes. Three things.
Assumption: We are talking about software that is run as a service, provided by your company (user installed software is a different beast).
1. There is no such thing as a major refactor that's broken for days. No wonder managers get heartburn when developers mentions the word "refactor", and then the XP people comes along and say "refactor mercilessly" and the managers dies of heart attacks.
2. When doing a big re-write in a messy, badly tested codebase a common solution is to implement the new version of the module parallel with the old module directly on master branch. This is called "branch by abstraction" and has nothing to do with version control branches. Perhaps we put the code under a new namespace/package, and then have a configuration flag so that the new code only runs locally when developing. Once the new module is "complete" we can expose the configuration flag so that the build and test environments can verify both implementations. Further, when we later deploy to production this flag can be used to turn on/off the new implementation at will. If things go badly - just turn it off and the old code is running instead. When the new implementation is deemed "good" it should become the default; we remove the configuration flags, the old implementation and possibly any other "complications" left in the code. This is properly managing risk by providing an escape hatch without needing to revert, rebuild, retest, redeploy anything.
3. Another way to do it is of course to make the changes in small increments and use automated tests and verification every step of the way. Jez stated it wonderfully: "...we have to work in very small batches. This is antithetical to the way lots of developers like to work: sitting off on their own going down a coding rabbit hole for days before re-emerging. The elevation of "flow" (by which I mean individual flow, not lean/team flow, which is actually inhibited by this behavior.) [is much at fault here]. Trunk-based development is about putting the needs of the team above the needs of individual. The premise of CI and trunk-based development is that coding is fundamentally a social, team activity. This presents a challenge to the mythos of the developer-as-hero which is still pervasive in our industry."
https://twitter.com/jezhumble/status/982988370942025728
1
-
1
-
1
-
1
-
1
-
1
-
1
-
@prdoyle Wall of text incoming - duck and cover!
1. Sure, but it is not integrated until it's ON main.
2. Even if you rebase your branch, you still only check integration with what has made it into main this far. Everything else that is not on main is an unknown.
As an example: If everyone "merge into main" at the end of the day or end of sprint, you won't know if everything actually integrates until that time. That is way to intermittent and is similar to old school waterfall "integrate and test at the end", although at least it's a more frequent than once per year. If everyone merges their code into main several times per day - fine - but what do you need FB/PR for in the first place if it only lasts a couple of hours anyway? Seems like a lot of ceremony.
I have actually been there, twice, kind of - in a highly compliant industry. We had armies of testers, reviewers, configuration management people, merge bosses, process and branches, and yet we had very expensive production issues all the time. Then I took over as tech lead for the most certification-critical components. The only changes we made was that we skipped all the ceremony/merges/handoffs and started doing proper XP/CI/TBD and our response time became 10x better and our production issues went down by 2 orders of magnitude. That alone saved the company $2M per year. A few years later a bunch of managers decided that "everyone must do FB/PR because we have to be compliant (BS, yadda, blah) and we read on the internet that that is everyone else is doing so we should too" and it became dogma and enforced by people not doing the actual work. After that we had slowed down and were right back in the tank and half of us had left within a year.
*We didn't need FB/PR - we were doing better without them already!*
Because so many are used to treat version control as a kind of personal junk-drawer where we throw things into a branch, fiddling around for a while, have it being broken for hours/days, flaky and unstable and then pick up the pieces into a whole at the end, then squash merge and PR; they get stuck in that cycle.
1. This is often a sign of bad design or issues with the culture.
2. There is a world of difference between "half baked", "incomplete" and "messy crap" - in XP/CI the software should still always work properly.
3. Code reviews are primarily there for learning and feedback. Async blocking reviews actually slows the organization down and have questionable effect on the quality. And even so not all changes requires review and not all reviews have to be blocking - see Martin Fowlers 3 types of reviews "ship-show-ask". FB/PR proponents are stuck 100% on "ask". You can always refactor and improve things after the fact, after it's on main, and even after it's gone live. Something people seem terrified of.
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
@vyli1 I have a question. What does it matter that trunk is broken just once-in-a-while (meaning seldom)? If it's broken so that unit tests, functional tests or acceptance tests fails it will be discovered (and fixed) quickly. If it's broken in such a way that it's only found during manual testing, you would probably have that happen anyway, and sometimes it's broken without us knowing - we call that a production bug.
There is this misunderstanding that "always releasable" means "mainline never broken, always pristine, represents production, etc". "Always" is not black and white! While we should strive to minimize the time mainline is broken (because it blocks everyone else) the goal is not to eradicate all mistakes (impossible). The purpose of CI, CD and TBD is fast feedback IF we broke something. It's okay if it was due to an honest mistake, if it was due to sloppiness then the team have some difficult learning to do. In both cases we have learned something we can act on. To para-quote Jez Humble: "Every build is a release candidate. The goal of the CI/CD pipeline is not to prove that it works, it is to prove that it does NOT work. If you can't prove that it doesn't work you should be able to deploy that version (if you choose to)." .
As I have written elsewhere, you don't release a branch, you don't deploy a branch, you deploy a good known commit/version of your code. If the mainline is broken the same hour that we need to deploy to production - it doesn't matter - because we always use/deploy the latest known working commit/version. Version control is not a file server and a branch is not a directory, it is a versioned graph of snapshots in time so we can use whichever one we'd like.
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
Copy-pasting my usual long rant regarding the belief that "we must have reviews to stop bad code from other bad programmers getting into the code base".
I think there is this idea that we programmers only get one chance to implement something. Everything is a one-off "task/project" and then we move on to the next thing. Nothing is ever perfect, learn to accept that and then do something about it. Life is messy, sh*t happens, deal with it and move on.
Committing unperfect code, unperfect design, janky solutions (that nevertheless works) is actually OK. What it requires is that people actually have to learn to refactor, redesign, and do continuous improvement *all* the time. It's never perfect, it's never done, so get into the mindset of always fixing and improving things that looks "off". It is a good idea for the team to have recurring reviews of the current state of the software and always improve issues as they are found. Keeping the code clean and working is more important than adding more features, or following a project plan or some rules put down by people not doing the job, all separated by 5 levels of management.
The second problem is that every programmer appears to believe that programming is a solemn activity. That is wrong; it's a social activity. In a company setting I see FB/PR as a symptom of missing teamwork and/or bad organizational leadership. If every developer wants to work alone in their corner with headphones and only intermittently "communicate" and "integrate" using PRs and "do their own stuff" you do not have a team. Don't you talk? And yes, we love to use dumb excuses like "but I'm an introvert", etc which is just BS to avoid having to talk with other people while getting paid to do our hobby.
1
-
Not looking for an argument here, just wanting to point some things out.
CI and TBD is designed to "solve" the problems you describe by exposing them instead of trying to work around them with band aids, processes, phases, hand-offs, etc. I would go so far as to say that the whole point/secret to (agile) software development has been forgotten in-place of process-itis and navel gazing. Just my opinion.
Here is a great quote about agile "Agile's biggest strength lies not in solving problems, per se, but rather in exposing your buried problems and making them visible so you can deal with them. For a team switching to agile it can feel like things are getting worse, not better, but that’s often because they’re being made aware of pre-existing problems they didn’t know they had. It may not seem like it but this is actually a good thing." - Someone over 10 years ago.
Or in the words of Jez Humble (co-author of continuous delivery) "If it hurts, do it more frequently, and bring the pain forward."
Instead of getting bogged down in the thinking that "we can't" or "we must", what if you asked yourself a different question? "What would need to change for master/main to no longer be blocked/deployable for new features, fixes, etc; to be able to do any of these things as needed, when needed, and never be more than a few minutes from being able to deploy to production?"
Problem-solving is more about asking the right questions than providing solutions. Adding complicated processes to hide the systemic issues is seldom the answer.
1
-
1
-
1
-
1
-
1
-
Yes, lets!
Like all the people that learned to code alone at night pumped up on cola. Like all the the people that can't collaborate, work together, interact and solve problems together. Like all the people that wants to sit alone in their corner, typing code and get paid to do their hobby. Like all the people that thinks that software engineering is just about tech, has no emotional intelligence, that don't understand that soft skills are 80% of the work and that always blames "introversion" why they can't. And all the people that's driven by nothing but ego, personal gain and only got into it due to the money. You mean, like those people?
You are right, there are a lot of idiots in the industry, it's just not the people you think.
1
-
1
-
1
-
1
-
1
-
1
-
It scales very well (given good tooling) Google is doing this - 20,000+ developers, billions of lines of code in a single repo, on a single branch. They do have review-gates, but there are no branches or pull requests involved as such. And the majority of checks are automated...
That the commit-build takes an hour has nothing to do with TBD or PR. That is always bad tooling, bad design and bad tests.
CI means the original as written, not whatever people redefines it as so they can put CI on their resumé. See wikipedia for example. Read Extreme Programming Explained.
Lot's of WIP and "not-done" changes waiting for review, checks, etc. is "not a good thing (tm)". I suggest reading The Goal by Goldratt and The Phoenix Project by Gene Kim, as well as other books in that space.
Management: "All hands on deck", "what do you mean, having two people at the same keyboard?", "Why aren't everybody busy, typing code? Do we need to fire people and trim the fat?", etc. Sigh. Yeah. I hear you. Get a new job or something.
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1
-
1