Comments by "Daniel Sandberg" (@ddanielsandberg) on "Continuous Delivery" channel.

CD does not mean publish/release every commit! It means publish the latest version/build that is regarded as "good" at your choosing (by your context/definition). As the saying goes "If you can't deploy right now, it isn't Continuous Delivery". Rebasing does not work if everyone's changes lives in their own branch. You will end up with lots of changes that are unintegrated since there is nothing to rebase with.
1
"oooor you know, you have continuous integration for the feature branches" - nope, you can't, because that is not CI.
1
@vyli1 I have a question. What does it matter that trunk is broken just once-in-a-while (meaning seldom)? If it's broken so that unit tests, functional tests or acceptance tests fails it will be discovered (and fixed) quickly. If it's broken in such a way that it's only found during manual testing, you would probably have that happen anyway, and sometimes it's broken without us knowing - we call that a production bug. There is this misunderstanding that "always releasable" means "mainline never broken, always pristine, represents production, etc". "Always" is not black and white! While we should strive to minimize the time mainline is broken (because it blocks everyone else) the goal is not to eradicate all mistakes (impossible). The purpose of CI, CD and TBD is fast feedback IF we broke something. It's okay if it was due to an honest mistake, if it was due to sloppiness then the team have some difficult learning to do. In both cases we have learned something we can act on. To para-quote Jez Humble: "Every build is a release candidate. The goal of the CI/CD pipeline is not to prove that it works, it is to prove that it does NOT work. If you can't prove that it doesn't work you should be able to deploy that version (if you choose to)." . As I have written elsewhere, you don't release a branch, you don't deploy a branch, you deploy a good known commit/version of your code. If the mainline is broken the same hour that we need to deploy to production - it doesn't matter - because we always use/deploy the latest known working commit/version. Version control is not a file server and a branch is not a directory, it is a versioned graph of snapshots in time so we can use whichever one we'd like.
1
By "Never" he means - don't branch for development purposes. There are still business-cases where we for example might want to support multiple versions of the software (for example if it is customer installed applications), but at the speed software moves/changes today we might just as well create a fork of the application and manually backport bugfixes. There is a reason that companies charges a lot of money for subscription licenses - maintaining old versions is expensive. To answer your question: Branch by abstraction and a feature toggle. That is; create a "variation" of the pages/auth/config/whatever and have a flag that can be turned on/off if the new auth system should be used or not. This way we can turn on the new auth in some test environments but have it off in production and we can test and perform audit/penetration tests until happy. We can even just have the auth flag turned on a subset of pages to begin with and expand from there. Later on we can remove the flag if we want so that the app/site always requires auth. In CI/TBD/CD we don't equate a release with merge and deployment in the traditional sense. Deployments comes *before* release, meaning that unfinished features are deployed, but disabled/hidden until ready. This is how Facebook released their chat way back in pre-historic times. They had "javascript bots" running behind the scenes on a subset of users timelines; testing, measuring and logging stability and security. Later on it was rolled out to a subset of users for trial, that subset then grew and after a few iterations (and performance fixes) it was rolled out to 1 billion users. Note that they did not do a "all hands on deck and merge and deploy the universe and hope it can securely handle 1 billion users" on release day - it had already been running in production for over 6 months.
1
Well, there's a positive video title if I've ever seen one. :)
1
You can choose any build/version that has passed muster during the CD-process and send that. CD doesn't mean that you have to *deploy* all the time. It's the ability to have the option to deploy on demand or (in your case) to send any version you want to certify to external test houses.
1
Check out Kelsey Hightower's github repo "nocode". It's such a meme and the 3700 issues 500 PRs are nerdy funny.
1
@porky1118 "The function point is a "unit of measurement" to express the amount of business functionality an information system (as a product) provides to a user." https://en.wikipedia.org/wiki/Function_point
1
I think there is this idea that we programmers only get one chance to implement something. Everything is a one-off "task/project" and then we move on to the next thing. Nothing is ever perfect, learn to accept that and then do something about it. Committing unperfect code, unperfect design, janky solutions (that nevertheless works) is actually OK. What it requires is that people actually have to learn to refactor, redesign, and do continuous improvement all the time. It's never perfect, it's never done, so get into the mindset of always fixing and improving things that looks "off". It is a good idea for the team to have reoccurring reviews of the current state of the software and always improve issues as they are found. The second problem is that every programmer thinks to believe that programming is a solemn activity. That is wrong; it's a social activity. In a company setting I see FB/PR as a symptom of missing teamwork and/or bad organizational leadership. If every developer wants to work alone and only intermittently "communicate" and "integrate" using PRs and "do their own stuff" you don't have a team. Don't you talk? And yes, we love to use dumb excuses like "but I'm an introvert", etc which is just BS to avoid having to talk with other people and getting paid doing our hobby. (I'm an introvert and I love pair programming and co-located teams and problem solving with a bunch of people in the same room. I hate parties and "social crap".) Like Jesus fing Christ. We programmers need to get over ourselves.
1
Jepp. Ska vi starta nytt företag inom FinTech? :D
1
Eh, I suggest you google the name Dave Farley and Jez Humble before telling Dave he does not understand the goal of SW engineering, especially when it comes to the importance of quality. 😬
1
@OzoneGrif A few short points in response: 1. "Pushing untested features to master prevents you from publishing hotfixes" - No, it doesn't. CI/CD enables you to deploy to production at any time, even in the middle of development of 23 unfinished features. 2. There is no "delay for next release". In CI/CD we can deploy at any time (if we deem it necessary); there is no difference between a normal deployment or an emergency hotfix - it's the same process. As the saying goes "If you can't deploy right now, it's not CD". 3. In traditional SDLC/gitflow the granularity of a "release" is "a feature". In CD it is "the current version"; which may be once per week or 60,000 times per day (that's what Amazon is doing).
1
And if it's rejected you have to switch back, remember what you did maybe 3 days ago. Fun. People are obsessed with busyness and chalking off tasks from a todo-list. The problem is that we are really *not done* unless it's running in production (or at-least staging). But people love to "implement tasks", then pass it to PR/test and pretend they are done and move on to the next task, when in reality nothing is done. You should really read about one-piece flow and theory of constraints and how inefficient async blocking workflows actually are.
1
"There's absolutely no reason why you can't CI every single commit no matter what branch it's on." So what is CI then? Define it please.
1
How the h*ll do you lose code in git? It's impossible, unless you allow "force push/rewrite history". But in that case you just screwed yourself. Every SCM tool I've used has mechanisms to block force push + branch protection rules to block rewriting of history.
1
Instead of associating a branch with an environment we deploy a selected version (versioned binary or specific commit) to out environment of choice. Maybe the confusion is related to the thinking that branches/tags are viewed as directories on a file server and somehow represent the "readiness" of the code. The state of a particular commit/build/version IS the "readiness", not "where" it is in version control. In short: *a branch is NOT an environment* and the term "master represents production" is a misnomer and completely misunderstood.
1
Is there anything in particular you have questions about?
1
@Ryosuke19 That is one kind of pair programming called "ping-pong programming". It's a fun little game to get going when working on something boring or "where do we even start" kind of task. Love it.
1
@babgab Pair programming is not - two - people - programming. Pair programming is "two people programming". It's collaborative, not cooperative. It's the difference between having fun with your friends and having fun with your spouse. I don't really care for optimizing individual developers time. What I do care about is to optimize throughput and work not done - ie. if you're working with FB/PR and only having someone review it at the end, and it turns out it's the wrong thing, the wrong way, a misunderstanding, etc. at what point is it "rejected"? The rubber duck came in after all that time and effort had already been applied. With pair programming you'll get feedback much sooner. The solution is of course to make sure that the work done in the PR is very small, less than an hour... but then the question is why even bother with an FB and not just do proper CI and have someone come over to your computer/hangout/screenshare and review it before you push to main (or after push if possible)? Or do proper pair programming to begin with. :)
1
Copy-pasting my usual long rant regarding the belief that "we must have reviews to stop bad code from other bad programmers getting into the code base". I think there is this idea that we programmers only get one chance to implement something. Everything is a one-off "task/project" and then we move on to the next thing. Nothing is ever perfect, learn to accept that and then do something about it. Life is messy, sh*t happens, deal with it and move on. Committing unperfect code, unperfect design, janky solutions (that nevertheless works) is actually OK. What it requires is that people actually have to learn to refactor, redesign, and do continuous improvement *all* the time. It's never perfect, it's never done, so get into the mindset of always fixing and improving things that looks "off". It is a good idea for the team to have recurring reviews of the current state of the software and always improve issues as they are found. Keeping the code clean and working is more important than adding more features, or following a project plan or some rules put down by people not doing the job, all separated by 5 levels of management. The second problem is that every programmer appears to believe that programming is a solemn activity. That is wrong; it's a social activity. In a company setting I see FB/PR as a symptom of missing teamwork and/or bad organizational leadership. If every developer wants to work alone in their corner with headphones and only intermittently "communicate" and "integrate" using PRs and "do their own stuff" you do not have a team. Don't you talk? And yes, we love to use dumb excuses like "but I'm an introvert", etc which is just BS to avoid having to talk with other people while getting paid to do our hobby.
1
Not looking for an argument here, just wanting to point some things out. CI and TBD is designed to "solve" the problems you describe by exposing them instead of trying to work around them with band aids, processes, phases, hand-offs, etc. I would go so far as to say that the whole point/secret to (agile) software development has been forgotten in-place of process-itis and navel gazing. Just my opinion. Here is a great quote about agile "Agile's biggest strength lies not in solving problems, per se, but rather in exposing your buried problems and making them visible so you can deal with them. For a team switching to agile it can feel like things are getting worse, not better, but that’s often because they’re being made aware of pre-existing problems they didn’t know they had. It may not seem like it but this is actually a good thing." - Someone over 10 years ago. Or in the words of Jez Humble (co-author of continuous delivery) "If it hurts, do it more frequently, and bring the pain forward." Instead of getting bogged down in the thinking that "we can't" or "we must", what if you asked yourself a different question? "What would need to change for master/main to no longer be blocked/deployable for new features, fixes, etc; to be able to do any of these things as needed, when needed, and never be more than a few minutes from being able to deploy to production?" Problem-solving is more about asking the right questions than providing solutions. Adding complicated processes to hide the systemic issues is seldom the answer.
1
Why does everyone of my comments get marked as spam?
1
Umm, refactoring?
1
"it's difficult for management to accept it", yeah about that... In the end they hired *you, the engineers* that *knows* how to make software, management don't. You should be able to tell them "You don't pay us to type, you pay us to make software and this is how we make software, we pair and collaborate." - Ask them if they would hire a doctor and then tell the doctor how to perform the surgery. - Ask them if they would hire a finance auditor and then tell them how to do their job (that's how the CEO and CFO ends up in prison). - Then ask them why they would hire smart people and engineers and tell them how to do their jobs and still blame the engineers when sh*t hits the fan. But I get it.. politics... you can't just ask serious questions... I'm just an angry "naive unprofessional ideologist" so what do I know... PS: Never heard about PSP. :)
1
Ok, explain to us what you think CI is then.
1
Yes, lets! Like all the people that learned to code alone at night pumped up on cola. Like all the the people that can't collaborate, work together, interact and solve problems together. Like all the people that wants to sit alone in their corner, typing code and get paid to do their hobby. Like all the people that thinks that software engineering is just about tech, has no emotional intelligence, that don't understand that soft skills are 80% of the work and that always blames "introversion" why they can't. And all the people that's driven by nothing but ego, personal gain and only got into it due to the money. You mean, like those people? You are right, there are a lot of idiots in the industry, it's just not the people you think.
1
Why would it take "some time to get the code into a compilable state"? Just throwing it out there; what would need to change for things to always be in a working state?
1
Yes. Version control is not our personal backup system. So if you can't finish before going home and you somehow lost it? Who cares? I can say by experience that starting over in the morning after a good nights sleep makes for much better code/solution than trying to pick up half-broken stuff I did at the end of day yesterday. So what is more important; "saving" the hour from yesterday, or better solutions? There are lots of people in the industry that thinks that every feature/solution should be implemented twice. By the time we write it the second time our understanding has increased and our ignorance has been reduced. There are companies out there that have scripts that deletes all local clones at the end of the day so everyone has to check out from scratch every morning. And just wait until you hear about the Test-Commit-Revert workflow. :)
1
@Omego2K That wasn't my point and had nothing to with CI directly. It was just an example in trying to empathize that it's NEVER about individual programmer productivity. Just that.
1
Unless it's merged, built, tested and verified on *mainline* - it's not CI - by definition. CI is not "a server that does stuff with code", it's a practice. Maintaining multiple versions (for bug-fixes) can be done easily with release branches where changes are either merged in or backported from master. Having a separate testing department is death to CI/TBD.
1
I promise you there is no regulation that says that you have to use feature branches, pull request and that it must be reviews by a senior dev. It is the organization that has interpreted the compliance requirements [developers shall not be able to make and deploy changes without proper attestations] that way because it's easy for the people in charge to just make blanket edict, clap their hands and declare "we are compliant". It's lazy and damaging to the quality and performance of the organization. Not that they would ever understand, since they are not the one's doing the job. Dave just posted a video about alternatives.
1
It scales very well (given good tooling) Google is doing this - 20,000+ developers, billions of lines of code in a single repo, on a single branch. They do have review-gates, but there are no branches or pull requests involved as such. And the majority of checks are automated... That the commit-build takes an hour has nothing to do with TBD or PR. That is always bad tooling, bad design and bad tests. CI means the original as written, not whatever people redefines it as so they can put CI on their resumé. See wikipedia for example. Read Extreme Programming Explained. Lot's of WIP and "not-done" changes waiting for review, checks, etc. is "not a good thing (tm)". I suggest reading The Goal by Goldratt and The Phoenix Project by Gene Kim, as well as other books in that space. Management: "All hands on deck", "what do you mean, having two people at the same keyboard?", "Why aren't everybody busy, typing code? Do we need to fire people and trim the fat?", etc. Sigh. Yeah. I hear you. Get a new job or something.
1
@AlanW I don't work at "the company". This is very well documented in articles as well as on youtube by employees. PS: My original comment was instantly deleted by "the company's comment bot" due to being negative about "the company". LOL!
1
So instead of actually trying to understand and argue the point, you resort to ad hominem attacks?
1
You don't seem to really understand the economics of software engineering. Pair programming does not costs twice as much. You have to look at it from a holistic perspective over time - the amount of bugs, the knowledge sharing, the amount of rework, the quality of solution, the clarity of code, the amount of unnecessary work not done, etc... Having another person, not doing the actual job and only there to review is the same old trap we fell in when we decided to separate architects from developers from testers from UX from the business and maintainers from programmers and... It's all just reductionism and excuses... Focusing on efficiency is a bad strategy at best and damaging at worst. What matters is throughput, sustainability over time and the ability to produce the expected result within reasonable time and cost - i.e. effectiveness. There is an old adage in the software industry - "if you want to go fast, you need to go slow".
1
@KyleGobel Yes, I know I may come off as "abrasive" and counterproductive. It's not personal, I'm sorry. I'm just tired of everything in IT right now... It's more complicated than that... having another person there is best done as the job is being done, not after it is done. What I mean is that all those things I listed comes from the practice itself. The effect is emergent, not strategic. See it like this: On paper it is cheaper to have a single surgeon doing the surgery. Having to pay extra for nurses and anesthesiologists seems inefficient. Right? But if we look at it from a holistic perspective and look at the amount of mistakes that requires more corrective surgeries, the pain, medication and recovery time, the loss of quality of life and at worst the loss of life. Then the premise of having multiple people collaborating, helping and checking the work *as it's being done* suddenly seems like a good idea. - Efficiency / cost accounting - make "healthcare" as cheap as possible. - Effectiveness / throughput accounting - keeping and making people healthy and return to life - result.
1
CI is more than a build server that runs checks against your code. And if it's ran against feature branches then CI is not even in the scope anymore. Yes, I know tool vendors would like you to think that CI is "jenkins, circleCI, whatever"...
1
@adamhenriksson6007 Yes, but when you said "CI is run on submit" I assumed you meant "running automated verification on feature branches" - which is, by definition, not CI. I'm adamant about this because we have gotten into this bad habit of redefining things to mean "whatever we are doing right now". I know what you meant and I'm just trying to break people out of the illusion that they think they know what CI is and that they are doing CI when they aren't. It's a little like when teams spend a week writing all the tests, and then coding for another week and then calling that TDD. Just tired of everything right now...
1
This is a false dichotomy. If programming was just typing it would be correct. But programming is thinking about the problem, figuring out solutions, troubleshooting existing code and talking with other people to make better informed decisions. And yes, there already is. It's called "mob programming" or "ensemble programming". You should watch his other video named "You Must Be CRAZY To Do Pair Programming".
1
This is really hard. Databases are amazing and terrible at the same time. I hate them, kind off... Anyway, in my Java projects we made sure that our "monolithic" application was properly modularized and that each module was responsible for it's own database-schema. Modules communicated with each other over normal Java APIs instead of "through complex SQL-queries with tons of foreign keys, joins", etc. In some places we used Flyway with versioned DDLs, etc. In some other places it was home-built python-scripts that executed on-demand before the application started. This made it much easier to manage the database as well as extract "services" from the monolith. Was it perfect? Hell no. So we used blue-green "staging" to save our bacon and not destroy the production DBs. Can we stop using databases? :)
1
@pablog5738 For modules that changed a lot it was worth it (always deployable). For things that changed much slower we could probably have achieved the same with good process and change management. This was at my previous job with 1000 devs and 1000s of servers. My current job is a startup with 50 devs and 20 servers which has many other problems than schema versions. It will matter in the future, but for now we just have a single database guy that handles all the DLCM.
1
@pablog5738 It was 7-8 years ago. I don't remember and I was the guy that avoided databases until I really needed one just because they hurt and they drag you down with them. For applications that gets deployed a lot, many small changes to the DB-schemas is fine. It's like refactoring normal code - it's never done - many small changes, always deployable. Sometimes we had to "branch by abstraction" the database and just support old and new schema at the same time and other times it was a backup followed by a straight data-migration. For those with applications that has a much slower rate of deployment (weekly or whatever) you can still do many small changes of the schema v1.2.1, v1.2.2, v1.2.3, etc. during the sprint, and then at the end merge all those intermediate schema versions into a v1.3.0 (and delete all the v.1.2.x). That works fine when having a more static release schedule. Not really CI but YMMV...
1
What are you trying to say? It's not like Dave doesn't know this and have been promoting pair programming for many years. Please explain your comment.
1
What does it matter if the thing you did this morning is rewritten/removed a few hours or a day later?
1
@etherweb6796 I do not understand this answer! I was merely pointing out common misconceptions around CI/CD and TBD. Develop can't be your source of truth *unless* you are building immutable production artifacts from "develop" and testing *that* in lower environments. But then you can just skip it and call it "main". Anyway...
1
Was this supposed to be an answer to cugamer8862's comment? I agree anyway. :)
1
@michaelrstover Event Sourcing has the concept of snapshots. I.e. we occasionally inject snapshots in the sequence of events which represents the state up till that point in time. When we are replaying we only need to go back to the latest snapshot but if we want to we can ignore the snapshots and go all the way back to the beginning. To avoid the same book being ordered multiple times we make sure that all actions are idempotent, which is hard, but doable.
1