Comments by "" (@ContinuousDelivery) on "Testing Strategy for DevOps: What to Test and When" video.

  1. Thank you. Good luck at ThoughtWorks, I hope that you have some fun.
    10
  2. Thanks. Of course "it depends". For some systems "5 minutes" sounds a ridiculously short time, for others it seems way too long. I think that 5 minutes is a reasonable upper bound, but always prefer it to be faster.
    6
  3. Thanks 😎
    3
  4. Thanks
    3
  5. I suppose it depends on how far you take unit testing and what you mean by the percentages. For a feature I'd generally expect to create a handful of acceptance criteria and an automated "Acceptance Test" for each. If you take my approach, most of these "executable specifications" will reuse lots of test infrastructure code and will usually add little new code. The test case itself is usually a few lines of code written in your test DSL. Unit testing is driven, for me, from TDD. So I'd create unit tests to support nearly all of my code. So I'd have quite a lot more code in unit tests than code in acceptance tests, though the testing infrastructure code for acceptance tests will be more complex. One that basis, in terms of effort then something like 70% unit vs 10% acceptance is probably about right, though a guideline rather than a rule to stick to. If you count tests, then I think it is harder to generalise. Some features, may already exist by accident, so you will write an acceptance test to validate the feature, but don't need to write any additional code or unit tests. Unusual, but I have seen it happen. Other code may need a simple acceptance test and loads of work, and so loads of unit tests, to accomplish. I confess that I am not as big a fan of the test pyramid as some other people, in part for these kinds of reasons. I think that it can constrain people's thinking. However, if you see it as a rough guide, then it makes sense. I would expect, as an average, over the life of a project, for there to be more unit tests than acceptance tests, lots more. The danger, and a trap that I have fallen into on my own teams, is that the acceptance tests are more visible and more understandable, so there is a temptation to write more of them. QA people for example, often say to me "we can't see what the devs do in unit tests, so we will cover everything in acceptance tests". This is wrong on multiple fronts. 1) it isn't the QA's responsibility to own the testing or the gatekeeping 2) its an inefficient way to test 3) it skews the team in the wrong direction, if the QAs test "everything" in acceptance tests it will be slow, flaky and inefficient but it will nevertheless tempt the devs to relax their own testing, and abdicate responsibility to the QAs. Ultimately I think that unit testing is more valuable as a tool, but that acceptance testing gives us insight and a viewpoint that we would miss without it.
    3
  6.  @portlyoldman  No, I think of myself as a big fan of ATDD, but that is not instead of TDD, it is as an adjunct to it. I don't think it a good way to specify the pixel-paint part of a UI though, for me it is a wholly behavioural approach. I want exec specs that define the behaviour of my system from the perspective of a user. The I will use TDD underneath until that spec is met.
    3
  7. Yes, me being lazy with my language. Testing is best treated as a "falsification" not a "validation", 💯!
    2
  8. I have worked on teams that did scheduled releases at various frequencies. I that that some of the costs that you mention, like “3 days merging and testing PRs” highlight some of the problems. In general my advice is, even if you run a scheduled release approach, the best way to do that is via Continuous Delivery. That is, work so that you software is ALWAYS ready for release. Shorter release-cycles are better, because they force you to address pain-points. If you sped-up, you would be forced to address the “3 days” and replace it with something better - True Continuous Integration is the best approach, getting to something that works at least daily. CD is the next step after that, getting something that you “could” release at least daily.
    2
  9. Probably not for pacemakers 😳 but it is certainly the most realistic.
    2
  10. Thanks.
    2
  11. Yes, that’s how I think of it.
    2
  12. I think it depends on the nature of the testing. Some of the kinds of testing I talk about are consistent, certainly the release-focused testing is largely the same and run on each release. Some of the product-focused testing is too, probably the more technical kinds of tests and monitoring that you run. The very product focused testing like A/B or other kinds of Biz performance monitoring are probably more targeted and variable in time. The best way to organise this is to think of each release as a small experiment. If that's the case, then part of each new feature is "how do I collect the results from this experiment?" so you build-in the case-specific data-collection to the feature.
    2
  13. Easier than that I think, I mean that if all the tests pass in the commit stage, 80% of the time, all the tests will pass in every other stage of the pipeline too.
    1
  14. No I prefer to run them all the time, but not necessarily on every commit. I divide deployment pipelines into 3 phases, commit, acceptance and production.  The commit phase runs on every commit and is optimised to give fast feedback. Its job is to keep development flowing. Acceptance is about determining releasability, tests are more complex, more "whole system" and so slower.  Prod is about release and monitoring and gathering feedback from production.  Let's imagine commit takes 5 minutes and acceptance 50. So during a single acceptance test run, we may have processed 10 commits. When the acceptance run finishes, the acceptance test logic looks for the newest successful release candidate - generated by a successful commit run, and deploys that and runs all the tests. So Acceptance testing "hops over" commits, which is ok because they are additive, the 10th commit included all the new features of the previous 9, so acceptance testing is testing everything as frequently as possible.
    1
  15. Thanks
    1
  16. For my description I don't have a 'sheet' - yet, but I probably will release this as an information sheet to people on my mail list, if you sign up for my ATDD guide, here, we'll let you know when the info sheet in this video is available. https://www.subscribepage.com/cd-atdd-guide The one from Cindy Sidrahan that I mention is already linked in the video description.
    1
  17. Wasn't SODR it was this: https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
    1