Comments by "" (@retagainez) on "Software's HUGE Impact On The World | Crowdstrike Global IT Outage" video.

Canary releasing is immediately the first overall strategy that I thought could've heavily mitigated many of the global effects of the outage. I'm speculating entirely from this point on, but I believe it couldn't be rolled back since the interaction between Crowdstrike's software interacts with Windows at a low-level and causes the O.S to be inaccessible once the software deployment went live. I believe they have to delete a file through some relatively straight-forward procedure, but it requires manual intervention to some degree and so it is painstakingly slow to recover everything when the downstream IT organizations don't have great remote controls over each individual instance of Windows. Indeed, it is most curious how and why early detection of such a blatant issue was not present here. Perhaps, some testing was neglected?
6