Comments by "" (@grokitall) on "ForrestKnight" channel.

@oshochso I'm sure there was a lot of rubbish software claiming "just run me and i will make you feel better", but the actual problem was much harder. i had a gps device where the hardware had this fault, even though the 2 byte window could just as easily been set to start from the date of first manufacture. lots of embedded hardware had the same software issue. not all of it could be updated as we were still using roms and eproms. the bigger issue was all the legacy software which used a date anywhere and had to be audited so it did not fail by surprise when a y2k date was needed. and some of it did still fail. the first warnings about y2k were raised in the 1950s, and they just kept ignoring it until it had to be done as an emergency.
4
but that assumes that they have a decent testing and deployment strategy, despite all the evidence to the contrary. to paraphrase terry pratchett, in the book raising steam, you can engineer around stupid, but nothing stops bloody stupid! 😊
3
with the amount of money lost, i am sure plenty will try, and bear in mind that a lot of the companies hit were fortune 500 companies with their own teams of lawyers, it just depends if they can find a venue which will exclude their choice of venue and other weasel clauses.
2
@diogotrindade444 every binary file type developed by microsoft includes an id type signature at the beginning, which can be checked so the file won't load if it is wrong. that would have stopped this. canary releasing from cloudstrike would have minimised the scope of this. proper ci and cd would have stopped this. a proper bad driver detector which just checks if it updated and crashed windows would have made the fix just turn it off and on again. not installing software which stopped you from checking it internally before updating every machine would have stopped this. plenty of blame to go around.
2
@ChristianWagner888 they did do something unusual. their installer downloaded an update which it then did not sanity check prior to putting it in the right place to use. then their driver basically assumed that all updates would be sane, and further failed to have any mitigations in place to stop it causing problems if it was not. their are additional mistakes they made to even let this get out into the wild, but that is a separate issue.
2
@ChristianWagner888 i agree, but antivirus companies in general live patch their signature files, so you don't get the luxury of waiting. and being on the previous version only works for the driver if it has the problem. this hit the live update, and so affected every version of the driver.
2
but the machines staying down is directly due to microsoft. if they had looked past safe mode and implemented something to detect and recover from bad driver updates, then it would have been a simple case of turning the machine of then on again and letting it recover.
2
the buggy code was downloaded, but did not break anything until it rebooted. at that point windows should have stepped in and on next reboot it would just be disabled. then to fix the machine, just turn it off then back on.
2
we have known how to write software so this does not happen since the moon landings. we have even better tools now. the only way for this to happen is for everyone including microsoft to ignore those lessons.
1
@JamesTSmirk87 of couse if you canary release to the test servers first, then to your own machines, and only then to the rest of the world, it would have been caught.
1
every os which allows third party kernel level drivers could do this. the only solution for locked down systems is to detect updates and block those drivers when the reboot fails.
1
it was this severe because it was kernel level driver code. but we just test it the same way as any other important code, with ci, cd, and canary deployments.
1
@jedipadawan7023 not quite. bsd was licensed from at&t, so was xenix. linux is a unix workalike, which is just api compatible so it can run unix programs this means that it does not need a license.
1
@cybervigilante this is not a vendor issue, it is a kernel space issue. nvidia could do this to linux tomorrow for people with nvidia cards. also, cloudstrike did the same thing to debian and rocky recently.
1
then you are lucky. this affected lots of stuff worldwide, including doctors surgeries, emergency rooms, operating theaters, ambulance operators, epos terminals in lots of shops, lots of places who took card payments, cash machines, banks, airlines, hotels, petrol stations, and the list goes on. this was a massive cock up, and it was 100% avoidable using methods learned starting with the moon landings, going through y2k, 9/11 and right up till today. not having this happen is trivial, they just did not bother to do it.
1
debian, mint, manjaro spring to mind.
1
@Songfugel because microsoft decided that safe mode was good enough and did not bother to implement a bad driver detection and disabler. not that any other os does either, but they don't outsource kernel driver development to all and sundry.
1
@Songfugel i agree, when the driver screws up, you have to panic and reboot. microsoft chose to allow lots of third parties to have full kernel access for their drivers, and then have minimal support for fixing it after the reboot. even more so for locked down machines which operate like embedded systems. as a major os vendor, microsoft have some responsibility for how their system behaves when broken drivers are installed, and the boot loop issue was entirely of their making.
1
@theswordslay3542 except safemode does not work for locked down systems. this is why every tech support is having to go out, unlock the system, do the trivial fix, and then relock the system. for locked down systems you cannot stop a bad kernel driver update causing a panic, but you can easily add something to spot an update, and something else to spot the reboot crashed, and disable all updated drivers so it recovers just by turning it of and then back on. this is a problem for any os who allows third party kernel driver updates.
1
@JeanPierreWhite as a malware mitigation company, new threat updates are necessary often, so there is implicit consent when you install the software. as to having the software, some commentators have said that this was not optional due to either their insurance companies, or their regulators. their only choice would be to use a different supplier, who you might not be able to determine is any better.
1
small point you seem to have missed is that a lot of the affected machines were locked down machines which might not even have a keyboard. this means the technique of just go into safe mode and fix it yourself does not work. your comment reminds me of the old pcs who by default ship with the bios setting which resulted in the error "keyboard not found, press f12 to continue booting".
1
@bzuidgeest some systems with this system installed fit those requirements, but commentators who were on the front lines handling this have said that in practice these are locked down systems with a lot of security, and to implement the trivial fix they actually had to turn up, plug in a keyboard, use secret keys to downgrade security, go into safe mode, do the trivial fix, and do the same thing in reverse to make it secure and locked down again. there are major security issues with the idea of handing those security keys to the very people it was implemented to keep out.
1
rubbish. this is kernel level code, and the wrong type of bug will crash the kernel on any operating system. the problem is it should never have been deployed, should have immediately stopped deployment when the machines started crashing, and windows needed a better response to a broken driver than just put your locked down machine into safe mode and fix it yourself.
1
 @JeanPierreWhite the issue is not that they had no recovery mechanism in place, it is that go into safe mode and fix it yourself does not work with locked down machines.
1
right in principle but the practice is only just a little bit harder.
1
don't he can make a lot of money doing this sort of thing how it should be done, which we already have known for many years.
1
@bzuidgeest i wish people would stop giving microsoft a get out of jail free card on this one. every os is vulnerable to this, as long as they can load third party kernel drivers. the problem for microsoft is that nearly every driver is provided by someone else, that is why they had to introduce safe mode in the first place. it just does not work with locked down devices, which is why you have lots of people having to manually go around and fix every machine. an os writer could easily implement a system where it does a simple test to see if and drivers were updated, and if so skip to special updated driver code. that code could then on the first go around flag up the updated drivers, and on the second boot, disable any driver that was part of the incomplete boot attempt. if they had done that, the system would have gone down, and tech support could just say to turn it of and on again and it could boot. the cloudstrike user land process could then phone home and sound alarm bells over the broken driver, and look for a later update to fix it. then none of this would have been newsworthy.
1
no, but emergency rooms, operating theaters, pharmacies and ambulance services do, to name just a few.
1
it was shipped on a friday because it was supposed tomcontain virus signatures but it was not caught in testing because they did not do any, else it would have been caught.
1
 @robertbutsch1802 yes it is called having a windows installer program so you can install your driver.
1
 @sirseven3 they are also reliant on electricity, but that does not make it an electrical problem.
1
@sirseven3 i did not. to get out of the boot loop, you have to go into safe mode, but on locked down machines the local users have no way to do that. pxe boot would solve that issue, but would create a raft of new ones which is why a lot of these companies have chosen not to use it. others are just not aware of it.
1
@sirseven3 i was not advocating it, only saying it was an alternative which changes one set of issues for another one. it might work for your use case or not, but if you don't know it exists you cannot make the informed choice about using it or not.
1
@sirseven3 pxe boot only needs to get you from boot to the root shell, after that you can manage it with ansible to make it as user specific as it needs to be. still not recommending it, just saying it is an option which might fit, so you need to know about it. as to scaling, i have seen systems where it clears the memory on startup, and does a full reboot on every logout for hotdesking. this system did not even have persistent drive partitions, as it recreated them on reboot, loaded the operating system, let you login using remote authentication, installed any additional programs your user name required, and then stopped you from saving stuff on the local drive. a few of the machines actually powered down when you logout, so it did not even have persistent memory. your files were required to be stored on the network drive, which was then redundantly merged with the remote site, so the site could burn down and you would not lose your data. it worked for them, and the only issue they had was new users complaining either that they could not save locally, or moaning if it could but did not survive the reboot. they soon stopped trying to save locally.
1