Comments by "" (@grokitall) on "Diving into the embarrassing engineering behind CrowdStrike" video.

only if kernel level code is already compromised.
2
the author of that video is an idiot. he claims being adware makes it a scam, when it was a common way to get funding from freely available software at the time. he claims being nagware to update to the full version makes it a scam. he uses his ignorance of the well known problems with the registry to claim it makes a registry cleaner a scam by definition. crud left behind leaves a raft of problems. he claims that a common shareware antivirus was a scam, just because it was adware. he admits that dave was running a shareware market place, but then says all of the shareware was by dave, and his previously misidentified scams make dave a scammer, despite that not being how a market place works. finally, dave got a nuisance lawsuit, and like most companies, it was cheaper to settle than to fight. but he claims that by settling dave admitted to everything in the claim, despite the settlement agreement which ended the lawsuit not saying that, and then claims that because of that when he says the lawsuit was mostly meritless in his autobiography, which agrees with the settlement agreement, he is trying to hide being a scammer. i am nit saying he could not be a scammer, but literally none of the claims in the video actually pass basic fact checking.
1
 @keithjohnson6510 it was a kernel bug, but if they had been testing against either linuses nightly build, or gkh's next kernel, they would have found it before it even shipped.
1
you cannot recover from the initial crash, you can recover from the boot loop which kept all the machines down. as to the disclaimer, leonard french covered it on youtube, and it is invalid in the case of negligence, which this clearly is.
1
yes, but your validator and driver both use code from the same library. the validator just puts a cli wrapper around it which lets it return true or false, so you can call it from a script or test runner.
1
@ПётрБ-с2ц i just said how we have known to do it for decades. given all the other basics they ignored, i agree they probably ignored this one as well, but it does not matter, as they wrote it to pass unless it recognised a fault, as opposed to failing unless it recognised correct code.
1
he was referring to the way antivirus software signature files do not have time to go through driver certification, and the same applies even if the files can spot a lot more. this is the whole justification cloudstrike uses for live updating the files in the first place, so it is relevant. he also states he is a web developer, which should give him a little wiggle room for not getting some of the subtleties which were irrelevant to what caused the outage.
1
@Xehlwan the truth has now come out as to what happened. they created a file with a proprietary binary format. they ran it through a validator designed to pass and only to fail known bad versions, then when it passed, immediately pushed it to everyone with no further testing. what should have happened it this: create a readable file in a text format which can be version controlled, test it, and commit it to version control. generate the binary file from the text file, with a text header at the start (like everyone has been doing since windows 3.11), and immediately create a signature file to go with it. have the validator compiled as a command line front end around the code used in the driver, designed to fail unless it is known to be good. this checks the signature, then looks for the text header (like in a gif file), then uses that header to decide which tests to run on the file, only passing it if all,of the tests pass. run the validator as part of your continuous integration system. this tells you the signature matches, the file is good, and all,other tests of the file and the driver passed, so it is ready for more testing. build the deliverable, and sign it. this pair of files is what gets sent to the customer. check the signature again, as part of continuous delivery, which deploys it to some test machines, which report back a successful full windows start. if it does not report back, it is not releasable. then do a release to your own machines. if it screw up there, you find out before your customers see it and you stop the release. finally, after it passes all tests, release it. when installing on a new machine, ask if it can be hot fixed by local staff. use the answer to split your deployment into two groups. when updating only let the fixable machines install it. the updater should again check the signature file. then it should phone home. if any of the machines don't phone home, stop the release. only when enough machines have phoned home does the unfixable list get added, as it is more important they stay up than that they get the update a few minutes earlier. if any of this had happened, we would not have even heard about it.
1
no they were not. the information was clear. the initial proposal was to make every other vendor of antivirus except microsoft have to use a different procedure, giving microsoft an unfair advantage in that market. the eu said no, then microsoft chose to remove the method instead of using it themselves. it was entirely up to microsoft how they resolved the issue.
1
@TreeLuvBurdpu windows defender is not an alternative to crowdstrike, it does a lot less. it is more of an alternative to norton or mcafee. as to the eu, they said you cannot have one set of rules for everyone else, and a better set for you, especially when we are already having issues with your monopoly abuse. microsoft could have made windows defender use the same apis as everyone else, but instead chose to rip out the entire subsystem and keep everyone in kernel space. that was Microsofts choice, not the eus.
1
@TreeLuvBurdpu sorry, but apple did not try and implement a 2 tier system to get an unfair advantage over their partners, so were not blocked. the regulations in this case exist to protect against the abuse of unfair power structures being abused, which is why apple is being investigated for taking a 30% cut of the sales price from all partners in the app store while also blocking the use of side loading and alternative payment methods. note: i am not saying that the regulations are good, only that they said you can't do that, and microsoft made a free choice of what to do instead. they could even have said that they were making their proposed api available at launch, but not mandatory until service pack 1, at which time defender would also use it, but they chose not to.
1
their own report says that they only ran the validator, and one it passed, immediately pushed to everyone. this bypassed all the other testing they do when they update the driver.
1
 @keithjohnson6510 the ceo said that all at once was company policy.
1
so is yours. in a mature process, the driver is signed, then put through a validator designed to fail it as part of the ci process leading to a release, which includes installing it on test machines. the signature says that the file the validator checked is the one you are about to ship. they can't have done that, as the zero file would not match the result in the signature file, and the client side update program would not have installed it if it did not match. then your validator should be designed to fail unless everything is found to be ok. then a new template like the one used could not even have been put where it could be used until the validator could pass it. this validator was designed to pass unless it spotted a known failure, which is not how you write them. then their custom binary file format did not contain a signature block at the start, despite this being standard practice since windows 3.11, and before the first web browser. then having passed the broken validator, they bypassed all other testing and shipped straight to everyone. obviously nothing can go wrong, which misses the entire point of testing, which exists not to prove you got it right, but to catch you when you get it wrong.
1
this is because the driver, while rubbish, was not broken. as soon as the broken file was deleted it worked fine. the signature file was designed to live update, so driver versioning does not help in this case.
1
the update released contained nothing but zeros. the ceo said that it had nothing to do with the problem. he then said the fix was to delete the file, proving himself to be either incompetent, a liar, or both.
1
@dominikdoom also, the eu just said "that is monopoly abuse and does not fly here, have another go". microsoft then decided to roll back the change rather than keep it and play on a level playing field. not the eu's fault.
1
@everyhandletaken the guy who shot the video makes a lot of allegations that things he does not like or does not understand makes everything on a shareware market place dave used to run a scam. the then goes on to say that everything hosted there was therefore dave being a scammer due to this, and that dave settling a nuisance lawsuit rather than fighting it al, the way makes him guilty of covering up being a scammer, despite that having nothing to do with what the settlement agreement actually says. the whole video was total bs from someone with a total lack of understanding of the subject he claims to be an expert on.
1
there is nothing you can do to stop a bad driver from causing the kernel to crash. there are lots of things you can do to stop the boot loop, which is what might leave microsoft on the hook as well. first you have windows write a flag to storage as soon as it is able to say it started booting. then you have it write over that flag which driver it is starting. then when it finishes booting, you write over the flag that it finished booting. then the kernel crashes and the system reboots. the windows system then knows that it crashed because the flag does not say it completed. it also knows which driver broke it, and can disable it. it can also treat the boot start flag as a request, and have an internal table of the few drivers like the filesystem which can't be disabled. after the crash it can downgrade the boot start flag internally so that when it crashes again, it can be disabled. if the driver recovers, it can be re-enabled on next boot. this gives the driver the chance to recover on reboot. they can automatically add drivers to the internal essential drivers list during certification by simply replacing the driver with a return statement, and seeing if it fails to boot. if it does, it cannot be blocked and is added to the list. they can then disable the driver on reboot, or second reboot if it is boot start, and put i huge warning on the screen that the broken driver was disabled, causing the customer to question why the broken driver was released. this could have been done by microsoft or any other os vendor after any of the previous high profile boot loop issues, but they did not. and the eu thing is just more microsoft misinformation.
1
@tma2001 if the channel file was different on different machines, it tells us something. they did not bother to sign it, or ship a signature, nor check such a signature when it was deployed before putting it into place. an update should be deterministic. it also tells us something else. they only had one deployment server, which the result implies got some sort of disk fault. with 8.5 million machines affected (according to microsoft), you should be using more than 1 for load balancing, and the one with the faulty update would be ignored after the checksum failed. the update client could even phone home and report it.
1
@tma2001 cloudstrike have made a number of dubious statements, some of which are obvious lies or the person saying them is clueless. take your statement about the update file. cloudstrike said it basically had nothing to do with the issue, but if you remove it, the problem goes away. both cannot be true. then there is the issue of it not containing all zeros, but lots of it guys have looked at the contents before deleting it and found it only had zeros. giving them the benefit of the doubt which their own statements say they don't deserve, even if the file contained a header, they obviously were not bothering to have the updater validate it prior to putting the file in place, nor having the kernel driver do so before blindly trying to read it. both are standard practice. similarly, their own statements make it clear that their only filter to shipping was running it against an obviously badly designed validator, and then skipping any other testing. for something running in kernel mode, every change should go through the entire test suite every time, and shipping it how they did should not even have been possible. even their public statement of what they intend to do to make it less likely in the future basically left people saying why were you shipping at all if you were not doing those things already. nothing about the information coming from cloudstrike makes them look good, from a single developer being able to live patch 8.5 million machines without testing, to a validator which is designed to pass everything unless it recognises specific things to be broken, to a minimal testing environment for the full driver, to not doing canary releasing. non of it makes them look good, and then having their idea of compensation for causing millions in damages being a generic 10 dollar gift voucher with uber eats, which promptly got cancelled because it looked like fraud because they did not talk to uber eats, it just makes you ask how much longer until they do anything right.
1