Do you find that the number of errors in Windows 10 updates has increased in the last two years? If so, do you know why? This question answers a recent question video former employee of Microsoft (Senior SDET) Jerry Berg, also known as Barnacules.
Berg has worked for Microsoft for 15 years and one of his roles was to design and develop tools and processes to automate tests in Microsoft's operating system. He left the company after the release of Windows 8.1.
So according to Berg, Microsoft has significantly changed its testing procedures over the last two years. The developer describes in detail how the tests were performed at the end of 2014 until the beginning of 2015, and how Microsoft's testing procedures have changed since then.
In 2014/2015, Microsoft employed an entire team dedicated to testing the operating system, builds, updates, drivers and other code. The team consisted of several sub-teams that ran tests and discussed bugs and issues in daily meetings. Testing was performed manually by each group as well as through automated controls. If the specific changes passed the tests, then we could integrate the code into Windows.
The teams conducted the tests in “real” hardware σε ένα εργαστήριο μέσω αυτοματοποιημένων ελέγχων. Υπήρχαν διαφορετικά συστήματα είχαν διαφορετικά hardware εξαρτήματα, π.χ. processors, σκληρούς δίσκους, κάρτες βίντεο και sound and other components to enable the company to cover a wide range of various system configurations. But this also helped to identify errors that only affected certain hardware.
So Microsoft stopped using almost the entire Windows test team as it began preparing three different systems: Windows, Windows Mobile and Xbox as a single system. The company began moving most of the testing to virtual machines. This means that the tests have stopped being performed on real systems and many different hardware configurations.
Microsoft employees could then use Windows under development on their own computers. The main idea behind this was to find bugs outside of working hours. Berg states in the video that this is not as widely used as before.
The main source of data, apart from the existing automated testing systems, comes from the telemetry and Windows Insiders. Windows Insider builds are installed on millions of devices, and Microsoft collects telemetry data from all those devices.
If something does not work properly, Microsoft receives information about what and how. One of the issues with telemetry data collection, however, is that they are simple data. The data do not accurately explain most errors. If something is not working properly, Microsoft may not be able to distinguish the relevant bits from the telemetry data. As for Insiders, some report problems and others do not. There are times when bugs appear to have been fixed on some Insiders systems, but that does not mean that they have been fixed for every hardware configuration. In addition, although Insiders can report errors, it is a fact that they do not provide all the necessary information to Microsoft, which creates huge problems for the developers tasked with resolving these issues.
2014 / 2015, the Microsoft test team, was tasked with analyzing bugs and problems. This gave the developers the data they needed to solve them.
Today, Berg says there is only telemetry that developers are trying to analyze to figure out how to fix these issues. The patches that are running for client devices running Insider Builds cannot be properly controlled due to lack of experience. This creates new errors.
One of the main reasons why Microsoft stopped releasing new feature updates to everyone at the same time was that problems that had not been identified could affect a large number of company customers.
So to avoid massive protests like after the circulation Windows 10 in the 1809 version, Windows releases with gradual rollouts rather than massively through Windows Update.
