Do you think the number of bugs in Windows 10 updates has increased in the last two years? If so, do you know why? This question is answered in a recent video former Microsoft (Senior SDET) employee Jerry Berg, also known as Barnacules.
Berg worked for Microsoft for 15 years and one of his roles was to design and develop tools and processes for automating testing in the Microsoft operating system. He left the company after the release of Windows 8.1.
According to Berg, Microsoft has significantly changed the testing procedures in the last two years. The developer describes in detail how the tests were conducted in late 2014 to early 2015, and how Microsoft's testing procedures have changed since then.
In 2014/2015, Microsoft had a whole team dedicated to testing the operating system, structures, updates, drivers and other code. The team consisted of many subgroups that conducted trials and discussed bugs and issues in everyday meetings. The tests were performed manually by each group but also through automated controls. If these changes passed the tests, then we could embed the code in Windows.
The teams performed the tests on "real" hardware in a laboratory through automated tests. There were different systems had different hardware components, e.g. processors, hard drives, video and audio cards and other components so that the company can cover a wide range of configurations of various systems. But it also helped to identify bugs that only affected certain hardware.
So Microsoft stopped using almost the entire Windows test team as it began building three different systems: Windows, Windows Mobile and Xbox as a single system. The company began transferring most of the tests to virtual machines. This means that the tests have stopped running on real systems and too many different hardware configurations.
Microsoft employees could then use Windows under development on their own computers. The main idea behind this was to be able to detect errors outside of working hours. Berg mentions in the video that this is not as widely used as before.
The main data source, in addition to the existing automated testing systems, comes from telemetry and Windows Insiders. Windows Insider builds are installed on millions of devices, and Microsoft collects telemetry data from all of these devices.
If something goes wrong, Microsoft gets information on what and how. But one of the issues with telemetry data collection is that it is just data. The data do not accurately explain most errors. If something is not working properly, Microsoft may not be able to distinguish the relevant bits from the telemetry data. As for the Insiders, some report the problems and some do not. There are times when bugs appear to have been fixed on some Insiders systems, but this does not mean that they have been fixed for every hardware configuration. In addition, while Insiders can report bugs, it is a fact that they do not provide all the necessary information to Microsoft, which creates huge problems for the developers who are in charge of resolving these issues.
In 2014/2015, the Microsoft test team was in charge of analyzing the bugs and problems. Thus they gave the developers the data needed to solve them.
Today, Berg says there is only telemetry that developers try to analyze to understand how they can fix these issues. The patches on the client devices running Insider Builds cannot be properly tested due to lack of experience. This creates new errors.
One of the main reasons Microsoft stopped releasing new feature updates to everyone at the same time was that unidentified issues could affect a large number of the company's customers.
So to avoid mass protests as after their release Windows 10 in version 1809, Windows is released in incremental rollouts rather than in bulk through Windows Update.