克服软件测试中的“警报疲劳”_AI阅读总结 — 包阅AI

A problem arises when the people responsible for pushing to production must also oversee that product’s quality. Sometimes, they must choose between two competing needs: Do I meet the production quota or deadline? Or do I meet the production quality requirements? The result is either delays or bad performance — the latter of which can quickly escalate to bigger problems. Missing alerts can also inevitably lead to outages on an epic and global scale, causing significant downtime at a major cost.

Before “shift left” became the norm, developers would write code that quality assurance testing teams would then comb through and identify the initial bugs in the product. Developers were then only tasked with reviewing the proofed end product to ensure it functioned as they initially envisioned. But now, the testing and quality control onus has been put on developers earlier and earlier. An outcome of this dynamic is that developers are becoming increasingly numb to the high volume of bugs they are coming across in the process, and as a result, they are pushing bad code to production. After all, if developers are critiquing their work, can they claim to be the most objective critics?

The Quality Cliff

The Disney Corporation famously developed one of its industry’s most novel quality control systems.

Disney’s strong production quality system was formed around a dedicated animators team that was only responsible for perfecting a select number of scenes. Then, the director would put all the pieces together and make sure they flowed. Each animating team had no idea what the final film would be like until it hit the silver screen. This dynamic is because if you ask those same animators to work across the entire film, the quality starts to decline. Disney found that they would get caught up in other elements like the overall narrative or storyline and lose sight of the flaws in individual scenes.

Similarly, when developers are overtaxed, they get lost in the big picture and start pushing to code rather than ensuring each element works properly. A recent survey of developers revealed that 67% have pushed to prod without testing, with 28% admitting they regularly do so, and 61% have used untested code generated by ChatGPT.

The Blow Back From Alert Fatigue

Quality control can take many forms, but quality and production are often competing goals. When the same teams are responsible for both, problems such as “Alert Fatigue” arise. This condition emerges when developers, overwhelmed by constant urgent notifications, ignore critical alerts. This lack of testing risks software quality, user experience, and system stability.

Without vital testing phases, error rates are high, and organizations expose themselves to security vulnerabilities that erode consumer trust and company reputation.

The U.S. customer experience (CX) quality has hit an all-time low — with an unprecedented 39% of brands significantly declining in CX scores in 2024. And while users are unfortunately all too used to disappointing software, there are severe ramifications in a world run by digital experiences. More than half of consumers (57%) would consider switching to a competitor after just 3-4 negative interactions with a company’s website, software, or mobile app, and 70% of people say poor digital experiences have a moderate, severe, or extreme impact on their purchasing decisions.

Organizations must ensure that vital testing phases are robust and well-defined to mitigate these adverse outcomes. These phases should include comprehensive automated testing, continuous integration (CI) practices, and rigorous manual testing by dedicated QA teams. Developers should focus on unit and integration tests, while QA teams handle system, regression acceptance, and exploratory testing. This division of labor enables developers to concentrate on writing and refining code while QA specialists ensure the software meets the highest quality standards before production.

By fostering a collaborative environment where developers and QA teams work closely, organizations can enhance software quality, user experience, and system stability. Implementing regular cross-functional meetings and integrated workflows helps both teams stay aligned on goals and processes. Embedding these practices into the company culture and investing in the right tools will protect against high error rates and security vulnerabilities, thereby maintaining consumer trust and safeguarding the company’s reputation.

A Path Forward

Developers do not neglect critical issues out of laziness — they are working with limited bandwidth and trying to prioritize the bugs they think matter the most. However, no system is in place to truly capture what is most crucial when developers are trying to take care of everything.

Organizations cannot continue dismantling QA teams and shifting the testing and quality burden onto developers. The developer experience must be reprioritized, with companies investing in comprehensive tools and processes that facilitate continuous quality. This means providing robust automated testing frameworks, real-time monitoring systems, and dedicated support teams to assist developers.

Rather than focusing on individual behavior, it is crucial to reexamine and restructure existing workflows to integrate quality at every stage. By cultivating a culture of quality throughout the organization and ensuring that teams have the resources they need to uphold it — such as advanced CI/CD pipelines, collaborative platforms, and ongoing training — the “red alert” problems will not be lost in a sea of chaos but instead addressed proactively and efficiently.