Chris Gervais

Resolving the Normalization of Deviance by Building a Culture of Communication

Building a Culture of Communication

Real-time monitoring and corresponding alerts are critical for maintaining the performance and security of today’s complex cloud infrastructures. Given the exorbitant amount of data effective network monitoring can produce, however, a troublesome problem often occurs: organizations and their Security, Operations and Development teams start to develop a normalization of deviance.

What’s a Normalization of Deviance?

A normalization of deviance is an incremental and gradual erosion of normal procedures, and it can lead to dire consequences. The explosion of the Challenger space shuttle in 1986 is, unfortunately, an infamous example of a normalization of deviance (and the resultant investigation is where Diane Vaughn developed this theory). NASA had been testing the limits of the joints on its solid rocket boosters and found they weren’t behaving as expected. Rather than halting the development process and dealing with the booster errors head on, NASA chose to accept the problem and move forward with the launch. This normalization of deviance led to the Challenger tragedy, as it was later confirmed that the O-ring gaskets on one of the problematic boosters were responsible for the disaster.

Cloud Disaster Recovery

The lesson to learn here is that the normalization of deviance stemmed from an organizational failure at NASA at the management level. It’s also a common occurrence amongst fast-growth technology companies and enterprises that are rapidly scaling their cloud-based infrastructures and adapting their architectures to changing business needs. In these settings, more tools are required to monitor infrastructure as the business grows and evolves and compute needs adapt to keep up. With more tools come more data and alerts, and as a result, operators have to balance the signal-to-noise ratio to ensure their teams can focus on the most important inputs.

The Harmful Effects of Burnout

With alerts coming in from a variety of different systems and tools, Security, Operations and Development teams can sometimes feel as if they’re at an obnoxiously loud party, with dozens of people having different conversations about different things at the same time. Without a systematic approach to compensate for this chaos, these teams can become desensitized, so that even when the system flags a truly anomalous activity, the alert may get ignored.

Burnout can lead to longer response times, create an unmanageable volume of technical debt, and generally have a negative effect on a company’s workforce. Team members who are struggling to keep up with never-ending alerts can experience anxiety, sleep deprivation, cognitive impairment and even increased blood pressure or headaches. A normalization of deviance and resulting burnout can also lead to a lack of interest in solving problems or helping customers, and as a result, negatively impact company culture.

A helpful way to determine if there’s a normalization of deviance in your own company is to watch how existing team members interact with new hires. When a new hire asks about an incoming alert, does your team brush it off and dismiss the problem as nothing to worry about? If so, your team has likely developed the habit of accepting bad practices as normal. This happens. It’s not a reason to upend everything, but a signal that leaders need to discover early and begin corrective action.

How to Prevent Desensitization

Chef CTO Adam Jacobs directly addressed burnout at the 2016 ChefCon: “We should make a conscious and intentional choice to build the future we want to be a part of, with our technology and culture.”

The most effective and long-lasting way to prevent a normalization of deviance from permeating your company and Security, Operations and Development teams is simply to communicate more and ensure those teams are empowered to enact change in their tools and process where needed. The fatigue and numbness that can result from a normalization of deviance is usually easier to spot in others than in ourselves, so be on the lookout for team members who may be struggling. Have burnout and personal health be a regular topic of discussion in one-on-one meetings, and make sure everyone is transparent about how current business goals or customer demands are physically and mentally affecting different teams. Perhaps most importantly, recognize that combating the normalization of deviance requires continuous effort. It’s not a task you can check off and then ignore.

Fast-growth technology companies and their Security, Operations and Development teams are all focused on moving at warp speed, building new cloud-based features and making sure complex platforms scale. But it’s equally important to prioritize building a culture of communication, honesty and improvement in order to catch and prevent a normalization of deviance before it sets in. This negative behavior pattern needs to be addressed, not tolerated, to ensure your company’s security defenses remain ahead of any adversaries’ offensive maneuvers.

By Chris Gervais

Chris Gervais

Chris Gervais, VP of Engineering. As Threat Stack’s head of Engineering, Chris is passionate about building, not only a rock solid, high-performance product, but also a team of elite engineers, industry best processes and a culture that attracts the best talent. Prior to Threat Stack, Chris held senior positions at lifeIMAGE, Enservio, Partners Healthcare, Inc., Inflexxion, Inc. and VIS Corporation, where he was responsible for engineering, technical operations, and technology strategy for cloud platforms.

View Website

CONTRIBUTORS

3 Ways to Protect Users From Ransomware With the Cloud

3 Ways to Protect Users From Ransomware With the Cloud

Protect Users From Ransomware The threat of ransomware came into sharp focus over the course of 2016. Cybersecurity trackers have ...
Chris Gerva

Why Containers Can’t Solve All Your Problems In The Cloud

Containers and the cloud Docker and other container services are appealing for a good reason - they are lightweight and ...
Bryan Doerr

Cyber-Threats and the Need for Secure Industrial Control Systems

Secure Industrial Control Systems (ICS) Industrial Control Systems (ICS) tend to be “out of sight, out of mind.” These systems ...
4 Open Source Business Intelligence Tools For Big Data Reporting

4 Open Source Business Intelligence Tools For Big Data Reporting

Open Source Business Intelligence Tools It’s impossible to take the right business decisions without having insightful information to back up ...
What the Dyn DDoS Attacks Taught Us About Cloud-Only EFSS

What the Dyn DDoS Attacks Taught Us About Cloud-Only EFSS

DDoS Attacks October 21st, 2016 went into the annals of Internet history for the large scale Distributed Denial of Service (DDoS) ...
What’s Next In Cloud And Data Security For 2017?

What’s Next In Cloud And Data Security For 2017?

Cloud and Data Security It has been a tumultuous year in data privacy to say the least – we’ve had ...
Two 2017 Trends From A Galaxy Far, Far Away

Two 2017 Trends From A Galaxy Far, Far Away

Reaching For The Stars People who know me know that I’m a huge Star Wars fan. I recently had the ...
The Five Rules of Security and Compliance in the Public Cloud Era

The Five Rules of Security and Compliance in the Public Cloud Era

Security and Compliance  With technology at the heart of businesses today, IT systems and data are being targeted by criminals, ...
Battle of the Clouds: Multi-Instance vs. Multi-Tenant Architecture

Battle of the Clouds: Multi-Instance vs. Multi-Tenant Architecture

Multi-Instance vs. Multi-Tenant Architecture  The cloud is part of everything we do. It’s always there backing up our data, pictures, ...
Countdown to GDPR: Preparing for Global Data Privacy Reform

Countdown to GDPR: Preparing for Global Data Privacy Reform

Preparing for Global Data Privacy Reform Multinational businesses who aren’t up to speed on the regulatory requirements of the European ...

NEWS

Deloitte TMT Predictions: Machine Learning Deployments, On-Demand Content and Live Events Will Continue to Drive Growth

Deloitte TMT Predictions: Machine Learning Deployments, On-Demand Content and Live Events Will Continue to Drive Growth

NEW YORK, Dec. 12, 2017 /PRNewswire/ -- Deloitte forecasts double digital growth in machine learning deployments for the enterprise, an increasing worldwide ...
email as a service

Google Data Analysis, Artificial Intelligence and Predicting Vaccine Scares

Social media trends can predict tipping points in vaccine scares Analyzing trends on Twitter and Google can help predict vaccine ...
Hackers shut down infrastructure safety system in attack: FireEye

Hackers shut down infrastructure safety system in attack: FireEye

Hackers shut down infrastructure safety system (Reuters) - Hackers likely working for a nation-state recently penetrated the safety system of ...