Preventing Disastrous Events Through The Power of Anomaly Detection In Machine Data

Preventing Disastrous Events Through The Power of Anomaly Detection In Machine Data

Preventing Disastrous Events Through the Power of Anomaly Detection in Machine Data

In August, a single server failed and the NASDAQ went down for three hours. In January, GlobalPayments reported that a hack compromising millions of credit card accounts cost them $93 million to recover from. In both cases, nobody took action until the damage was done because quite simply, nobody could. IT planners could not proactively head off the failure or breach because they had no suspicious behavior or early warning system to alert them that something was going amiss; all they had were mountains of event logs that needed to be pored through after the damage was already said and done.

nasdaq2

(Image Source: Shutterstock)

To prevent such catastrophes in the future, three things are required:

  • A machine learning engine that can analyze and learn from data – as well as human interaction and feedback – in order to get smarter over time;
  • Big Data technologies;
  • And of course, a cloud platform for ease of management.

Machine logs are the output of every application, website, server and supporting IT infrastructure component in the enterprise. This means that IT teams are inundated by massive amounts of machine log data. Digging through all of this data for something meaningful is not only unwieldy and unappealing, but also ultimately drives down productivity and increases costs. IT teams need to be able to not only visualize, but also analyze machine data in a way that can provide clear insight into what events in that stream of data are benign and what events are malicious that would require immediate attention. By being able to make sense of machine data from an “event” perspective, IT teams can create optimal functionality around any environment and also bring a true, proactive approach to IT management.

The goal of combining Big Data and a machine learning engine all within a cloud platform is to make events easily known and to procure insight on such events prior to their occurrences, which would result in significantly fewer headaches for IT managers and CIOs. Big Data technologies enable a holistic approach to analysis of data without binding to schemas, volumes or batch analytics. A machine learning engine provides advanced algorithms that learn and analyze from data as well as humans to increase intelligence over time. And lastly, the combination of these components in a cloud-based management platform enables an elastic compute at the massive scale that’s needed to analyze this amount of data in real-time across all vectors. By having this capability, IT managers can then create playbooks and remediation steps to prevent certain events and anticipate the impact to their organization.

Event detection can play a big role in optimizing system availability and performance; when a process, application or infrastructure component fails or slows down, it’s typically presaged by multiple events occurring simultaneously or in rapid succession. Out-of-the-ordinary or “anomaly” event detection can decipher how this series of events and their patterns vary from the norm, and what the variation means to the business. The power behind anomaly detection is neither a single technology nor a single technique. It’s typically a set of algorithms that work synergistically, leveraging machine learning techniques as well as mathematical and statistical analysis.

The benefits are clear, but there are many solutions out there that claim to do this, but do not leverage the three critical components for successful anomaly detection: machine learning engine, big data analytics and a cloud management platform. Without these three things working harmoniously together, it is that much more difficult to proactively manage the IT environment. Coming full circle, disastrous events such as the NASDAQ going down or a company losing millions of dollars could have been prevented through the power of machine data and anomaly detection.

sanjayBy Sanjay Sarathy, CMO of Sumo Logic

Sanjay joins Sumo Logic with over seventeen years of marketing, business development and community building experience in both SaaS and enterprise software environments. Prior to Sumo Logic, he was at Vindicia, a SaaS company that provides online billing and marketing solutions for for companies selling digital content and services. He also spent time at Above All Software, Qualys, Sun Microsystems and NetDynamics. Sanjay has a BA in Quantitative Economics from Stanford University and a MBA from the Haas School of Business at UC Berkeley.

About CloudTweaks

Established in 2009, CloudTweaks is recognized as one of the leading authorities in connected technology information and services.

We embrace and instill thought leadership insights, relevant and timely news related stories, unbiased benchmark reporting as well as offer green/cleantech learning and consultive services around the world.

Our vision is to create awareness and to help find innovative ways to connect our planet in a positive eco-friendly manner.

In the meantime, you may connect with CloudTweaks by following and sharing our resources.

View All Articles

Sorry, comments are closed for this post.

Comic
Cukes and the Cloud

Cukes and the Cloud

The Cloud, through bringing vast processing power to bear inexpensively, is enabling artificial intelligence. But, don’t think Skynet and the Terminator. Think cucumbers! Artificial Intelligence (A.I.) conjures up the images of vast cool intellects bent on our destruction or at best ignoring us the way we ignore ants. Reality is a lot different and much…

Ransomware’s Great Lessons

Ransomware’s Great Lessons

Ransomware The vision is chilling. It’s another busy day. An employee arrives and logs on to the network only to be confronted by a locked screen displaying a simple message: “Your files have been captured and encrypted. To release them, you must pay.” Ransomware has grown recently to become one of the primary threats to…

InformationWeek Reveals Top 125 Vendors Taking the Technology Industry by Storm

InformationWeek Reveals Top 125 Vendors Taking the Technology Industry by Storm

InformationWeek Reveals Top 125 Vendors Five-part series details companies to watch across five essential technology sectors SAN FRANCISCO, Sept. 27, 2016 /PRNewswire/ — InformationWeek released its list of “125 Vendors to Watch” in 2017. Selected by InformationWeek’s expert editorial team, the companies listed fall into one of five key themes: infrastructure, security, cloud, data management and…

Part 1 – Connected Vehicles: Paving The Way For IoT On Wheels

Part 1 – Connected Vehicles: Paving The Way For IoT On Wheels

Connected Vehicles From cars to combines, the IoT market potential of connected vehicles is so expansive that it will even eclipse that of the mobile phone. Connected personal vehicles will be the final link in a fully connected IoT ecosystem. This is an incredibly important moment to capitalize on given how much time people spend…

Embedded Sensors and the Wearable Personal Cloud

Embedded Sensors and the Wearable Personal Cloud

The Wearable Personal Cloud Wearable tech is one avenue of technology that’s encouraging cloud connections and getting us all onto interconnected networks, and with the continued miniaturization and advancement of computing the types of wearable tech are always expanding and providing us with new opportunities. A few years ago, smartwatches were rather clunky devices with…

Despite Record Breaches, Secure Third Party Access Still Not An IT Priority

Despite Record Breaches, Secure Third Party Access Still Not An IT Priority

Secure Third Party Access Still Not An IT Priority Research has revealed that third parties cause 63 percent of all data breaches. From HVAC contractors, to IT consultants, to supply chain analysts and beyond, the threats posed by third parties are real and growing. Deloitte, in its Global Survey 2016 of third party risk, reported…

Digital Transformation: Not Just For Large Enterprises Anymore

Digital Transformation: Not Just For Large Enterprises Anymore

Digital Transformation Digital transformation is the acceleration of business activities, processes, and operational models to fully embrace the changes and opportunities of digital technologies. The concept is not new; we’ve been talking about it in one way or another for decades: paperless office, BYOD, user experience, consumerization of IT – all of these were stepping…

5 Ways To Ensure Your Cloud Solution Is Always Operational

5 Ways To Ensure Your Cloud Solution Is Always Operational

Ensure Your Cloud Is Always Operational We have become so accustomed to being online that we take for granted the technological advances that enable us to have instant access to everything and anything on the internet, wherever we are. In fact, it would likely be a little disconcerting if we really mapped out all that…

Your Biggest Data Security Threat Could Be….

Your Biggest Data Security Threat Could Be….

Paying Attention To Data Security Your biggest data security threat could be sitting next to you… Data security is a big concern for businesses. The repercussions of a data security breach ranges from embarrassment, to costly lawsuits and clean-up jobs – particularly when confidential client information is involved. But although more and more businesses are…

Cloud Infographic – Big Data Predictions By 2023

Cloud Infographic – Big Data Predictions By 2023

Big Data Predictions By 2023 Everything we do online from social networking to e-commerce purchases, chatting, and even simple browsing yields tons of data that certain organizations collect and poll together with other partner organizations. The results are massive volumes of data, hence the name “Big Data”. This includes personal and behavioral profiles that are stored, managed, and…

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

What Cloud Technology Has Allowed Us to Do Better The cloud has made our working lives easier, with everything from virtually unlimited email storage to access-from-anywhere enterprise resource planning (ERP) systems. It’s no wonder the 2013 cloud computing research IDG survey revealed at least 84 percent of the companies surveyed run at least one cloud-based application.…

Report: Enterprise Cloud Computing Moves Into Mature Growth Phase

Report: Enterprise Cloud Computing Moves Into Mature Growth Phase

Verizon Cloud Report Enterprises using the cloud, even for mission-critical projects, is no longer new or unusual. It’s now firmly established as a reliable workhorse for an organization and one that can deliver great value and drive transformation. That’s according to a new report from Verizon entitled “State of the Market: Enterprise Cloud 2016.” which…

The Business of Security: Avoiding Risks

The Business of Security: Avoiding Risks

The Business of Security Security is one of those IT concerns that aren’t problematic until disaster strikes. It might be tomorrow, it could be next week or next year. The fact is that poor security leaves businesses wide open for data loss and theft. News outlets just skim the surface, but hackers cost business up…

Big Data and Financial Services – Security Threat or Massive Opportunity?

Big Data and Financial Services – Security Threat or Massive Opportunity?

Big Data and Financial Services Cloud Banking Insights Series focuses on big data in the financial services industry and whether it is a security threat or actually a massive opportunity. How does big data fit into an overall cloud strategy? Most FI’s have a positive mind-set towards cloud IT consumption as it not only enables…