Potential Powerful Pitfalls – Big Data, Big Trouble

Potential Powerful Pitfalls – Big Data, Big Trouble

Potential Powerful Pitfalls – Big Data, Big Trouble

Information has always been a great source of power. Kingdoms were often won and lost through scout reports, misinformation and treachery. Even now we fear those that hold too much of it and often cry foul in the name of privacy. But if you asked me, as a collective social generation, we have forsaken our privacy willingly so we should all just calm down about it.

Data Breach Comic

Traditionally, we have collected only relevant data and stored those according to some form of sorting process so that we can make sense of all of them. For example if we wanted to know how many babies are born in Manhattan in a day we would only need to collect data from hospitals and clinics within Manhattan. This is the traditional way of collecting data because of the limited capacities of our data systems. But Big Data aims to change all of that, it is not about just collecting specific data like the number of births in Manhattan, but rather it wants to collect from each hospital from each state and the whole world as well, and not just regarding births. This can provide us with any information we need and allow us to predict future trends more accurately, or so we thought.

But Big Data is prone to some very powerful pitfalls. One is called the ‘lottery paradox,’ here we tend to give emphasis on something that is very improbable to happen to us simply because of the payoff just like in a lottery. There will always be a winner, and though chances are 175,000,000 to 1, we still hold on to the chance that we get to be that “1”. How this applies to big data is simple: the larger the data sets are the smaller the chance it is to find that piece of important information that gives out the biggest payoff. These small gems within the vast sea of data increases in number but does not increase in frequency, we simply find more of them the more data we have. So we then invest more in finding these gems, these “outrageous events”, and when we fail to find what we expect, we simply wonder if it was all worth the effort and money.

Another pitfall according to ZapThink is the ‘more is better’ paradox which is the assumption that if a certain quantity of data is good, then more of it is better. But this is not necessarily true as we might simply be encouraging the collection of more irrelevant and redundant data. Then we rely on services like Hadoop to make sense of all that chaff. But the truth is no process or software is ever capable of making sense of everything. Big Data is not about being selective in the collection process, a very big downside if you ask me.

By Abdul Salam

Sorry, comments are closed for this post.

Comic
Part 1 – Connected Vehicles: Paving The Way For IoT On Wheels

Part 1 – Connected Vehicles: Paving The Way For IoT On Wheels

Connected Vehicles From cars to combines, the IoT market potential of connected vehicles is so expansive that it will even eclipse that of the mobile phone. Connected personal vehicles will be the final link in a fully connected IoT ecosystem. This is an incredibly important moment to capitalize on given how much time people spend…

Embedded Sensors and the Wearable Personal Cloud

Embedded Sensors and the Wearable Personal Cloud

The Wearable Personal Cloud Wearable tech is one avenue of technology that’s encouraging cloud connections and getting us all onto interconnected networks, and with the continued miniaturization and advancement of computing the types of wearable tech are always expanding and providing us with new opportunities. A few years ago, smartwatches were rather clunky devices with…

SWIFT Says Bank Hacks Set To Increase

SWIFT Says Bank Hacks Set To Increase

Bank Hacks Set To Increase SWIFT, whose messaging network is used by banks to send payment instructions worth trillions of dollars each day, said three clients were hacked over the summer and cyber attacks on banks are set to increase. The theft of $81 million in February from Bangladesh’s central bank using SWIFT messages rocked…

Security: Avoiding A Hatton Garden-Style Data Center Heist

Security: Avoiding A Hatton Garden-Style Data Center Heist

Data Center Protection In April 2015, one of the world’s biggest jewelry heists occurred at the Hatton Garden Safe Deposit Company in London. Posing as workmen, the criminals entered the building through a lift shaft and cut through a 50cm-thick concrete wall with an industrial power drill. Once inside, the criminals had free and unlimited…

Automated Application Discovery Introduced By Savision At Microsoft Ignite 2016

Automated Application Discovery Introduced By Savision At Microsoft Ignite 2016

Automated Application Discovery ATLANTA, GEORGIA – September 26, 2016 – Savision, a market leader in service-oriented monitoring solutions that unify IT operations with IT service management, today announced the release of its automated application discovery module for Unity iQ. Savision’s automated application discovery module offers an agentless, trigger-based discovery method that captures infrastructure elements, application dependencies, as well…

THE FUTURE OF BIG DATA AND DNS ANALYTICS

THE FUTURE OF BIG DATA AND DNS ANALYTICS

Big Data and DNS Analytics Big Data is revolutionizing the way admins manage their DNS traffic. New management platforms are combining historical data with advanced analytics to inform admins about possible performance degradation in their networks. Not only that, but they also have the ability to suggest ways to optimize network configurations for faster routing.…

Cloud-based GRC Intelligence Supports Better Business Performance

Cloud-based GRC Intelligence Supports Better Business Performance

Cloud-based GRC Intelligence All businesses need a strategy and processes for governance, risk and compliance (GRC). Many still view GRC activity as a burdensome ‘must-do,’ approaching it reactively and managing it with non-specialized tools. GRC is a necessary business endeavor but it can be elevated from a cost drain to a value-add activity. By integrating…

Data Breaches: Incident Response Planning – Part 1

Data Breaches: Incident Response Planning – Part 1

Incident Response Planning – Part 1 The topic of cybersecurity has become part of the boardroom agendas in the last couple of years, and not surprisingly — these days, it’s almost impossible to read news headlines without noticing yet another story about a data breach. As cybersecurity shifts from being a strictly IT issue to…

Using Cloud Technology In The Education Industry

Using Cloud Technology In The Education Industry

Education Tech and the Cloud Arguably one of society’s most important functions, teaching can still seem antiquated at times. Many schools still function similarly to how they did five or 10 years ago, which is surprising considering the amount of technical innovation we’ve seen in the past decade. Education is an industry ripe for innovation…

The Cloud Is Not Enough! Why Businesses Need Hybrid Solutions

The Cloud Is Not Enough! Why Businesses Need Hybrid Solutions

Why Businesses Need Hybrid Solutions Running a cloud server is no longer the novel trend it once was. Now, the cloud is a necessary data tier that allows employees to access vital company data and maintain productivity from anywhere in the world. But it isn’t a perfect system — security and performance issues can quickly…

Cloud Infographic: The Explosive Growth Of The Cloud

Cloud Infographic: The Explosive Growth Of The Cloud

The Explosive Growth Of The Cloud We’ve been covering cloud computing extensively over the past number of years on CloudTweaks and have truly enjoyed watching the adoption and growth of it. Many novices are still trying to wrap their mind around what the cloud it is and what it does, while others such as thought…

Disaster Recovery – A Thing Of The Past!

Disaster Recovery – A Thing Of The Past!

Disaster Recovery  Ok, ok – I understand most of you are saying disaster recovery (DR) is still a critical aspect of running any type of operations. After all – we need to secure our future operations in case of disaster. Sure – that is still the case but things are changing – fast. There are…

Is The Fintech Industry The Next Tech Bubble?

Is The Fintech Industry The Next Tech Bubble?

The Fintech Industry Banks offered a wide variety of services such as payments, money transfers, wealth management, selling insurance, etc. over the years. While banks have expanded the number of services they offer, their core still remains credit and interest. Many experts believe that since banks offered such a wide multitude of services, they have…

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

What Cloud Technology Has Allowed Us to Do Better The cloud has made our working lives easier, with everything from virtually unlimited email storage to access-from-anywhere enterprise resource planning (ERP) systems. It’s no wonder the 2013 cloud computing research IDG survey revealed at least 84 percent of the companies surveyed run at least one cloud-based application.…

Cloud Infographic – Interesting Big Data Facts

Cloud Infographic – Interesting Big Data Facts

Big Data Facts You Didn’t Know The term Big Data has been buzzing around tech circles for a few years now. Forrester has defined big data as “Technologies and techniques that make capturing value from data at an extreme scale economical.” The key word here is economical. If the costs of extracting, processing, and making use…

How Data Science And Machine Learning Is Enabling Cloud Threat Protection

How Data Science And Machine Learning Is Enabling Cloud Threat Protection

Data Science and Machine Learning Security breaches have been consistently rising in the past few years. Just In 2015, companies detected 38 percent more security breaches than in the previous year, according to PwC’s Global State of Information Security Survey 2016. Those breaches are a major expense — an average of $3.79 million per company,…