Potential Powerful Pitfalls – Big Data, Big Trouble

Potential Powerful Pitfalls – Big Data, Big Trouble

Potential Powerful Pitfalls – Big Data, Big Trouble

Information has always been a great source of power. Kingdoms were often won and lost through scout reports, misinformation and treachery. Even now we fear those that hold too much of it and often cry foul in the name of privacy. But if you asked me, as a collective social generation, we have forsaken our privacy willingly so we should all just calm down about it.

Data Breach Comic

Traditionally, we have collected only relevant data and stored those according to some form of sorting process so that we can make sense of all of them. For example if we wanted to know how many babies are born in Manhattan in a day we would only need to collect data from hospitals and clinics within Manhattan. This is the traditional way of collecting data because of the limited capacities of our data systems. But Big Data aims to change all of that, it is not about just collecting specific data like the number of births in Manhattan, but rather it wants to collect from each hospital from each state and the whole world as well, and not just regarding births. This can provide us with any information we need and allow us to predict future trends more accurately, or so we thought.

But Big Data is prone to some very powerful pitfalls. One is called the ‘lottery paradox,’ here we tend to give emphasis on something that is very improbable to happen to us simply because of the payoff just like in a lottery. There will always be a winner, and though chances are 175,000,000 to 1, we still hold on to the chance that we get to be that “1”. How this applies to big data is simple: the larger the data sets are the smaller the chance it is to find that piece of important information that gives out the biggest payoff. These small gems within the vast sea of data increases in number but does not increase in frequency, we simply find more of them the more data we have. So we then invest more in finding these gems, these “outrageous events”, and when we fail to find what we expect, we simply wonder if it was all worth the effort and money.

Another pitfall according to ZapThink is the ‘more is better’ paradox which is the assumption that if a certain quantity of data is good, then more of it is better. But this is not necessarily true as we might simply be encouraging the collection of more irrelevant and redundant data. Then we rely on services like Hadoop to make sense of all that chaff. But the truth is no process or software is ever capable of making sense of everything. Big Data is not about being selective in the collection process, a very big downside if you ask me.

By Abdul Salam

About Abdul

Abdul is a senior consultant with Energy Services, and author of numerous blogs, books, white papers, and tutorials on cloud computing and accomplished technical writer with CloudTweaks. He earned his bachelor’s degree in Information Technology, followed by an MBA-IT degree and certifications by Cisco and Juniper Networks.

He has recently co-authored: Deploying and Managing a Cloud Infrastructure: Real-World Skills for the CompTIA Cloud+ Certification (Wiley).

View All Articles

Sorry, comments are closed for this post.

Finally, The Time For Security Information Event Management (SIEM)

Finally, The Time For Security Information Event Management (SIEM)

The Time For SIEM Security Information Event Management (SIEM) tools have been around for a long time. My first encounter with a SIEM vendor was about twenty years ago while being courted to resell their product. To this day, I still recall two vivid memories from that meeting; the product was very complex and quite…

Where Is The Tipping Point For The Flying Drone Market?

Where Is The Tipping Point For The Flying Drone Market?

The Flying Drone Market In the past year I have written a number of articles here on CloudTweaks about Drones. I enjoy flying drones. One thing I’ve used my drone for is proving I didn’t need to do the worst fall maintenance job ever. Cleaning the gutters of the house by flying the drone around…

You’ve Moved Your Data To The Cloud…What Now?

You’ve Moved Your Data To The Cloud…What Now?

Data To The Cloud…What Now? You’ve done the research. You’ve vetted multiple cloud solutions. You’ve chosen the right vendor. And you’ve migrated your data to the cloud. All done, right? Not so fast… While it has become a bit of a scary proposition for many organizations to put their trust in the cloud and start…

Disaster Recovery – A Thing Of The Past!

Disaster Recovery – A Thing Of The Past!

Disaster Recovery  Ok, ok – I understand most of you are saying disaster recovery (DR) is still a critical aspect of running any type of operations. After all – we need to secure our future operations in case of disaster. Sure – that is still the case but things are changing – fast. There are…

SaaS Freemium Models and the Hidden Cost of Free

SaaS Freemium Models and the Hidden Cost of Free

SaaS Freemium Models We’ve all been lured into sexy “try before you buy” freemium models that provide just the right amount of functionality to get you started. Yet, it’s not quite enough to complete the job. “Getting the job done” often requires stepping up to a paid or premium version that provides more functionality, capabilities…

From Illusion To Reality: Up Personal On Cloud Computing Privacy

From Illusion To Reality: Up Personal On Cloud Computing Privacy

Cloud Computing Privacy We are under an illusion around the cloud computing. Without a doubt the benefits are incredible. However, cloud impacts personal user privacy and potentially exposes their private data in ways they may not have anticipated. Given that users don’t usually read the terms of service and privacy policies, it is unclear how…

Cloud as Our Children’s Playground

Cloud as Our Children’s Playground

Cloud as Our Children’s Playground We read everyday about the fast pace of business adoption of the cloud.  But children are adopting the cloud at just as an aggressive pace.  The Cloud is changing our children’s lives – the way they play, learn, and interact with others and the world around them.  I see children…

CloudTweaks is recognized as one of the leading influencers in cloud computing, infosec, big data and the internet of things (IoT) information. Our goal is to continue to build our growing information portal by providing the best in-depth articles, interviews, event listings, whitepapers, infographics and much more.

Advertising