Potential Powerful Pitfalls – Big Data, Big Trouble

Potential Powerful Pitfalls – Big Data, Big Trouble

Potential Powerful Pitfalls – Big Data, Big Trouble

Information has always been a great source of power. Kingdoms were often won and lost through scout reports, misinformation and treachery. Even now we fear those that hold too much of it and often cry foul in the name of privacy. But if you asked me, as a collective social generation, we have forsaken our privacy willingly so we should all just calm down about it.

Data Breach Comic

Traditionally, we have collected only relevant data and stored those according to some form of sorting process so that we can make sense of all of them. For example if we wanted to know how many babies are born in Manhattan in a day we would only need to collect data from hospitals and clinics within Manhattan. This is the traditional way of collecting data because of the limited capacities of our data systems. But Big Data aims to change all of that, it is not about just collecting specific data like the number of births in Manhattan, but rather it wants to collect from each hospital from each state and the whole world as well, and not just regarding births. This can provide us with any information we need and allow us to predict future trends more accurately, or so we thought.

But Big Data is prone to some very powerful pitfalls. One is called the ‘lottery paradox,’ here we tend to give emphasis on something that is very improbable to happen to us simply because of the payoff just like in a lottery. There will always be a winner, and though chances are 175,000,000 to 1, we still hold on to the chance that we get to be that “1”. How this applies to big data is simple: the larger the data sets are the smaller the chance it is to find that piece of important information that gives out the biggest payoff. These small gems within the vast sea of data increases in number but does not increase in frequency, we simply find more of them the more data we have. So we then invest more in finding these gems, these “outrageous events”, and when we fail to find what we expect, we simply wonder if it was all worth the effort and money.

Another pitfall according to ZapThink is the ‘more is better’ paradox which is the assumption that if a certain quantity of data is good, then more of it is better. But this is not necessarily true as we might simply be encouraging the collection of more irrelevant and redundant data. Then we rely on services like Hadoop to make sense of all that chaff. But the truth is no process or software is ever capable of making sense of everything. Big Data is not about being selective in the collection process, a very big downside if you ask me.

By Abdul Salam

About Abdul

Abdul is a senior consultant with Energy Services, and author of numerous blogs, books, white papers, and tutorials on cloud computing and accomplished technical writer with CloudTweaks. He earned his bachelor’s degree in Information Technology, followed by an MBA-IT degree and certifications by Cisco and Juniper Networks.

He has recently co-authored: Deploying and Managing a Cloud Infrastructure: Real-World Skills for the CompTIA Cloud+ Certification (Wiley).

View All Articles

Sorry, comments are closed for this post.

Conquering Disease with Artificial Intelligence and IBM Watson

Conquering Disease with Artificial Intelligence and IBM Watson

Artificial Intelligence and IBM Watson Artificial Intelligence, or AI, is growing increasingly pervasive in today’s modern world. Perhaps the most publicized and recognizable application of AI to date, IBM’s Jeopardy-winning computer, Watson, is now being used to help cure cancer. IBM announced the development of Watson for Genomics on Wednesday at the National Cancer Moonshot…

The Fundamentals of Predictive Analysis

The Fundamentals of Predictive Analysis

Predictive Analysis  Article sponsored by SAS Software and Big Data Forum Analytics is playing an increasingly important role in our lives thanks in large part to internet of things (IoT) developments and a greater appreciation of Big Data. With solutions that range across business productivity, health care, individual and national security, new insights are regularly…

Clouding Around With The Unicorns

Clouding Around With The Unicorns

The Social Unicorn Early investors and technology consumers alike love the poignantly named “Unicorn” companies, or private startups that reach the $1 billion valuation. The most recent unicorn spotting is social media game-changer Snapchat, which was recently valued at $16 billion. According to a recent infographic from Alexa discovered via Adweek, Snapchat also proves to…

Achieving Network Security In The IoT

Achieving Network Security In The IoT

Security In The IoT The network security market is experiencing a pressing and transformative change, especially around access control and orchestration. Although it has been mature for decades, the network security market had to transform rapidly with the advent of the BYOD trend and emergence of the cloud, which swept enterprises a few years ago.…

Increasing Efficiency and Reducing Cost with Managed Printing Services

Increasing Efficiency and Reducing Cost with Managed Printing Services

Managed Printing Services This is a sponsored post written on behalf of HP MPS.  Today’s business leaders recognize the value of shared services, significantly providing a wide range of enterprises with the sophisticated tools they need to compete with big business, while additionally driving costs down. But an area often overlooked in our tech-savvy world…

Infographic Introduction – Benefits of Cloud Computing

Infographic Introduction – Benefits of Cloud Computing

Benefits of Cloud Computing Based on Aberdeen Group’s Computer Intelligence Dataset, there are more than 1.6 billion permutations to choose from when it comes to cloud computing solutions. So what, on the face of it, appears to be pretty simple is actually both complex and dynamic regardless of whether you’re in the market for networking,…

Driving Success: 6 Key Metrics For Every Recurring Revenue Business

Driving Success: 6 Key Metrics For Every Recurring Revenue Business

Recurring Revenue Business Metrics Recurring revenue is the secret sauce behind the explosive growth of powerhouses like Netflix and Uber. Unsurprisingly, recurring revenue is also quickly gaining ground in more traditional industries like healthcare and the automotive business. In fact, nearly half of U.S. businesses have adopted or are planning to adopt a recurring revenue model,…

Cloud Computing Is Greener Than You Think

Cloud Computing Is Greener Than You Think

Cloud Computing Is Greener Than You Think Last week we touched upon how a project in Finland had blended two of the world’s most important industries, cloud computing and green technology, to produce a data centre that used nearby sea water to both cool their servers and heat local homes.  Despite such positive environmental projects, there…

Five Signs The Internet of Things Is About To Explode

Five Signs The Internet of Things Is About To Explode

The Internet of Things Is About To Explode By 2020, Gartner estimates that the Internet of Things (IoT) will generate incremental revenue exceeding $300 billion worldwide. It’s an astoundingly large figure given that the sector barely existed three years ago. We are now rapidly evolving toward a world in which just about everything will become…

Cloud Infographic: The Future of File Storage

Cloud Infographic: The Future of File Storage

 The Future of File Storage A multi-billion dollar market Data storage has been readily increasing for decades. In 1989, an 8MB Macintosh Portable was top of the range; in 2006, the Dell Inspiron 6400 became available, boasting 160GB; and now, we have the ‘Next Generation’ MacBook Pro with 256GB of storage built in. But, of course,…

Cloud Infographic – The Internet Of Things In 2020

Cloud Infographic – The Internet Of Things In 2020

The Internet Of Things In 2020 The growing interest in the Internet of Things is amongst us and there is much discussion. Attached is an archived but still relevant infographic by Intel which has produced a memorizing snapshot at how the number of connected devices have exploded since the birth of the Internet and PC.…

4 Industries Being Transformed By The Internet of Things

4 Industries Being Transformed By The Internet of Things

Compelling IoT Industries Every year, more and more media organizations race to predict the trends that will come to shape the online landscape over the next twelve months. Many of these are wild and outlandish and should be consumed with a pinch of salt, yet others stand out for their sober and well-researched judgements. Online…