Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Big Data has all the corporate heads up and about in excitement since it promises to uncover golden nuggets of information out from an ocean of mundane and redundant data. But here’s the problem sticking everybody in the side, Big Data is big, as in it can reach the levels of “we-can’t-come-up-with-enough-names” bytes big. And with current upload speeds nowhere near as fast as download speeds, all the fancy analytics software and techniques aren’t going to do us any good if we can’t get our data where we need them.

bandwidth-straw

It is called the Skinny Straw or Drinking Straw problem and it is the biggest and most obvious problem being faced by Big Data. The analogy is simple; imagine passing an elephant through a drinking straw. Sure you can grind the elephant into very tiny bits so it can fit through the straw, but how long is that going to take? I admit that was a little gory, the real analogy was filling a swimming pool using a drinking straw, but you get the picture. The straw represents bandwidth and how small it is compared to the amount of data that needs to get to the other side of that straw.

The only real solution we can think of right off the bat is to get a bigger straw, but usually that would require major infrastructure upgrades on the part of the ISP or backbone provider, and we are talking about extreme amounts of cash (or credit if that’s how you roll). There are also the obvious technology limitations, we can upgrade to the best there is and it might not still be 100% enough. Some Big Data providers have tried their own proprietary ideas to try and get around this issue, or at least lessen it to some degree.

Here are some ways and techniques that are being used in the industry right now:

  1. We have the data compression and de-duplication techniques to make data transfers faster. That’s the “grinding the elephant and pushing it through the straw as fast as possible” solution.
  2. There is the “tinker with current protocols” direction by combining the reliability of TCP connections and the speed and bandwidth of UDP transfers into something that they call FASP. This ensures that communication is fast and secure while doing away with various handshaking processes that TCP requires.
  3. We can also work with various protocol optimizations in order to get around the problem. But one way that is really worth mentioning is the tried and tested transfer method –the old SneakerNet approach. Providers that use this method allow their customers to mail their hard drives to the company address so that they can transfer the data and then mail the hard drives back. This method is often faster at moving extremely large amounts of data quickly even taking into consideration the delivery time.

By Abdul Salam

(Image Source: ShutterStock)

About Abdul

Abdul is a senior consultant with Energy Services, and author of numerous blogs, books, white papers, and tutorials on cloud computing and accomplished technical writer with CloudTweaks. He earned his bachelor’s degree in Information Technology, followed by an MBA-IT degree and certifications by Cisco and Juniper Networks.

He has recently co-authored: Deploying and Managing a Cloud Infrastructure: Real-World Skills for the CompTIA Cloud+ Certification (Wiley).

View All Articles

Sorry, comments are closed for this post.

Why Small Businesses Need A Business Intelligence Dashboard

Why Small Businesses Need A Business Intelligence Dashboard

The Business Intelligence Dashboard As a small business owner you would certainly know the importance of collecting and analyzing data pertaining to your business and transactions. Business Intelligence dashboards allow not only experts but you also to access information generated by analysis of data through a convenient display. Anyone in the company can have access…

4 Different Types of Attacks – Understanding the “Insider Threat”

4 Different Types of Attacks – Understanding the “Insider Threat”

Understanding the “Insider Threat”  The revelations that last month’s Sony hack was likely caused by a disgruntled former employee have put a renewed spotlight on the insider threat. The insider threat first received attention after Edward Snowden began to release all sorts of confidential information regarding national security. While many called him a hero, what…

The Big Data Movement Gets Bigger

The Big Data Movement Gets Bigger

The Big Data Movement In recent years, Big Data and Cloud relations have been growing steadily. And while there have been many questions raised around how best to use the information being gathered, there is no question that there is a real future between the two. The growing importance of Big Data Scientists and the…

M2M, IoT and Wearable Technology: Where To Next?

M2M, IoT and Wearable Technology: Where To Next?

M2M, IoT and Wearable Technology Profiling 600 companies and including 553 supporting tables and figures, recent reports into the M2M, IoT and Wearable Technology ecosystems forecast opportunities, challenges, strategies, and industry verticals for the sectors from 2015 to 2030. With many service providers looking for new ways to fit wearable technology with their M2M offerings…

Explosive Growth Of Data-Driven Marketing

Explosive Growth Of Data-Driven Marketing

Data-Driven Marketing There is an absolute endless amount of data that is being accumulated, dissected, analyzed with the important bits extracted and used for a number of purposes. With the amount of data in the world has already reached into multiple zettabytes annually. A Zettabyte is one million petabytes or one thousand exabytes. With data…

Cloud Infographic – The Data Scientist

Cloud Infographic – The Data Scientist

Data Scientist Report The amount of data in our world has been exploding in recent years. Managing big data has become an integral part of many businesses, generating billions of dollars of competitive innovations, productivity and job growth. Forecasting where the big data industry is going has become vital to corporate strategy. Enter the Data…

Five Cloud Questions Every CIO Needs To Know How To Answer

Five Cloud Questions Every CIO Needs To Know How To Answer

The Hot Seat Five cloud questions every CIO needs to know how to answer The cloud is a powerful thing, but here in the CloudTweaks community, we already know that. The challenge we have is validating the value it brings to today’s enterprise. Below, let’s review five questions we need to be ready to address…

Disaster Recovery And The Cloud

Disaster Recovery And The Cloud

Disaster Recovery And The Cloud One of the least considered benefits of cloud computing in the average small or mid-sized business manager’s mind is the aspect of disaster recovery. Part of the reason for this is that so few small and mid-size businesses have ever contemplated the impact of a major disaster on their IT…

Cloud Computing – The Game Changer

Cloud Computing – The Game Changer

Global Cloud Index In October, Cisco released its Global Cloud Index (GCI) report for 2014-2019, projecting a near 3-fold growth of global data center traffic, with predictions that this traffic will reach 8.6 zettabytes (cloud data center traffic) and 10.4 zettabytes (total data center traffic) per year in 2019 and 80% of it will come…

Disaster Recovery – A Thing Of The Past!

Disaster Recovery – A Thing Of The Past!

Disaster Recovery  Ok, ok – I understand most of you are saying disaster recovery (DR) is still a critical aspect of running any type of operations. After all – we need to secure our future operations in case of disaster. Sure – that is still the case but things are changing – fast. There are…