Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Big Data has all the corporate heads up and about in excitement since it promises to uncover golden nuggets of information out from an ocean of mundane and redundant data. But here’s the problem sticking everybody in the side, Big Data is big, as in it can reach the levels of “we-can’t-come-up-with-enough-names” bytes big. And with current upload speeds nowhere near as fast as download speeds, all the fancy analytics software and techniques aren’t going to do us any good if we can’t get our data where we need them.

bandwidth-straw

It is called the Skinny Straw or Drinking Straw problem and it is the biggest and most obvious problem being faced by Big Data. The analogy is simple; imagine passing an elephant through a drinking straw. Sure you can grind the elephant into very tiny bits so it can fit through the straw, but how long is that going to take? I admit that was a little gory, the real analogy was filling a swimming pool using a drinking straw, but you get the picture. The straw represents bandwidth and how small it is compared to the amount of data that needs to get to the other side of that straw.

The only real solution we can think of right off the bat is to get a bigger straw, but usually that would require major infrastructure upgrades on the part of the ISP or backbone provider, and we are talking about extreme amounts of cash (or credit if that’s how you roll). There are also the obvious technology limitations, we can upgrade to the best there is and it might not still be 100% enough. Some Big Data providers have tried their own proprietary ideas to try and get around this issue, or at least lessen it to some degree.

Here are some ways and techniques that are being used in the industry right now:

  1. We have the data compression and de-duplication techniques to make data transfers faster. That’s the “grinding the elephant and pushing it through the straw as fast as possible” solution.
  2. There is the “tinker with current protocols” direction by combining the reliability of TCP connections and the speed and bandwidth of UDP transfers into something that they call FASP. This ensures that communication is fast and secure while doing away with various handshaking processes that TCP requires.
  3. We can also work with various protocol optimizations in order to get around the problem. But one way that is really worth mentioning is the tried and tested transfer method –the old SneakerNet approach. Providers that use this method allow their customers to mail their hard drives to the company address so that they can transfer the data and then mail the hard drives back. This method is often faster at moving extremely large amounts of data quickly even taking into consideration the delivery time.

By Abdul Salam

(Image Source: ShutterStock)

About Abdul

Abdul is a senior consultant with Energy Services, and author of numerous blogs, books, white papers, and tutorials on cloud computing and accomplished technical writer with CloudTweaks. He earned his bachelor’s degree in Information Technology, followed by an MBA-IT degree and certifications by Cisco and Juniper Networks.

He has recently co-authored: Deploying and Managing a Cloud Infrastructure: Real-World Skills for the CompTIA Cloud+ Certification (Wiley).

View All Articles

Sorry, comments are closed for this post.

Finally, The Time For Security Information Event Management (SIEM)

Finally, The Time For Security Information Event Management (SIEM)

The Time For SIEM Security Information Event Management (SIEM) tools have been around for a long time. My first encounter with a SIEM vendor was about twenty years ago while being courted to resell their product. To this day, I still recall two vivid memories from that meeting; the product was very complex and quite…

How Will The Internet of Things Affect The Business World?

How Will The Internet of Things Affect The Business World?

Internet of Things and the Business World? Experts predict that by 2020, the Internet of Things will have a market greater than $1 Trillion, yet despite it being the new buzzword, many people have no idea how this predicted revolution will shape the business world. We’ve all heard predictions of various appliances with sensors, wearable…

Cloud Computing: On The Fringe or Mainstream?

Cloud Computing: On The Fringe or Mainstream?

Cloud Computing Revenue Numbers Imagine that your viewpoint is shared by 5% of people. You’d be in a fringe political party, perhaps, or share a fondness for some obscure sport or hobby. Or you’d be a proponent of Cloud Computing. Because the reality is that after many years of technical development, entrepreneurial freneticism, and big-company marketing,…

The Evolution Of The Connected Cloud

The Evolution Of The Connected Cloud

The Connected Cloud Cloud computing is interesting first, but not only, because of the prevalence of cloud projects. There are many of them launched every day. Some have lofty expectations for business benefits (cost saving of 20 percent or more) and others carry even more intriguing goals. In 2005 “the cloud” was new. Shared computing…

Connecting To Information With Cyber Physical Systems

Connecting To Information With Cyber Physical Systems

CPS Device Trends On The Rise It isn’t, “Do you remember who starred in XYZ Movie?” It’s, “Can you look it up please?” “Did you ever think you would sit at the dinner table, and when a question came up, someone would look up the answer and share it with everyone?” The words echoed at…

4 Cloud Technology Trends To Look Out For

4 Cloud Technology Trends To Look Out For

Cloud Technology Trends When you are reading articles on the future of tech on the Internet you cannot escape mention of the cloud: it is set to be the best thing that has ever happened to us, pundits assure us, and there is a promise of some serious money to be made. Reading these articles you can tell there is…

Cloud Pricing: What In The End Am I Actually Paying For?

Cloud Pricing: What In The End Am I Actually Paying For?

Cloud Pricing: What In The End Am I Actually Paying For? I’ve been blogging about this concept now for a while. What is the cost of a cloud solution and what in the end am I actually paying for? I know I am not the only person considering this as I’ve seen any number of articles…

CloudTweaks is recognized as one of the leading influencers in cloud computing, infosec, big data and the internet of things (IoT) information. Our goal is to continue to build our growing information portal by providing the best in-depth articles, interviews, event listings, whitepapers, infographics and much more.

Sponsor