Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Big Data has all the corporate heads up and about in excitement since it promises to uncover golden nuggets of information out from an ocean of mundane and redundant data. But here’s the problem sticking everybody in the side, Big Data is big, as in it can reach the levels of “we-can’t-come-up-with-enough-names” bytes big. And with current upload speeds nowhere near as fast as download speeds, all the fancy analytics software and techniques aren’t going to do us any good if we can’t get our data where we need them.

bandwidth-straw

It is called the Skinny Straw or Drinking Straw problem and it is the biggest and most obvious problem being faced by Big Data. The analogy is simple; imagine passing an elephant through a drinking straw. Sure you can grind the elephant into very tiny bits so it can fit through the straw, but how long is that going to take? I admit that was a little gory, the real analogy was filling a swimming pool using a drinking straw, but you get the picture. The straw represents bandwidth and how small it is compared to the amount of data that needs to get to the other side of that straw.

The only real solution we can think of right off the bat is to get a bigger straw, but usually that would require major infrastructure upgrades on the part of the ISP or backbone provider, and we are talking about extreme amounts of cash (or credit if that’s how you roll). There are also the obvious technology limitations, we can upgrade to the best there is and it might not still be 100% enough. Some Big Data providers have tried their own proprietary ideas to try and get around this issue, or at least lessen it to some degree.

Here are some ways and techniques that are being used in the industry right now:

  1. We have the data compression and de-duplication techniques to make data transfers faster. That’s the “grinding the elephant and pushing it through the straw as fast as possible” solution.
  2. There is the “tinker with current protocols” direction by combining the reliability of TCP connections and the speed and bandwidth of UDP transfers into something that they call FASP. This ensures that communication is fast and secure while doing away with various handshaking processes that TCP requires.
  3. We can also work with various protocol optimizations in order to get around the problem. But one way that is really worth mentioning is the tried and tested transfer method –the old SneakerNet approach. Providers that use this method allow their customers to mail their hard drives to the company address so that they can transfer the data and then mail the hard drives back. This method is often faster at moving extremely large amounts of data quickly even taking into consideration the delivery time.

By Abdul Salam

(Image Source: ShutterStock)

Abdul

Abdul Salam is IT professional and an accomplished technical writer with CloudTweaks. He earned his undergraduate degree in Information Technology followed by a postgraduate degree in Business Informatics. Abdul possess over 3 years’ experience in technical & business writing with deep knowledge in Cloud Computing, VMware,Oracle, Oracle ERP, Cloud ERP, Microsoft Technologies and Network Communications (Cisco, Juniper). Visit his LinkedIn profile at: http://linkd.in/TtFu7X

Sorry, comments are closed for this post.

Join Our Newsletter

Receive updates each week on news, tips, events, comics and much more...

Can I Contribute To CloudTweaks?

Yes, much of our focus in 2015 will be on working with other influencers in a collaborative manner. If you're a technology influencer looking to collaborate long term with CloudTweaks – a globally recognized leader in cloud computing information – drop us an email with “tech influencer” in the subject line.

Please review the guidelines before applying.

Contributors

Cloud Infographic – Wearable Tech And Preventative Healthcare

Cloud Infographic – Wearable Tech And Preventative Healthcare

Wearable Tech And Preventative Healthcare There are so many exciting new opportunities available to utilize wearable technology in the future.  Areas such as nanotechnology disease monitoring, crowdfunding to wearable accessories are some excellent examples of the potential. Estimates vary, but appear to suggest that the market will produce between $14-50 Billion over the next few years. Included below

Ten Tips For Successful Business Intelligence Implementation

Ten Tips For Successful Business Intelligence Implementation

Ten Tips for Successful Business Intelligence Implementation The cost of Business Intelligence (BI) software goes far beyond the purchase price. Time spent researching, implementing, and maintaining your BI investment can snowball quickly and mistakes are often expensive. Your time is valuable – save it by learning from other businesses’ experiences. We’ve compiled the top ten

Knots And Cloud Service Providers

Knots And Cloud Service Providers

How Do These Two Compare? In Boy Scouts, I learned how to tie knots. The quickest knot you can tie is the slipknot. It’s very effective for connecting one thing to another via the rope you have. It was used in setting up tents, mooring boats to docks temporarily and lifting your food up into

Aggregated News

Popular News Sources

Storage Considerations for SharePoint Backups

Storage Considerations for SharePoint Backups

Storage Considerations for SharePoint Backups Wednesday, October 29, 2014 @ 9:00 am/12:00pm ET. Backup and Restore of a SharePoint environment can be a complex endeavor as the product consists of multiple components running at various tiers, each with their own backup and restore requirements. In addition, SharePoint documents are stored as Binary Large Objects (BLOBs) in

OpenDNS Deployment Leads to Twenty-Fold Decrease in Malware Infections at Hamamatsu

OpenDNS Deployment Leads to Twenty-Fold Decrease in Malware Infections at Hamamatsu

Decreases in Malware Infections at Hamamatsu OpenDNS, a leading provider of cloud-delivered security, today announced that it has enabled Hamamatsu, a Japanese manufacturer of optical sensor technologies, to virtually eliminate malware infections across its U.S. Read the source article at Finance News About Latest Posts AbdulAbdul Salam is IT professional and an accomplished technical writer with

IBM and Microsoft – What Are They Doing With The Hybrid Cloud?

IBM and Microsoft – What Are They Doing With The Hybrid Cloud?

What Are They Doing With The Hybrid Cloud? “Microsoft is committed to helping enterprise customers realize the tremendous benefits of cloud computing across their own systems, partner clouds and Microsoft Azure,” said Scott Guthrie, executive vice president,Cloud and Enterprise, Microsoft. “With this … Read the source article at CNNMoney About Latest Posts AbdulAbdul Salam is IT professional and an