Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Big Data has all the corporate heads up and about in excitement since it promises to uncover golden nuggets of information out from an ocean of mundane and redundant data. But here’s the problem sticking everybody in the side, Big Data is big, as in it can reach the levels of “we-can’t-come-up-with-enough-names” bytes big. And with current upload speeds nowhere near as fast as download speeds, all the fancy analytics software and techniques aren’t going to do us any good if we can’t get our data where we need them.

bandwidth-straw

It is called the Skinny Straw or Drinking Straw problem and it is the biggest and most obvious problem being faced by Big Data. The analogy is simple; imagine passing an elephant through a drinking straw. Sure you can grind the elephant into very tiny bits so it can fit through the straw, but how long is that going to take? I admit that was a little gory, the real analogy was filling a swimming pool using a drinking straw, but you get the picture. The straw represents bandwidth and how small it is compared to the amount of data that needs to get to the other side of that straw.

The only real solution we can think of right off the bat is to get a bigger straw, but usually that would require major infrastructure upgrades on the part of the ISP or backbone provider, and we are talking about extreme amounts of cash (or credit if that’s how you roll). There are also the obvious technology limitations, we can upgrade to the best there is and it might not still be 100% enough. Some Big Data providers have tried their own proprietary ideas to try and get around this issue, or at least lessen it to some degree.

Here are some ways and techniques that are being used in the industry right now:

  1. We have the data compression and de-duplication techniques to make data transfers faster. That’s the “grinding the elephant and pushing it through the straw as fast as possible” solution.
  2. There is the “tinker with current protocols” direction by combining the reliability of TCP connections and the speed and bandwidth of UDP transfers into something that they call FASP. This ensures that communication is fast and secure while doing away with various handshaking processes that TCP requires.
  3. We can also work with various protocol optimizations in order to get around the problem. But one way that is really worth mentioning is the tried and tested transfer method –the old SneakerNet approach. Providers that use this method allow their customers to mail their hard drives to the company address so that they can transfer the data and then mail the hard drives back. This method is often faster at moving extremely large amounts of data quickly even taking into consideration the delivery time.

By Abdul Salam

(Image Source: ShutterStock)

Sorry, comments are closed for this post.

Comic
The Lighter Side Of The Cloud – Data Merge

The Lighter Side Of The Cloud – Data Merge

By Christian Mirra Please feel free to share our comics via social media networks such as Twitter, Facebook, LinkedIn, Instagram, Pinterest. Clear attribution (Twitter example: via @cloudtweaks) to our original comic sources is greatly appreciated.

The Rise Of Threat Intelligence Sharing

The Rise Of Threat Intelligence Sharing

Threat Intelligence Sharing  Security has been discussed often on CloudTweaks and for good reason. It is one of the most sought after topics of information in the technology industry.  It is virtually impossible to wake up and not read a headline that involves the words “Breached, Hacked, Compromised or Extorted (Ransomware)“. Included (below) is an…

Moving Your Email To The Cloud? Beware Of Unintentional Data Spoliation!

Moving Your Email To The Cloud? Beware Of Unintentional Data Spoliation!

Cloud Email Migration In today’s litigious society, preserving your company’s data is a must if you (and your legal team) want to avoid hefty fines for data spoliation. But what about when you move to the cloud? Of course, you’ve probably thought of this already. You’ll have a migration strategy in place and you’ll carefully…

Higher Education Institutions Increasing Cloud Use In Next 5 Years

Higher Education Institutions Increasing Cloud Use In Next 5 Years

Cloud Computing Advancing Edtech In a new research study by ResearchMoz it’s predicted that the global cloud computing market in higher education will grow steadily at a CAGR of 24.57% over the period 2016 to 2020. Making use of computing resources connected by either public or private networks provides the benefits of scalable infrastructure, greater…

Big Data and AI Hold Greatest Promise For Healthcare Technologies

Big Data and AI Hold Greatest Promise For Healthcare Technologies

Digital Healthcare Executives and Investors Addressed Opportunities and Challenges Facing the Industry New York City – September 21, 2016 – According to a survey of 122 founders, executives and investors in health-tech companies released today by Silicon Valley Bank, big data and artificial intelligence will have the greatest impact on the industry in the year ahead. Healthcare…

The Cloud Is Not Enough! Why Businesses Need Hybrid Solutions

The Cloud Is Not Enough! Why Businesses Need Hybrid Solutions

Why Businesses Need Hybrid Solutions Running a cloud server is no longer the novel trend it once was. Now, the cloud is a necessary data tier that allows employees to access vital company data and maintain productivity from anywhere in the world. But it isn’t a perfect system — security and performance issues can quickly…

HOW THE CFAA RULING AFFECTS INDIVIDUALS AND PASSWORD-SHARING

HOW THE CFAA RULING AFFECTS INDIVIDUALS AND PASSWORD-SHARING

Individuals and Password-Sharing With the 1980s came the explosion of computing. In 1980, the Commodore ushered in the advent of home computing. Time magazine declared 1982 was “The Year of the Computer.” By 1983, there were an estimated 10 million personal computers in the United States alone. As soon as computers became popular, the federal government…

Connecting With Customers In The Cloud

Connecting With Customers In The Cloud

Customers in the Cloud Global enterprises in every industry are increasingly turning to cloud-based innovators like Salesforce, ServiceNow, WorkDay and Aria, to handle critical systems like billing, IT services, HCM and CRM. One need look no further than Salesforce’s and Amazon’s most recent earnings report, to see this indeed is not a passing fad, but…

5 THINGS TO CONSIDER ABOUT YOUR NEXT ENTERPRISE FILE SHARING SOLUTION

5 THINGS TO CONSIDER ABOUT YOUR NEXT ENTERPRISE FILE SHARING SOLUTION

Enterprise File Sharing Solution Businesses have varying file sharing needs. Large, multi-regional businesses need to synchronize folders across a large number of sites, whereas small businesses may only need to support a handful of users in a single site. Construction or advertising firms require sharing and collaboration with very large (several Gigabytes) files. Financial services…

Achieving Network Security In The IoT

Achieving Network Security In The IoT

Security In The IoT The network security market is experiencing a pressing and transformative change, especially around access control and orchestration. Although it has been mature for decades, the network security market had to transform rapidly with the advent of the BYOD trend and emergence of the cloud, which swept enterprises a few years ago.…

Protecting Your Web Applications In A Hybrid Cloud Environment

Protecting Your Web Applications In A Hybrid Cloud Environment

Protecting Your Web Applications It’s no secret that organizations are embracing the cloud and all the benefits that it entails. Whether its cost savings, increased flexibility or enhanced productivity – businesses around the world are leveraging the cloud to scale their business and better serve their customers. They are using a variety of cloud solutions…

Cloud Computing Is Greener Than You Think

Cloud Computing Is Greener Than You Think

Cloud Computing Is Greener Than You Think Last week we touched upon how a project in Finland had blended two of the world’s most important industries, cloud computing and green technology, to produce a data centre that used nearby sea water to both cool their servers and heat local homes.  Despite such positive environmental projects, there…

The Future Of Cybersecurity

The Future Of Cybersecurity

The Future of Cybersecurity In 2013, President Obama issued an Executive Order to protect critical infrastructure by establishing baseline security standards. One year later, the government announced the cybersecurity framework, a voluntary how-to guide to strengthen cybersecurity and meanwhile, the Senate Intelligence Committee voted to approve the Cybersecurity Information Sharing Act (CISA), moving it one…

Disaster Recovery – A Thing Of The Past!

Disaster Recovery – A Thing Of The Past!

Disaster Recovery  Ok, ok – I understand most of you are saying disaster recovery (DR) is still a critical aspect of running any type of operations. After all – we need to secure our future operations in case of disaster. Sure – that is still the case but things are changing – fast. There are…

Infographic Introduction – Benefits of Cloud Computing

Infographic Introduction – Benefits of Cloud Computing

Benefits of Cloud Computing Based on Aberdeen Group’s Computer Intelligence Dataset, there are more than 1.6 billion permutations to choose from when it comes to cloud computing solutions. So what, on the face of it, appears to be pretty simple is actually both complex and dynamic regardless of whether you’re in the market for networking,…

Protecting Devices From Data Breach: Identity of Things (IDoT)

Protecting Devices From Data Breach: Identity of Things (IDoT)

How to Identify and Authenticate in the Expanding IoT Ecosystem It is a necessity to protect IoT devices and their associated data. As the IoT ecosystem continues to expand, the need to create an identity to newly-connected things is becoming increasingly crucial. These ‘things’ can include anything from basic sensors and gateways to industrial controls…