Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Passing Big Data Through A Drinking Straw

Big Data has all the corporate heads up and about in excitement since it promises to uncover golden nuggets of information out from an ocean of mundane and redundant data. But here’s the problem sticking everybody in the side, Big Data is big, as in it can reach the levels of “we-can’t-come-up-with-enough-names” bytes big. And with current upload speeds nowhere near as fast as download speeds, all the fancy analytics software and techniques aren’t going to do us any good if we can’t get our data where we need them.

bandwidth-straw

It is called the Skinny Straw or Drinking Straw problem and it is the biggest and most obvious problem being faced by Big Data. The analogy is simple; imagine passing an elephant through a drinking straw. Sure you can grind the elephant into very tiny bits so it can fit through the straw, but how long is that going to take? I admit that was a little gory, the real analogy was filling a swimming pool using a drinking straw, but you get the picture. The straw represents bandwidth and how small it is compared to the amount of data that needs to get to the other side of that straw.

The only real solution we can think of right off the bat is to get a bigger straw, but usually that would require major infrastructure upgrades on the part of the ISP or backbone provider, and we are talking about extreme amounts of cash (or credit if that’s how you roll). There are also the obvious technology limitations, we can upgrade to the best there is and it might not still be 100% enough. Some Big Data providers have tried their own proprietary ideas to try and get around this issue, or at least lessen it to some degree.

Here are some ways and techniques that are being used in the industry right now:

  1. We have the data compression and de-duplication techniques to make data transfers faster. That’s the “grinding the elephant and pushing it through the straw as fast as possible” solution.
  2. There is the “tinker with current protocols” direction by combining the reliability of TCP connections and the speed and bandwidth of UDP transfers into something that they call FASP. This ensures that communication is fast and secure while doing away with various handshaking processes that TCP requires.
  3. We can also work with various protocol optimizations in order to get around the problem. But one way that is really worth mentioning is the tried and tested transfer method –the old SneakerNet approach. Providers that use this method allow their customers to mail their hard drives to the company address so that they can transfer the data and then mail the hard drives back. This method is often faster at moving extremely large amounts of data quickly even taking into consideration the delivery time.

By Abdul Salam

(Image Source: ShutterStock)

Sorry, comments are closed for this post.

Comic
Fintech Systems, Advancements and Investments

Fintech Systems, Advancements and Investments

Fintech Growth According to a recent report, global investment in fintech companies including both venture-backed and non-venture-backed businesses reached $9.4 billion in the second quarter of 2016; investment in venture capital-backed fintech startups, however, fell by 49%. Nevertheless, the Pulse of Fintech, published jointly by KPMG International and CB Insights, suggests venture capital investment in…

How Identity Governance Can Secure The Cloud Enterprise

How Identity Governance Can Secure The Cloud Enterprise

Securing The Cloud Enterprise Cloud adoption is accelerating for most enterprises, and cloud computing is becoming an integral part of enterprise IT and security infrastructure. Based on current adoption trends, it’s clear that the vast majority of new applications purchased by organizations will be SaaS applications. The allure is evident, from cost savings to speed…

Significant Emerging Technologies To Lookout For In 2017

Significant Emerging Technologies To Lookout For In 2017

Emerging Technologies The entire world is being transformed right before our eyes. Emerging technologies are developing at break-neck speeds, and the global community needs to be prepared for what lies in the horizon. As with anything new or evolving there is benefit versus risk to consider. Most of the up-and-coming technologies that will soon affect…

In The Fast Lane: Connected Car Hacking A Big Risk

In The Fast Lane: Connected Car Hacking A Big Risk

Connected Car Hacking Researchers and cybersecurity experts working hard to keep hackers out of the driver’s seat. Modern transportation has come a million miles, and most all of today’s vehicles are controlled entirely by digital technology. Millions of drivers are not aware that of the many devices in their digital arsenal, the most complex of…

Having Your Cybersecurity And Eating It Too

Having Your Cybersecurity And Eating It Too

The Catch 22 The very same year Marc Andreessen famously said that software was eating the world, the Chief Information Officer of the United States was announcing a major Cloud First goal. That was 2011. Five years later, as both the private and public sectors continue to adopt cloud-based software services, we’re interested in this…

Why Security Practitioners Need To Apply The 80-20 Rules To Data Security

Why Security Practitioners Need To Apply The 80-20 Rules To Data Security

The 80-20 Rule For Security Practitioners  Everyday we learn about yet another egregious data security breach, exposure of customer data or misuse of data. It begs the question why in this 21st century, as a security industry we cannot seem to secure our most valuable data assets when technology has surpassed our expectations in other regards.…

What You Need To Know About Choosing A Cloud Services Provider

What You Need To Know About Choosing A Cloud Services Provider

Selecting The Right Cloud Services Provider How to find the right partner for cloud adoption on an enterprise scale The cloud is capable of delivering many benefits, enabling greater collaboration, business agility, and speed to market. Cloud adoption in the enterprise has been growing fast. Worldwide spending on public cloud services will grow at a…

Moving To The Cloud: A Smart Move Or Proceed With Caution?

Moving To The Cloud: A Smart Move Or Proceed With Caution?

Moving To The Cloud You hear it over and over again: The future of computing is in the cloud. Businesses should be moving to the cloud. If you’re not in the cloud, you’re behind the times. But trusting your essential data to a cloud may sound like a dubious proposition. Below are some of the…

Are Cloud Solutions Secure Enough Out-of-the-box?

Are Cloud Solutions Secure Enough Out-of-the-box?

Out-of-the-box Cloud Solutions Although people may argue that data is not safe in the Cloud because using cloud infrastructure requires trusting another party to look after mission critical data, cloud services actually are more secure than legacy systems. In fact, a recent study on the state of cloud security in the enterprise market revealed that…

Using Cloud Technology In The Education Industry

Using Cloud Technology In The Education Industry

Education Tech and the Cloud Arguably one of society’s most important functions, teaching can still seem antiquated at times. Many schools still function similarly to how they did five or 10 years ago, which is surprising considering the amount of technical innovation we’ve seen in the past decade. Education is an industry ripe for innovation…

Digital Marketing Driven by Cloud, Big Data and IoT

Digital Marketing Driven by Cloud, Big Data and IoT

Digital Marketing Successful digital marketing campaigns are being driven largely by trending technologies, specifically the Internet of Things (IoT), Big Data, and The Cloud. These may be used for a huge number of marketing applications, from optimizing the performance of sports teams to improving science and research, even helping to aid law enforcement. Amazon Web…

Disaster Recovery And The Cloud

Disaster Recovery And The Cloud

Disaster Recovery And The Cloud One of the least considered benefits of cloud computing in the average small or mid-sized business manager’s mind is the aspect of disaster recovery. Part of the reason for this is that so few small and mid-size businesses have ever contemplated the impact of a major disaster on their IT…

Cloud Infographic – Guide To Small Business Cloud Computing

Cloud Infographic – Guide To Small Business Cloud Computing

Small Business Cloud Computing Trepidation is inherently attached to anything that involves change and especially if it involves new technologies. SMBs are incredibly vulnerable to this fear and rightfully so. The wrong security breach can incapacitate a small startup for good whereas larger enterprises can reboot their operations due to the financial stability of shareholders. Gordon Tan contributed an…

Who’s Who In The Booming World Of Data Science

Who’s Who In The Booming World Of Data Science

The World of Data Science The nature of work and business in today’s super-connected world means that every second of every day, the world produces an astonishing amount of data. Consider some of these statistics; every minute, Facebook users share nearly 2.5 million pieces of content, YouTube users upload over 72 hours of content, Apple…

Low Cost Cloud Computing Gives Rise To Startups

Low Cost Cloud Computing Gives Rise To Startups

Balancing The Playing Field For Startups According to a Goldman Sachs report, cloud infrastructure and platform spending could reach $43 billion by 2018, which is up $16 billion from last year, representing a growth of around 30% from 2013 said the analyst. This phenomenal growth is laying the foundation for a new breed of startup…

Moving Your Enterprise Apps To The Cloud Is A Business Decision

Moving Your Enterprise Apps To The Cloud Is A Business Decision

Moving Your Enterprise Apps Whether it be enterprise apps or any other, if there is any heavy data that is going to be transacted in and through an app, then affiliating it with the Cloud becomes a must. And then an important question arises: How do you decide when to integrate your enterprise app with…