Storing Massive Files On The Cloud

Storing Massive Files On The Cloud

On December 9th, Amazon announced, in a blog post, an increase in the maximum size of file that could be stored on the Amazon Simple Storage Service (S3). Before the increase, the largest file that could be stored was 5GB – anything larger had to be split into chunks and put back together within an application, using an intermediate server or on the client device. Now, the limit has been increased to 5TB, an increase of approximately 1000 times.

This new limit allows the storage of large databases, video files or scientific data, but to make use of it properly, the object needs to be uploaded using Amazon’s new Multipart Upload API. The original objects are split into smaller parts and then uploaded separately before being stitched together on the Amazon S3 cloud. Objects can then be worked on either as a whole, or in part, on the Amazon Elastic Compute Cloud (EC2) before storing the finished product back on S3.

Data stored on Amazon S3 is designed to have 99.99% annual availability and to withstand the concurrent failure of data in two facilities, so you can be certain your data would be available at all times. For less critical data, Amazon’s Reduced Redundancy Storage (RRS) offers a cheaper way of storing your non-critical data, by utilizing less storage facilities. There is an associated loss of data durability, but the quoted availability is still 99.99%.

OpenStack have also announced that they are working on increasing the size of files that can be stored on its open-source private cloud system, and it is expected that a 5TB file size limit will be available with the “Bexar” release of OpenStack early in 2011.

A file size of 5TB is a massive amount of data, but it is clear that the cloud storage limits are going to increase even further in the near future. It is only a matter of time before the maximum object size is in the petabyte or even exabyte range.

The only question then is exactly how you can generate enough data to need that amount of storage! Amazon Web Services Blog

By Glenn Blake

About Glenn Blake

Glenn Blake is a writer for CloudTweaks and has been writing about technology trends for over 25 years.

View Website
View All Articles

Sorry, comments are closed for this post.

Comics
2017 Brings DLP Technology and IoT’s Weaknesses to Light

2017 Brings DLP Technology and IoT’s Weaknesses to Light

DLP Technology In regards to data loss prevention (DLP), in the last five years many companies rushed to implement DLP solutions without taking the time to first identify the data that should not transit egress points. Most of these rushed implementations have not been successful. Security analysts, in particular 451 Research, have been recommending that…

5 Things To Consider About Your Next Enterprise Sharing Solution

5 Things To Consider About Your Next Enterprise Sharing Solution

Enterprise File Sharing Solution Businesses have varying file sharing needs. Large, multi-regional businesses need to synchronize folders across a large number of sites, whereas small businesses may only need to support a handful of users in a single site. Construction or advertising firms require sharing and collaboration with very large (several Gigabytes) files. Financial services…

Why Security Practitioners Need To Apply The 80-20 Rules To Data Security

Why Security Practitioners Need To Apply The 80-20 Rules To Data Security

The 80-20 Rule For Security Practitioners  Everyday we learn about yet another egregious data security breach, exposure of customer data or misuse of data. It begs the question why in this 21st century, as a security industry we cannot seem to secure our most valuable data assets when technology has surpassed our expectations in other regards.…

Cyber Security: McAfee on IoT Threats and Autonomous Cars

Cyber Security: McAfee on IoT Threats and Autonomous Cars

IoT Threats and Autonomous Cars Autonomous cars are just around the corner, there have been controversies surrounding their safety, and a few doubts still hang in the minds of people who don’t like the idea of a computer driving their car. However, the biggest news stories surrounding this topic have been to do with how…

Three Factors For Choosing Your Long-term Cloud Strategy

Three Factors For Choosing Your Long-term Cloud Strategy

Choosing Your Long-term Cloud Strategy A few weeks ago I visited the global headquarters of a large multi-national company to discuss cloud strategy with the CIO. I arrived 30 minutes early and took a tour of the area where the marketing team showcased their award winning brands. I was impressed by the digital marketing strategy…

How To Overcome Data Insecurity In The Cloud

How To Overcome Data Insecurity In The Cloud

Data Insecurity In The Cloud Today’s escalating attacks, vulnerabilities, breaches, and losses have cut deeply across organizations and captured the attention of, regulators, investors and most importantly customers. In many cases such incidents have completely eroded customer trust in a company, its services and its employees. The challenge of ensuring data security is far more…

Ending The Great Enterprise Disconnect

Ending The Great Enterprise Disconnect

Five Requirements for Supporting a Connected Workforce It used to be that enterprises dictated how workers spent their day: stuck in a cubicle, tied to an enterprise-mandated computer, an enterprise-mandated desk phone with mysterious buttons, and perhaps an enterprise-mandated mobile phone if they traveled. All that is history. Today, a modern workforce is dictating how…

Digital Identity Trends 2017 – Previewing The Year Ahead

Digital Identity Trends 2017 – Previewing The Year Ahead

Digital Identity Trends 2017 The lack of security of the Internet of Things captured public attention this year as massive distributed denial of service attacks took down much of the internet. The culprits? Unsecured connected devices that were easily accessed and manipulated to do the bidding of shadowy hackers. When you can’t access Netflix anymore,…

The Rise Of BI Data And How To Use It Effectively

The Rise Of BI Data And How To Use It Effectively

The Rise of BI Data Every few years, a new concept or technological development is introduced that drastically improves the business world as a whole. In 1983, the first commercially handheld mobile phone debuted and provided workers with an unprecedented amount of availability, leading to more productivity and profits. More recently, the Cloud has taken…