Storing Massive Files On The Cloud

Storing Massive Files On The Cloud

On December 9th, Amazon announced, in a blog post, an increase in the maximum size of file that could be stored on the Amazon Simple Storage Service (S3). Before the increase, the largest file that could be stored was 5GB – anything larger had to be split into chunks and put back together within an application, using an intermediate server or on the client device. Now, the limit has been increased to 5TB, an increase of approximately 1000 times.

This new limit allows the storage of large databases, video files or scientific data, but to make use of it properly, the object needs to be uploaded using Amazon’s new Multipart Upload API. The original objects are split into smaller parts and then uploaded separately before being stitched together on the Amazon S3 cloud. Objects can then be worked on either as a whole, or in part, on the Amazon Elastic Compute Cloud (EC2) before storing the finished product back on S3.

Data stored on Amazon S3 is designed to have 99.99% annual availability and to withstand the concurrent failure of data in two facilities, so you can be certain your data would be available at all times. For less critical data, Amazon’s Reduced Redundancy Storage (RRS) offers a cheaper way of storing your non-critical data, by utilizing less storage facilities. There is an associated loss of data durability, but the quoted availability is still 99.99%.

OpenStack have also announced that they are working on increasing the size of files that can be stored on its open-source private cloud system, and it is expected that a 5TB file size limit will be available with the “Bexar” release of OpenStack early in 2011.

A file size of 5TB is a massive amount of data, but it is clear that the cloud storage limits are going to increase even further in the near future. It is only a matter of time before the maximum object size is in the petabyte or even exabyte range.

The only question then is exactly how you can generate enough data to need that amount of storage! Amazon Web Services Blog

By Glenn Blake

About Glenn Blake

Glenn Blake is a writer for CloudTweaks and has been writing about technology trends for over 25 years.

View Website
View All Articles

Sorry, comments are closed for this post.

The Importance of Cloud Backups: Guarding Your Data Against Hackers

The Importance of Cloud Backups: Guarding Your Data Against Hackers

The Importance of Cloud Backups Cloud platforms have become a necessary part of modern business with the benefits far outweighing the risks. However, the risks are real and account for billions of dollars in losses across the globe per year. If you’ve been hacked, you’re not alone. Here are some other companies in the past…

Digital Twin And The End Of The Dreaded Product Recall

Digital Twin And The End Of The Dreaded Product Recall

The Digital Twin  How smart factories and connected assets in the emerging Industrial IoT era along with the automation of machine learning and advancement of artificial intelligence can dramatically change the manufacturing process and put an end to the dreaded product recalls in the future. In recent news, Samsung Electronics Co. has initiated a global…

Ending The Great Enterprise Disconnect

Ending The Great Enterprise Disconnect

Five Requirements for Supporting a Connected Workforce It used to be that enterprises dictated how workers spent their day: stuck in a cubicle, tied to an enterprise-mandated computer, an enterprise-mandated desk phone with mysterious buttons, and perhaps an enterprise-mandated mobile phone if they traveled. All that is history. Today, a modern workforce is dictating how…

Lavabit, Edward Snowden and the Legal Battle For Privacy

Lavabit, Edward Snowden and the Legal Battle For Privacy

The Legal Battle For Privacy In early June 2013, Edward Snowden made headlines around the world when he leaked information about the National Security Agency (NSA) collecting the phone records of tens of millions of Americans. It was a dramatic story. Snowden flew to Hong Kong and then Russia to avoid deportation to the US,…

Four Recurring Revenue Imperatives

Four Recurring Revenue Imperatives

Revenue Imperatives “Follow the money” is always a good piece of advice, but in today’s recurring revenue-driven market, “follow the customer” may be more powerful. Two recurring revenue imperatives highlight the importance of responding to, and cherishing customer interactions. Technology and competitive advantage influence the final two. If you’re part of the movement towards recurring…

Cloud-Based Services vs. On-Premises: It’s About More Than Just Dollars

Cloud-Based Services vs. On-Premises: It’s About More Than Just Dollars

Cloud-Based Services vs. On-Premises The surface costs might give you pause, but the cost of diminishing your differentiators is far greater. Will a shift to the cloud save you money? Potential savings are historically the main business driver cited when companies move to the cloud, but it shouldn’t be viewed as a cost-saving exercise. There…

Data Breaches: Incident Response Planning – Part 1

Data Breaches: Incident Response Planning – Part 1

Incident Response Planning – Part 1 The topic of cybersecurity has become part of the boardroom agendas in the last couple of years, and not surprisingly — these days, it’s almost impossible to read news headlines without noticing yet another story about a data breach. As cybersecurity shifts from being a strictly IT issue to…

Choosing IaaS or a Cloud-Enabled Managed Hosting Provider?

Choosing IaaS or a Cloud-Enabled Managed Hosting Provider?

There is a Difference – So Stop Comparing We are all familiar with the old saying “That’s like comparing apples to oranges” and though we learned this lesson during our early years we somehow seem to discount this idiom when discussing the Cloud. Specifically, IT buyers often feel justified when comparing the cost of a…

The Rise Of BI Data And How To Use It Effectively

The Rise Of BI Data And How To Use It Effectively

The Rise of BI Data Every few years, a new concept or technological development is introduced that drastically improves the business world as a whole. In 1983, the first commercially handheld mobile phone debuted and provided workers with an unprecedented amount of availability, leading to more productivity and profits. More recently, the Cloud has taken…