Storing Massive Files On The Cloud

Storing Massive Files On The Cloud

On December 9th, Amazon announced, in a blog post, an increase in the maximum size of file that could be stored on the Amazon Simple Storage Service (S3). Before the increase, the largest file that could be stored was 5GB – anything larger had to be split into chunks and put back together within an application, using an intermediate server or on the client device. Now, the limit has been increased to 5TB, an increase of approximately 1000 times.

This new limit allows the storage of large databases, video files or scientific data, but to make use of it properly, the object needs to be uploaded using Amazon’s new Multipart Upload API. The original objects are split into smaller parts and then uploaded separately before being stitched together on the Amazon S3 cloud. Objects can then be worked on either as a whole, or in part, on the Amazon Elastic Compute Cloud (EC2) before storing the finished product back on S3.

Data stored on Amazon S3 is designed to have 99.99% annual availability and to withstand the concurrent failure of data in two facilities, so you can be certain your data would be available at all times. For less critical data, Amazon’s Reduced Redundancy Storage (RRS) offers a cheaper way of storing your non-critical data, by utilizing less storage facilities. There is an associated loss of data durability, but the quoted availability is still 99.99%.

OpenStack have also announced that they are working on increasing the size of files that can be stored on its open-source private cloud system, and it is expected that a 5TB file size limit will be available with the “Bexar” release of OpenStack early in 2011.

A file size of 5TB is a massive amount of data, but it is clear that the cloud storage limits are going to increase even further in the near future. It is only a matter of time before the maximum object size is in the petabyte or even exabyte range.

The only question then is exactly how you can generate enough data to need that amount of storage! Amazon Web Services Blog

By Glenn Blake

About Glenn Blake

Glenn Blake is a writer for CloudTweaks and has been writing about technology trends for over 25 years.

Find out more
View All Articles

Sorry, comments are closed for this post.

5 Ways The Latest Cloud Tools Can Help Increase Sales

5 Ways The Latest Cloud Tools Can Help Increase Sales

Cloud Tools Can Help Increase Sales How new cloud technology can help you source, manage and close more leads. We take a look at how you can use cloud software to boost your sales team’s productivity and clinch more of those all-important deals! Selling and Marketing is integral to every business and a company’s sales team…

Hot Emerging Trends – The Pizza Delivery Drone

Hot Emerging Trends – The Pizza Delivery Drone

The Pizza Delivery Drone Recently, drone delivery systems have been discussed as future state plans for many vendors. I personally think the traceable IoT-based pizza delivery drone is the next big thing. If you think about it, your pizza makes it to your house much faster. Since the drone carries both a temperature and a…

Transforming Traditional DevOps To A Modern Cloud-Centric Operation

Transforming Traditional DevOps To A Modern Cloud-Centric Operation

Transforming Traditional DevOps Over the last year, I’ve been hearing about more and more instances of companies asking the question that the title suggests – how do you transform a DevOPs process into a more cloud-centric operation? To start, we must all assume that there is some notion of a traditional DevOPs process built into…

The Conflict Of Net Neutrality And DDoS-Attacks!

The Conflict Of Net Neutrality And DDoS-Attacks!

The Conflict Of Net Neutrality And DDoS-Attacks! So we are all cheering as the FCC last week made the right choice in upholding the principle of net neutrality! For the general public it is a given that an ISP should be allowed to charge for bandwidth and Internet access but never to block or somehow…

What Agency Can Benefit The Most From IoT/CSP?

What Agency Can Benefit The Most From IoT/CSP?

What Agency Can Benefit The Most? I was recently giving a talk to some students when one of them asked me a question I hadn’t heard before. She said “How does the Internet of Things really impact the government? What agency benefits the most from IoT/CPS?” I paused for a moment, it is a great…

Cyber Physical Systems (CPS) And BYOD Security

Cyber Physical Systems (CPS) And BYOD Security

Cyber Physical Systems (CPS)  Technology has changed the world radically. But, to date, the world hasn’t changed as fast as technology to. There are a couple of concepts that are concerning as we head into the reality of CPS-deployed systems. The first is that standards don’t exist. Many companies are considering the Bring Your Own…

5 Reasons Why Your Startup Will Grow Faster In The Cloud

5 Reasons Why Your Startup Will Grow Faster In The Cloud

Cloud Startup Fast-tracking Start-ups face many challenges, the biggest of which is usually managing growth. A start-up that does not grow is at constant risk of failure, whereas a new business that grows faster than expected may be hindered by operational constraints, such as a lack of staff, workspace and networks. It is an unfortunate…

CloudTweaks is recognized as one of the leading influencers in cloud computing, infosec, big data and the internet of things (IoT) information. Our goal is to continue to build our growing information portal by providing the best in-depth articles, interviews, event listings, whitepapers, infographics and much more.

Advertising