Storing Massive Files On The Cloud

Storing Massive Files On The Cloud

On December 9th, Amazon announced, in a blog post, an increase in the maximum size of file that could be stored on the Amazon Simple Storage Service (S3). Before the increase, the largest file that could be stored was 5GB – anything larger had to be split into chunks and put back together within an application, using an intermediate server or on the client device. Now, the limit has been increased to 5TB, an increase of approximately 1000 times.

This new limit allows the storage of large databases, video files or scientific data, but to make use of it properly, the object needs to be uploaded using Amazon’s new Multipart Upload API. The original objects are split into smaller parts and then uploaded separately before being stitched together on the Amazon S3 cloud. Objects can then be worked on either as a whole, or in part, on the Amazon Elastic Compute Cloud (EC2) before storing the finished product back on S3.

Data stored on Amazon S3 is designed to have 99.99% annual availability and to withstand the concurrent failure of data in two facilities, so you can be certain your data would be available at all times. For less critical data, Amazon’s Reduced Redundancy Storage (RRS) offers a cheaper way of storing your non-critical data, by utilizing less storage facilities. There is an associated loss of data durability, but the quoted availability is still 99.99%.

OpenStack have also announced that they are working on increasing the size of files that can be stored on its open-source private cloud system, and it is expected that a 5TB file size limit will be available with the “Bexar” release of OpenStack early in 2011.

A file size of 5TB is a massive amount of data, but it is clear that the cloud storage limits are going to increase even further in the near future. It is only a matter of time before the maximum object size is in the petabyte or even exabyte range.

The only question then is exactly how you can generate enough data to need that amount of storage! Amazon Web Services Blog

By Glenn Blake

About Glenn Blake

Glenn Blake is a writer for CloudTweaks and has been writing about technology trends for over 25 years.

Find out more
View All Articles

Sorry, comments are closed for this post.

Comic
The Annual Compliance & Ethics Institute:  Hot Topics – Cyber Security, Big Data, Privacy Breach Response

The Annual Compliance & Ethics Institute: Hot Topics – Cyber Security, Big Data, Privacy Breach Response

Cyber Security, Big Data, Privacy Breach Response MINNEAPOLIS, Aug. 30, 2016 /PRNewswire-USNewswire/ — Cyber security, social media, modern slavery, anti-corruption, export controls and sanctions, and privacy top the list of “Hot Issues” compliance and ethics professionals face each day. These and many other compliance and ethics concerns will be addressed at the 2016 Compliance & Ethics…

Top 5 Digital Health Trends

Top 5 Digital Health Trends

Digital Health Trends It is very important to keep up with the changing technology. However, it is also just as important to advance the consumer experience, care delivery methods and create opportunities for career development for the healthcare workforce. Five trends that are proven to be effective in winning in the digital age have been…

Technological Advances In The Healthcare Industry

Technological Advances In The Healthcare Industry

The Healthcare Industry The use of smart devices in healthcare is expanding, and according to a report by Technavio the global smart wearable healthcare device and services market will show a compounded annual growth rate of over 18%. Thanks to modern medicine, lifespans are increasing, resulting in aging populations and a higher frequency of chronic…

How Secure Is Your School Campus Network?

How Secure Is Your School Campus Network?

School Networks School related networks are one of the most attacked sectors today, coming in third worldwide to healthcare and retail. Because of the ever growing threat of cybercrime, IT professionals everywhere aren’t thinking in terms of “what if our network gets attacked?” Now, they think in terms of “when will our network be attacked?”…

IBM and VMware Expand Partnership to Enable Easy Hybrid Cloud Adoption

IBM and VMware Expand Partnership to Enable Easy Hybrid Cloud Adoption

IBM and VMware Expand Partnership More than 500 new clients, including Marriott International are now running VMware software on IBM Cloud since the strategic cloud partnership was announced;Introduction of VMware Cloud Foundation on IBM Cloud helps move existing apps to the cloud within hours; More than 4,000 IBM service professionals trained to help organizations extend…

5% Of Companies Have Embraced The Digital Innovation Fostered By Cloud Computing

5% Of Companies Have Embraced The Digital Innovation Fostered By Cloud Computing

Embracing The Cloud We love the stories of big complacent industry leaders having their positions sledge hammered by nimble cloud-based competitors. Saleforce.com chews up Oracle’s CRM business. Airbnb has a bigger market cap than Marriott. Amazon crushes Walmart (and pretty much every other retailer). We say: “How could they have not seen this coming?” But, more…

Which Is Better For Your Company: Cloud-Based or On-Premise ERP Deployment?

Which Is Better For Your Company: Cloud-Based or On-Premise ERP Deployment?

Cloud-Based or On-Premise ERP Deployment? You know how enterprise resource management (ERP) can improve processes within your supply chain, and the things to keep in mind when implementing an ERP system. But do you know if cloud-based or on-premise ERP deployment is better for your company or industry? While cloud computing is becoming more and…

Why Security Practitioners Need To Apply The 80-20 Rules To Data Security

Why Security Practitioners Need To Apply The 80-20 Rules To Data Security

The 80-20 Rule For Security Practitioners  Everyday we learn about yet another egregious data security breach, exposure of customer data or misuse of data. It begs the question why in this 21st century, as a security industry we cannot seem to secure our most valuable data assets when technology has surpassed our expectations in other regards.…

Protecting Devices From Data Breach: Identity of Things (IDoT)

Protecting Devices From Data Breach: Identity of Things (IDoT)

How to Identify and Authenticate in the Expanding IoT Ecosystem It is a necessity to protect IoT devices and their associated data. As the IoT ecosystem continues to expand, the need to create an identity to newly-connected things is becoming increasingly crucial. These ‘things’ can include anything from basic sensors and gateways to industrial controls…

Ending The Great Enterprise Disconnect

Ending The Great Enterprise Disconnect

Five Requirements for Supporting a Connected Workforce It used to be that enterprises dictated how workers spent their day: stuck in a cubicle, tied to an enterprise-mandated computer, an enterprise-mandated desk phone with mysterious buttons, and perhaps an enterprise-mandated mobile phone if they traveled. All that is history. Today, a modern workforce is dictating how…

4 Different Types of Attacks – Understanding the “Insider Threat”

4 Different Types of Attacks – Understanding the “Insider Threat”

Understanding the “Insider Threat”  The revelations that last month’s Sony hack was likely caused by a disgruntled former employee have put a renewed spotlight on the insider threat. The insider threat first received attention after Edward Snowden began to release all sorts of confidential information regarding national security. While many called him a hero, what…

The Business of Security: Avoiding Risks

The Business of Security: Avoiding Risks

The Business of Security Security is one of those IT concerns that aren’t problematic until disaster strikes. It might be tomorrow, it could be next week or next year. The fact is that poor security leaves businesses wide open for data loss and theft. News outlets just skim the surface, but hackers cost business up…

The Internet of Things – Redefining The Digital World As We Know It

The Internet of Things – Redefining The Digital World As We Know It

Redefining The Digital World According to Internet World Stats (June 30th, 2015), no fewer than 3.2 billion people across the world now use the internet in one way or another. This means an incredible amount of data sharing through the utilization of API’s, Cloud platforms and inevitably the world of connected Things. The Internet of Things is a…

Who’s Who In The Booming World Of Data Science

Who’s Who In The Booming World Of Data Science

The World of Data Science The nature of work and business in today’s super-connected world means that every second of every day, the world produces an astonishing amount of data. Consider some of these statistics; every minute, Facebook users share nearly 2.5 million pieces of content, YouTube users upload over 72 hours of content, Apple…

6 Tech Predictions To Have A Major Impact In 2016

6 Tech Predictions To Have A Major Impact In 2016

6 Tech Predictions To Have A Major Impact The technology industry moves at a relentless pace, making it both exhilarating and unforgiving. For those at the forefront of innovation it is an incredibly exciting place to be, but what trends are we likely to see coming to the fore in 2016? Below are six predictions…