Storing Massive Files On The Cloud

Storing Massive Files On The Cloud

On December 9th, Amazon announced, in a blog post, an increase in the maximum size of file that could be stored on the Amazon Simple Storage Service (S3). Before the increase, the largest file that could be stored was 5GB – anything larger had to be split into chunks and put back together within an application, using an intermediate server or on the client device. Now, the limit has been increased to 5TB, an increase of approximately 1000 times.

This new limit allows the storage of large databases, video files or scientific data, but to make use of it properly, the object needs to be uploaded using Amazon’s new Multipart Upload API. The original objects are split into smaller parts and then uploaded separately before being stitched together on the Amazon S3 cloud. Objects can then be worked on either as a whole, or in part, on the Amazon Elastic Compute Cloud (EC2) before storing the finished product back on S3.

Data stored on Amazon S3 is designed to have 99.99% annual availability and to withstand the concurrent failure of data in two facilities, so you can be certain your data would be available at all times. For less critical data, Amazon’s Reduced Redundancy Storage (RRS) offers a cheaper way of storing your non-critical data, by utilizing less storage facilities. There is an associated loss of data durability, but the quoted availability is still 99.99%.

OpenStack have also announced that they are working on increasing the size of files that can be stored on its open-source private cloud system, and it is expected that a 5TB file size limit will be available with the “Bexar” release of OpenStack early in 2011.

A file size of 5TB is a massive amount of data, but it is clear that the cloud storage limits are going to increase even further in the near future. It is only a matter of time before the maximum object size is in the petabyte or even exabyte range.

The only question then is exactly how you can generate enough data to need that amount of storage! Amazon Web Services Blog

By Glenn Blake

About Glenn Blake

Glenn Blake is a writer for CloudTweaks and has been writing about technology trends for over 25 years.

Find out more
View All Articles

Sorry, comments are closed for this post.

Comic
Virtual Immersion And The Extension/Expansion Of Virtual Reality

Virtual Immersion And The Extension/Expansion Of Virtual Reality

Virtual Immersion And Virtual Reality This is a term I created (Virtual Immersion). Ah…the sweet smell of Virtual Immersion Success! Virtual Immersion© (VI) an extension/expansion of Virtual Reality to include the senses beyond visual and auditory. Years ago there was a television commercial for a bathing product called Calgon. The tagline of the commercial was Calgon…

Your Biggest Data Security Threat Could Be….

Your Biggest Data Security Threat Could Be….

Paying Attention To Data Security Your biggest data security threat could be sitting next to you… Data security is a big concern for businesses. The repercussions of a data security breach ranges from embarrassment, to costly lawsuits and clean-up jobs – particularly when confidential client information is involved. But although more and more businesses are…

Protecting Devices From Data Breach: Identity of Things (IDoT)

Protecting Devices From Data Breach: Identity of Things (IDoT)

How to Identify and Authenticate in the Expanding IoT Ecosystem It is a necessity to protect IoT devices and their associated data. As the IoT ecosystem continues to expand, the need to create an identity to newly-connected things is becoming increasingly crucial. These ‘things’ can include anything from basic sensors and gateways to industrial controls…

Cloud Services Providers – Learning To Keep The Lights On

Cloud Services Providers – Learning To Keep The Lights On

The True Meaning of Availability What is real availability? In our line of work, cloud service providers approach availability from the inside out. And in many cases, some never make it past their own front door given how challenging it is to keep the lights on at home let alone factors that are out of…

How You Can Improve Customer Experience With Fast Data Analytics

How You Can Improve Customer Experience With Fast Data Analytics

Fast Data Analytics In today’s constantly connected world, customers expect more than ever before from the companies they do business with. With the emergence of big data, businesses have been able to better meet and exceed customer expectations thanks to analytics and data science. However, the role of data in your business’ success doesn’t end…

Four Recurring Revenue Imperatives

Four Recurring Revenue Imperatives

Revenue Imperatives “Follow the money” is always a good piece of advice, but in today’s recurring revenue-driven market, “follow the customer” may be more powerful. Two recurring revenue imperatives highlight the importance of responding to, and cherishing customer interactions. Technology and competitive advantage influence the final two. If you’re part of the movement towards recurring…

Staying on Top of Your Infrastructure-as-a-Service Security Responsibilities

Staying on Top of Your Infrastructure-as-a-Service Security Responsibilities

Infrastructure-as-a-Service Security It’s no secret many organizations rely on popular cloud providers like Amazon and Microsoft for access to computing infrastructure. The many perks of cloud services, such as the ability to quickly scale resources without the upfront cost of buying physical servers, have helped build a multibillion-dollar cloud industry that continues to grow each…

What Top SaaS Vendors Do To Ensure Successful Onboarding

What Top SaaS Vendors Do To Ensure Successful Onboarding

What Top SaaS Vendors Do I am not going to mention names in this article, but if you want to be the best, you must look at what the best do – and do it better. The importance of investing in SaaS onboarding can be easily overlooked in favor of designing efficient and powerful software…

Fintech Investments Are Seeing Consistent Growth

Fintech Investments Are Seeing Consistent Growth

The Financial Services Cloud Fintech investment has been seeing consistent growth in 2015, with some large moves being made this year. The infographic (Courtesy of Venturescanner) below shows the top Fintech investors and the amount of companies they’re currently funding: Just this week, a financial data startup known as Orchard Platform raised $30 million in…

The Storytelling Machine: Big Content and Big Data

The Storytelling Machine: Big Content and Big Data

Bridging The Gap Between Big Content and Big Data Advances in cloud computing, along with the big data movement, have transformed the business IT landscape. Leveraging the cloud, companies are now afforded on demand capacity and mobile accessibility to their business-critical systems and information. At the same time, the amount of structured and unstructured data…

Containerization: The Bold Face Of The Cloud In 2016

Containerization: The Bold Face Of The Cloud In 2016

Containerization And The Cloud “Right now, the biggest technology shift in the cloud is a rapid evolution from simple virtual machine (VM) hosting toward containerization’’ says the CTO of Microsoft Azure, Mark Russinovitch, a man who deals with the evolving cloud infrastructure every day. In his words, containerization is “an incredibly efficient, portable, and lightweight…

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

The Future Of Work: What Cloud Technology Has Allowed Us To Do Better

What Cloud Technology Has Allowed Us to Do Better The cloud has made our working lives easier, with everything from virtually unlimited email storage to access-from-anywhere enterprise resource planning (ERP) systems. It’s no wonder the 2013 cloud computing research IDG survey revealed at least 84 percent of the companies surveyed run at least one cloud-based application.…

Four Reasons Why CIOs Must Transform IT Into ITaaS To Survive

Four Reasons Why CIOs Must Transform IT Into ITaaS To Survive

CIOs Must Transform IT The emergence of the Cloud and its three delivery models of Infrastructure as a Service (IaaS), Software as a Service (SaaS) and Platform as a Service (PaaS) has dramatically impacted and forever changed the delivery of IT services. Cloud services have pierced the veil of IT by challenging traditional method’s dominance…

Expert Insights Into The Yahoo Breach

Expert Insights Into The Yahoo Breach

Yahoo Breach Latest reports suggest that the recent Yahoo! data breach may exceed 500 million records, with some sources implying millions more records penetrated, upping the total number of records stolen in various recent hacks to approximately 3.5 billion. CloudTweaks spoke to Kevin O’Brien, CEO of GreatHorn, for expert insight into this latest violation. GreatHorn…

Using Private Cloud Architecture For Multi-Tier Applications

Using Private Cloud Architecture For Multi-Tier Applications

Cloud Architecture These days, Multi-Tier Applications are the norm. From SharePoint’s front-end/back-end configuration, to LAMP-based websites using multiple servers to handle different functions, a multitude of apps require public and private-facing components to work in tandem. Placing these apps in entirely public-facing platforms and networks simplifies the process, but at the cost of security vulnerabilities. Locating everything…

Big Data’s Significant Role In Fintech

Big Data’s Significant Role In Fintech

Data Banking Fintech covers a range of financial fields such as retail banking, investments, and lending and thanks to the mobile and internet innovations of late is a thriving sector. Offering improvements which drive customer satisfaction and education in an area previously inscrutable and dictated by gigantic inflexible corporations, fintech is helping put the power…

Battle of the Clouds: Multi-Instance vs. Multi-Tenant

Battle of the Clouds: Multi-Instance vs. Multi-Tenant

Multi-Instance vs. Multi-Tenant The cloud is part of everything we do. It’s always there backing up our data, pictures, and videos. To many, the cloud is considered to be a newer technology. However, cloud services actually got their start in the late 90s when large companies used it as a way to centralize computing, storage,…

Infographic: 9 Things To Know About Business Intelligence (BI) Software

Infographic: 9 Things To Know About Business Intelligence (BI) Software

Business Intelligence (BI) Software  How does your company track its data? It’s a valuable resource—so much so that it’s known as Business Intelligence, or BI. But using it, integrating it into your daily processes, that can be significantly difficult. That’s why there’s software to help. But when it comes to software, there are lots of…