A Hitchhikers Guide To The Cloud – Database Challenges To Consider

A Hitchhikers Guide To The Cloud – Database Challenges To Consider

A Hitchhikers Guide To The Cloud 

The cloud has turned traditional data management on its head. People still think they will be able to command endless resources while being able to deploy, run and consume a distributed data solution anywhere, anytime. But there are key database challenges when putting your applications in the cloud that are unique and can’t be ignored.

1. High Availability



In the cloud, high availability isn’t just about hardware resiliency anymore. Because customers are removed from the actual hardware, you can no longer plug in an extra power supply, network card or swap hard drives if anything goes wrong or if you need additional resources. The cloud is not predictable and needs to be closely monitored and managed. Besides the large-scale cloud outages we all know about (such as the Amazon-EC2 US-East meltdown recently), we have to face the facts: server crashes, hardware malfunctions and other slipups are all part of the territory in the cloud. With that in mind – cloud users need to be prepared. There are hardly any SLAs for databases in the cloud. To prepare and run effectively on the dynamic cloud environment, every database, regardless of its size, must run in a replicable set-up, which is typically more complex and expensive. Maintaining high availability depends on the availability of “more of the same” resources and the ability to dynamically provision them on the fly once a failure is identified.

Often, when speaking about high availability of the database, the most obvious component is the availability of the data itself, via replication. Yet, if we remember that high availability means that the application continues to communicate with the database as usual, then the plot thickens.

Any solution to ensure availability of the data storage layer must address the availability of the front-end layer of accessing the data so that the connection between the application and any operational DB replicas is not compromised.

On the data storage level, you’ll need to replicate the data, ensure data consistency and synchronization across replicas. Then you’ll need to set up an auto-failover mechanism to monitor the cloud for any failures, identify when a replica is unresponsive and continue service from the remaining surviving replicas.

This then creates the issue of how to manage high-availability of the DB connection. Procedures to ensure high availability, as well as numerous scaling considerations – in particular scale-out – result in several copies of the database that can all be accessed via different addresses. If a replica crashes, you need to monitor which hostname is no longer available and then stop directing traffic from the application to this database.

How do you manage high availability of your database connection string, in a dynamic environment prone to failures in addition to being prone to scaling-out to additional servers (and additional connection addresses) to accommodate bursts in demand?

One way to ensure high availability of the database connection would be to provide users with the multiple front-end addresses and ports and let them handle it on the application level. Users will have to manage connection failovers, load balancing, etc. Needless to say, this will be a major headache as the developer would need to manage the DB connections 24/7 by connecting the app with an available database front-end (very similar to phone operators in the early days).

Another way is to embed high availability into a driver. The driver will be supplied with the initial front-end details and would be updated automatically with any additional front-ends. The driver will balance the connections between front-ends, handle failover automatically, and move connections seamlessly in scale-in and scale-out cases without closing the connection on the user side.

The third alternative is a balancer component on servers, which is installed with each database instance. This component takes care of balancing and availability between all the front-end nodes and seamlessly moves connections when scaling in or out.

2. Management

Database management systems are not only complex systems, but they are also key components in the operation of most software stacks. Given its criticality and complexity, operating a database can be a daunting task that requires significant expertise and considerable resources that are not always readily available to everyone.

Maintaining high availability requires continuously monitoring the cloud environment for any failures, configuring auto-failover mechanisms and keeping multiple copies of the database tier always synchronized and ready to spring into action. Ensuring elasticity means you need to monitor and re-configure and deploy your servers (and sometimes change your app) to add additional resources or to remove them if they go underutilized.

Developers flock to the cloud, and with good reason. The flip side is that once the application gains momentum, it requires a skill set not readily available for most developers. To allow developers to focus on their code rather than on the IT, the cloud ecosystem provides a myriad of off-the-shelf development platforms and cloud services to integrate with to streamline development and time-to-production.

3. Scalability

Scalability and elasticity are the trendiest words in the database arena these days – everybody scales, and everybody claims that only they scale the right way.

Scaling an application (by adding additional servers and load balancers) is pretty much a no-brainer, and many cloud providers offer that. Some, like Amazon EC2, even offer the automatic addition of servers to scale an application once CPU usage is high.

While scaling an application is pretty straightforward, scaling the database tier is more difficult, particularly when scaling out by adding nodes. Scaling a database in general is no trivial task because of its “statefull” nature (unlike the cloud’s stateless environment), and in the cloud, it is even more difficult.

Cloud applications are often characterized with fluctuating demand (spiking at any moment). Databases need to be able to instantly and automatically scale both in throughput and size to accommodate increasing demand from the application.

When evaluating a database solution, ask yourself how it scales and see if it scales in a way that would be optimal for the needs of your application.

4. Elasticity

The cloud is all about flexibility in resources – allowing you to add/remove resources to match your needs, with no need for over-provisioning or over-paying to prepare for any future peaks. Elasticity isn’t just about increasing resources when you need to by scaling up or out, but also shrinking those back down when your database is underutilized, to save on costs. Elasticity needs to be supported to accommodate very granular increases in resources so that to gain +0.X more power doesn’t mean you need to commit any pay for a much larger (+XXX) machine.

Understanding these key challenges can help to successfully deploy, run and consume a database in the cloud. Developers who are aware of the dynamic nature of clouds will take extra care protecting their assets and will be able maintain a successful application that is usable, satisfies users and produces revenue.

By Razi Sharir,

Razi Sharir, CEO of Xeround, has more than 20 years of management experience in product/solution development. Prior to Xeround, Razi has led the strategic transition from traditional data centers to cloud computing at BMC Software and the Incubator/Innovation Lab business unit.

About CloudTweaks

Established in 2009, CloudTweaks is recognized as one of the leading authorities in connected technology information and services.

We embrace and instill thought leadership insights, relevant and timely news related stories, unbiased benchmark reporting as well as offer green/cleantech learning and consultive services around the world.

Our vision is to create awareness and to help find innovative ways to connect our planet in a positive eco-friendly manner.

In the meantime, you may connect with CloudTweaks by following and sharing our resources.

View All Articles

One Response to A Hitchhikers Guide To The Cloud – Database Challenges To Consider

  1. There is a lot of talk about how cloud computing is evolving
    in develop nations, and how companies in most cases startups are using this new
    technology to their advantage. But what is often not said is it takes a lot of
    knowledge and expertise to implement an IaaS solution and truly get its
    benefits, thats why you can clearly see that the bast majority of clients of
    popular public clouds like AWS and Rackspace are silicon valley startups and
    organizations with deep technical knowledge. The business model of those public
    clouds is designed for the DIY approach of the technology startups , but where
    does this leave the SME organizations that doesn’t know how to take advantage
    of all the benefits of cloud computing. That why we need another breed of
    public cloud provider, that doesn’t seek mass adoption of its platform and full
    automation, but instead looks to help and be a partners with clients to easy
    their migration to this new world of IT. This is more relevant in Latin
    American countries where the average organization does not have a clue of how
    to move to the cloud. In Colombia there are several companies that are  appering just to do that , close that knowledge
    gap that exist beetween campanies in the USA and companies in Latin America so
    that so that we can also benefit from cloud computing , here is an example of a company that is doing that: http://www.kumo.com.co/

Comic
When Sci-Fi Predictions Come To Fruition

When Sci-Fi Predictions Come To Fruition

Evolution of Technologies To paraphrase science fiction author Arthur C. Clark, those who make predictions about the future are either “considered conservative now and mocked later, or mocked now and proved right when they are no longer around to enjoy the acclaim.” The one thing we can be sure about, Clark ventured, is that “[the…

Facebook Hopes To Extend Internet Connectivity With Solar-Powered Drones

Facebook Hopes To Extend Internet Connectivity With Solar-Powered Drones

Facebook Inc (FB.O) said on Thursday it had completed a successful test flight of a solar-powered drone that it hopes will help it extend internet connectivity to every corner of the planet. Aquila, Facebook’s lightweight, high-altitude aircraft, flew at a few thousand feet for 96 minutes in Yuma, Arizona, Chief Executive Mark Zuckerberg wrote in…

When Will Women In Tech Become The Norm?

When Will Women In Tech Become The Norm?

Tech Diversity It is well known that the technology industry has been dominated by men, but it is also clear that the industry is working to change that. Diversity in the tech industry, especially where it applies to women in tech, has been a topic of discussion for years. Recently the Washington Technology Industry Association…

Four Keys For Telecoms Competing In A Digital World

Four Keys For Telecoms Competing In A Digital World

Competing in a Digital World Telecoms, otherwise largely known as Communications Service Providers (CSPs), have traditionally made the lion’s share of their revenue from providing pipes and infrastructure. Now CSPs face increased competition, not so much from each other, but with digital service providers (DSPs) like Netflix, Google, Amazon, Facebook, and Apple, all of whom…

Edtech and Virtual Reality – Exciting Learning Environment

Edtech and Virtual Reality – Exciting Learning Environment

Customizing Edutech Customized edtech learning solutions are becoming more commonplace as the education industry recognises their potential and begins transforming the traditional structures so as to incorporate innovative developments. From textbooks to tablets, chalkboards to virtual reality, edtech promises not only dynamic and exciting learning environments but better learning strategies and solutions. Virtual Reality and…

How Formal Verification Can Thwart Change-Induced Network Outages and Breaches

How Formal Verification Can Thwart Change-Induced Network Outages and Breaches

How Formal Verification Can Thwart  Breaches Formal verification is not a new concept. In a nutshell, the process uses sophisticated math to prove or disprove whether a system achieves its desired functional specifications. It is employed by organizations that build products that absolutely cannot fail. One of the reasons NASA rovers are still roaming Mars…

Maintaining Network Performance And Security In Hybrid Cloud Environments

Maintaining Network Performance And Security In Hybrid Cloud Environments

Hybrid Cloud Environments After several years of steady cloud adoption in the enterprise, an interesting trend has emerged: More companies are retaining their existing, on-premise IT infrastructures while also embracing the latest cloud technologies. In fact, IDC predicts markets for such hybrid cloud environments will grow from the over $25 billion global market we saw…

Four Keys For Telecoms Competing In A Digital World

Four Keys For Telecoms Competing In A Digital World

Competing in a Digital World Telecoms, otherwise largely known as Communications Service Providers (CSPs), have traditionally made the lion’s share of their revenue from providing pipes and infrastructure. Now CSPs face increased competition, not so much from each other, but with digital service providers (DSPs) like Netflix, Google, Amazon, Facebook, and Apple, all of whom…

The Fully Aware, Hybrid-Cloud Approach

The Fully Aware, Hybrid-Cloud Approach

Hybrid-Cloud Approach For over 20 years, organizations have been attempting to secure their networks and protect their data. However, have any of their efforts really improved security? Today we hear journalists and industry experts talk about the erosion of the perimeter. Some say it’s squishy, others say it’s spongy, and yet another claims it crunchy.…

Adopting A Cohesive GRC Mindset For Cloud Security

Adopting A Cohesive GRC Mindset For Cloud Security

Cloud Security Mindset Businesses are becoming wise to the compelling benefits of cloud computing. When adopting cloud, they need a high level of confidence in how it will be risk-managed and controlled, to preserve the security of their information and integrity of their operations. Cloud implementation is sometimes built up over time in a business,…

Cloud Infographic – Disaster Recovery

Cloud Infographic – Disaster Recovery

Disaster Recovery Business downtime can be detrimental without a proper disaster recovery plan in place. Only 6% of businesses that experience downtime without a plan will survive long term. Less than half of all businesses that experience a disaster are likely to reopen their doors. There are many causes of data loss and downtime —…

Cloud Computing Myths That SMBs Should Know

Cloud Computing Myths That SMBs Should Know

Cloud Computing and SMBs Cloud Computing is the hottest issue among IT intellects of Small and Medium Businesses (SMBs). Like any other computer-orientated technology, Cloud Computing has some misconceptions and myths that often kick-start arguments among the two opposing groups: Cloud Supporters and Cloud Opponents. Both of these groups have their own ideology and reasons…

Cloud Infographic – What Is The Internet of Things?

Cloud Infographic – What Is The Internet of Things?

What Is The Internet of Things? “We’re still in the first minutes of the first day of the Internet revolution.”  – Scott Cook The Internet of Things (IOT) and Smart Systems are based on the notions of Sensors, Connectivity, People and Processes. We are creating a new world to view and measure anything around us through…

Containerization: The Bold Face Of The Cloud In 2016

Containerization: The Bold Face Of The Cloud In 2016

Containerization And The Cloud “Right now, the biggest technology shift in the cloud is a rapid evolution from simple virtual machine (VM) hosting toward containerization’’ says the CTO of Microsoft Azure, Mark Russinovitch, a man who deals with the evolving cloud infrastructure every day. In his words, containerization is “an incredibly efficient, portable, and lightweight…

12 Promising Business Intelligence (BI) Services For Your Company

12 Promising Business Intelligence (BI) Services For Your Company

12 Promising Business Intelligence (BI) Services Business Intelligence (BI) services have recently seen an explosion of innovation and choices for business owners and entrepreneurs. So many choices, in fact, that many companies aren’t sure which business intelligence company to use. To help offer you a solution, we’ve compiled a list of 12 Business Intelligence companies…

Why Small Businesses Need A Business Intelligence Dashboard

Why Small Businesses Need A Business Intelligence Dashboard

The Business Intelligence Dashboard As a small business owner you would certainly know the importance of collecting and analyzing data pertaining to your business and transactions. Business Intelligence dashboards allow not only experts but you also to access information generated by analysis of data through a convenient display. Anyone in the company can have access…