The next BriefingsDirect strategic storage and business continuity case study discussion explores how Norway’s venerable meat processing business, Fatland, relied on rapid backup and recovery solutions to successfully defended against a nasty ransomware attack.
The comprehensive backup and recovery stack allowed Fatland’s production processing systems to snap back to use after only a few hours, but the value of intelligent and increasingly hybrid storage approaches go much further to assure the ongoing integrity of both systems — and business outcomes.
Here to explain how vertically integrated IT infrastructure and mirrored data strategies can prevent data loss and business downtime are Terje Wester, the CEO at Fatland, based in Norway, and Patrick Osborne, Vice President and General Manager of Big Data and Secondary Storage at Hewlett Packard Enterprise (HPE). The discussion is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions.
Here are some excerpts:
Gardner: Terje, getting all of your systems back up in a few hours after an aggressive ransomware attack in 2017 probably wasn’t what first drove you to have a comprehensive backup and recovery capability. What were the early drivers that led you to put in a more modern approach to data lifecycle management?
Wester: First of all, we have HPE end-to-end at Fatland. We have four production sites. At one production site we have our servers. We are running a meat business, doing everything from slaughtering to processing and packing. We deal with the farmers; we deal with the end customers. It’s really important to have good IT systems, also safe systems.
When we last invested in these HPE systems, we wanted something that was in front of the line, which was safe, because the uptime in the company is so important. Our IT people had the freedom to choose what they thought was the best solution for us. And HPE was the answer. We tested that really hard on this ransomware episode we had in September.
Gardner: Patrick, are you finding in the marketplace that people have primary reasons for getting into a comprehensive data protection mode? It can become a gift that keeps giving.
Osborne: A lot of our customers are now focusing on security. It’s definitely top of mind. What we are trying to provide is more of an integrated approach, so it’s not a secondary or an afterthought that you bolt on.
Whether it’s our server products, with silicon root of trust, or our storage products, with things like we have done for Fatland such as Recovery Manager Central (RMC), or with our integrated offerings such as our hyper-converged infrastructure (HCI) product line — the theme is the same. What we are trying to weave through this is that data protection and availability are an endemic piece of the architecture. You get it on day one when you move to a modernized architecture, as opposed to running into a ransomware or an availability issue and then having to re-architect after-the-fact.
What we are trying to do with a number of customers is, from day one, when you renew your infrastructure, it has all of this availability and security built in. That’s one of the biggest things that we see, that’s helpful for customers these days.
Gardner: Data and security integration are, in fact, part of the architecture. Security is not a separate category or a chunk that you bolt on later.
Gardner: Terje, tell us a about the .NM4 crypto virus. In 2017, this hit a lot of people. Some were out for days. What happened when this hit your organization?
Rapid response, recovery
Wester: These people were trying to attack us. They started to visit our servers and got in on a Thursday. They worked until that Friday night and found an opening. This was something that happened in the middle of the night and they closed down the servers. They put in this ransomware, so that closed down everything.
On Saturday, we had no production. So, Saturday and Sunday for us were the days to work on and solve the problem. We contacted HPE for consultants, to determine what to do. They came over from Oslo on Sunday, and from Sunday afternoon to early Monday morning we recovered everything.
On Monday morning we started up, I think, only about 30 minutes behind schedule and the business was running. That was extremely important for us. We have live animals coming in on Sunday to be slaughtered on Monday. We have rapid processing. Christmas was around the corner and everything that we produce is important every day. The quick recovery was really important for us.
Gardner: You are an older, family-run organization, dating back to 1892. So, you have a very strong brand to protect.
Wester: That’s right, yes.
Gardner: You don’t want to erode that brand. People want to continue to hold the trust they have had in you for 125 years.
Wester: They do. The farmers have been calling us for slaughtering of their cattle for generations. We have the typical supermarket chains in Norway as our main customers. We have a big daily turnover, especially in September through October, when all the lambs are coming in. It’s just a busy period and everybody trusts that we should work for them every day, and that’s our goal, too.
Gardner: Patrick, what was it about the HPE approach, the Recovery Manager Central and StoreOnce, that prevented the ransomware attack, in this case, from causing the significant downtime that we saw in other organizations?
Osborne: One of the important things to focus on is that in the case of Fatland it’s not so much the money that you would have had to pay for the ransomware, it’s the downtime. That is key.
Using our architecture, you can take application or data-specific point-in-time copies of the data that’s critical — either mission-critical or business-critical — at a very granular level. You can orchestrate that, and then send that all off to a secondary system. That way you have an additional layer of security.
What we announced in November 2017 at Discover in Madrid is the ability to go even further beyond that and send an additional copy to the cloud. At all layers of the infrastructure, you will be able to encrypt that data. We designed the system around not so much backup — but to be able to restore quickly.
The goal is to provide a very aggressive recovery time objective (RTO) in a very granular recovery point objective. So, when a team like Terje’s at Fatland recognizes that they have a breach, you can mitigate that, essentially staunch the issue, and be able to rapidly recover from a well-known set of data that wasn’t compromised.
For us it’s all about architecting to rapidly recover, of making that RTO as quickly as possible. And we see a lot of older architectures where you have a primary storage solution that has all of your data on it and then not a really good backup infrastructure.
What turned into two days of disruption for Fatland could have been many more days, if not weeks, in older infrastructure. We really just are focused on mitigation of RTO.
Gardner: In the case of the cryptovirus, did the virus not encrypt the data at all, or was it encrypted but you were able to snap back to the encryption-free copies of the data fast?
Osborne: When we do this at the storage layer, we are able to take copies of that data and then move it off to a secondary system, or even a tertiary system. You then have a well-known copy of that data before it’s been encrypted. You are able to roll back to a point in time in your infrastructure before that data has been compromised, and then we can actually go a step further.
Some of the techniques allow you to have encryption on your primary storage. That usually helps if you are changing disk drives and whatnot. It’s from a security perspective. Then we are actually able to encrypt again at the data level on secondary storage. In that case, you have a secure piece of the infrastructure with data that’s already been encrypted at a well-known point in time, and you are able to recover. That really helps out a lot.
Gardner: So, their encryption couldn’t get past your encryption?
Gardner: The other nice thing about this rapid recovery approach is that it doesn’t have to be a ransomware or a virus or even a security issue. It could be a natural disaster; it could be some human error. What’s important is the business continuity.
Now that you have been through the ransomware attack, how is your confidence in always being up and running and staying in business in general, Terje?
Business continuity bonus
Wester: We had been discussing this quite a lot before this ransomware issue. We established better backup systems, but now we are looking into extending them even more, to have another system that can run from the minute the main servers are down. We have a robotized system picking out meat for the supermarket chains 24×7, and when their main server stops, something should be able to take over and run the business. So, within a very short time we will also have that solution in place, with good help from HPE.
Gardner: Patrick, not that long ago the technology to do this may have been there, but the costs were prohibitive. The network and latency and issues were prohibitive. What’s happened in the past several years that allows you to go to a company such as Fatland and basically get them close to 99.9999 percent availability across the board?
Osborne: In the past, you had customers with a preferred vendor for servers, a preferred vendor for networking, and another preferred vendor for storage. That azimuth is changing to a vertically oriented stack. So, when Terje has a set of applications or business needs, we are able to, as a portfolio company, bring together that whole stack.
In the past, the customer was the integrator, and the cost was in bringing many, many different disparate solutions together. They would act as the integrator. That was probably the largest cost back in the day.
Now, we’re bringing together something that’s more vertically oriented and that has security and data protection availability throughout the stack. We’re making these techniques and levels of availability for customers of any size, where IT is not really their core competency. At the end of day, it’s a business enabler, right?
Wester: Right, absolutely.
Osborne: The second piece from a networking perspective is that very large and low-cost bandwidth has definitely changed the game in terms of being able to move data, replicate data from on-premise, even off-premise to the cloud, that’s certainly been an enabler as well.
Gardner: We are seeing mirroring of entire data centers in amazing amounts of time.
Also, you have an integrated stack approach, with HPE focused on security engineered in, across the board, from the silicon up. What are some of the newer technologies that we can expect to see that further increases higher availability, lower risk and lower cost?
Shared signature knowledge
Osborne: Terje’s team had cryptovirus on-premise, a breach with a number of different signatures. We are now focusing on artificial intelligence (AI) for the data center. So, taking the human factor out of it to help recognize the problems faster.
So, if they have a breach, and that has certain signatures found in the infrastructure, we can take that and apply that knowledge to other customers. And likewise, they may have some things that happened to them that can benefit Fatland as well.
Using machine learning techniques, we have a number of things that we have brought to the table for what we call predictive analytics in the data center. So HPE Aruba on the networking side has a number of capabilities, too.
We are bringing InfoSight, which is our predictive analytics for storage, and extending that to other parts of the infrastructure. So, servers, networking, and storage. You can start to see signatures in more places.
The General Data Protection Regulation (GDPR) has been implemented, and there are some high fines. You have to report within 72 hours. So, anything you can do to take the human factor out of this, from a technology perspective is a win for everyone, and we have a big investment in that.
Gardner: And that gets back to the idea that strategic data protection is the gift that keeps giving. As more systems are integrated, the more data analysis can be done, signatures patterns shared with other organizations, and you can ultimately become predictive rather than reactive.
Terje, the level of confidence that you have seems to be high, it’s perhaps going to get higher. What other recommendations might you have for other organizations that are thinking about this? Did it turn out to be a good investment, and what sort of precautions might you have for others if they haven’t done this already?
Communication is key
Wester: Data itself is not part of our core business. But communication is. It is extremely important for us to communicate internally and externally all the time.
In every organization, IT people need to talk to the management and the board about these safety issues. I think that should be brought to the table before these problems come up.
We have good systems, HPE end-to-end. Of course, one thing that is important is to have modern technology in place, so we could have a quick recovery, and that was a good thing.
Most important for us was that the IT management had the trust from us — the management and the board — to invest in what they thought was the best solution. We still saw some operational breaches and we need to do better. This is a big focus with us. Every organization should invest time to look into the infrastructure to see what to do to make it safer for quick recovery, which is important for any company. Bring it on to the table for the board, for the management, for a really good discussion — it’s worth that.
By Dana Gardner