The Cloud as a recovery tool was the subject of a very good meeting today. Although the meeting turned around multiple scenarios in the aftermath of Maria's disaster, it actually was all about thought leadership, contingency plans and risk management measures in emergency situations.
With companies like Cooperativa de Seguros Múltiples; B. Fernández; Coca Cola; Auxilio Mutuo; AT&T and First Bank; we had a great representation of different industries (Banking, Insurance, Distribution, Manufacturing, Telecommunications and Healthcare) to share experiences and ideas on how to continue innovating despite the setbacks mother nature can throw at us.
A common understanding was:
"We had a great plan, we rehearsed the plan with Irma, we executed the plan perfectly but Maria’s wrath was so dramatic that we couldn’t plan for everything, particularly the loss of power and telecommunications across the island”.
The telecommunications back up plan varied among the group, some had single redundancy, some had up to four levels of redundancy with different vendors and even with that, it was not enough.
This is to say, this company had to activate satellite connections. The problem with satellite connection, as explained by this company, ware cost as well as performance, comparing it to modem connectivity back in the 80’s. And here's where the Cloud as a recovery tool gains value.
Reflections on Maria's Disaster: Why using the Cloud as a recovery tool
One relevant comment was that all plans assumed that technology was going to be available and therefore no plans for manual processes, with adequate controls, were prepared.
One big surprise presented by one the participants was that, contrary to what happened in the 90’s where users showed apprehension to the use of technology, in this case they showed great resistance to the use of paper. One user even started crying stating that she couldn’t handle the matter. We guess she missed the spellchecker!
We asked the group about their strategy in the Cloud as recovery tool pre-María's strike. Most of them had a Cloud as a recovery tool plan and a real commitment to the Cloud. Now, it's time to reevaluate those plans, particularly because of their impact in communications.
The group agreed that Cloud is the future of software & technology and somehow, the country and their organizations need to overcome this situation. Basically, there is no option, telecommunication and energy in Puerto Rico must be hurricane proof.
In terms of protection strategies during and after the event, here are some tips that came up during the conversation:
Tip #1 Timing & Backup
When coordinating your closing of operations, allot enough time to perform a last minute back up of all on premise applications and leave sufficient time for your IT staff to reach their homes safely. This will be your #1 advantage when using the Cloud as a recovery tool.
Tip #2 Prevision & Organization
In one of the institutions, the CIO was responsible of all the back-up media for the applications. Name yours beforehand and make sure the person lives in a well-protected home, close to your operations. After the hurricane, it was very difficult to move around as trees and floods made most roads impassable.
Tip #3 Power Backup Plan
If your company doesn’t have customers outside of Puerto Rico that requires connections to your servers during the event, you should shut down all your equipment. Power outages came in different and unexpected ways. Assume your energy back up plan will fail and have a plan B.
Power generator was stolen to one of our clients. Of course, this caused a hard shutdown of the servers and great damages. In another instance, their solar panels were destroyed by the event, and the most common one, the backup generator failed.
Tip #4 Storage fuel for your generators
Have enough fuel storage for your generators. During the first weeks, the distribution network was heavily impacted for different reasons. For example, truck drivers were impacted by the hurricane as well and they were protecting their loved ones, not being able to transport fuel.
Also, demand for gas and diesel rose sharply those weeks and there were not enough drivers, nor trucks to deliver. Thus, ideally, you should have two weeks’ worth of fuel on site before the event.
Tip #5 Store spare parts
Emergency generators are made to be used as backups. Hence, when you use them for thirty days in a row, they break. Therefore, if all companies are running on generators, there are a lot of generator failing resulting in a shortage of spare parts. Hint, keep enough spare parts on site, particularly oil and filters.
Tip #6 Learn how to fix your own generators
When generators fail, you need a mechanic to fix them. Thus, if many generators are failing, the result is a shortage of mechanics. Hint, learn the basics of your generator, at least learn how to change the oil and filter to prevent the failure.
Tip #7 Have more than one generator
Managing your concentration risk is key. Having just one generator is not the best idea. Try to separate generator by type of equipment. If you have industrial equipment that might cause heavy usage peaks, separate those from your more sensitive equipment, like servers.
One customer purchased a smaller, high quality generator just for their data center. It’s a shame that they did it after the servers were damaged because of unstable power produced by their main generator and had to replace them.
Tip #8 Design and train your personnel in more than one contingency plans, besides using the Cloud as a recovery tool
Remember, if your business was hit hard, so was the home of all your employees. Consider alternate plans for them to work remotely. Here are some ideas:
- Reserve hotel rooms. One of our customer operates across the island. They reserved rooms in hotels close to their offices in different regions. Although the plan did not work (hotels were whether destroyed or occupied by FEMA and the military forces); we think it was a clever, clever idea.
- Have your own Cloud space. In Fusionworks specific case, we are a Cloud company. Our people can work anywhere, and they did. Some employees worked from the continental United States. Yes, they left immediately. They left but they could work. We talked to customers and friends and found various places.
- Create alliances. For example, those who lived in Caguas or Gurabo could work out of Edic College in Caguas. The CEO, a great friend and a Fusionworks customer was kind enough to accommodate us. You must make your best effort to help your people. Having those consultants spend two hours in traffic to get to their homes on streets with no light is dangerous.
- Keep stress under control. Arriving home with no power or water is just brutal. One must get creative and flexible to adjust to the new reality. As we all know here in PR, during this situation one can lose composure very easily, thus reducing stress in your people is a must.
- Have at least one communication land line. As we learned, communications land lines never failed during the emergency. Thus, if possible, have your core team activate their land lines in their home. One company activated a conference call where all the management team would communicate through their respective land lines and could make timely decisions even during the event.
Tip #9 Be prepared for the worst scenario
When preparing your contingency plan, one scenario needs to be on assuming technology won't be available, meaning software applications specifically. All the panelists plans assumed their applications were going to be up and running and available as usual; and as we all know now, that was not the case.
How are you going to deliver and bill your goods; pay your employees? How will you know who owes you what; or how will you keep control over your inventory? Bottom line, you know your business, run through your processes and get ready to go manual.
Tip #10 Save some cash
Heard the phrase, cash in king? We learned it the hard way. No banks nor ATM’s working for about two to three weeks. If you’re caught short on cash, both personally and in your business, you will understand what panic is. Have enough cash to sustain your family for at least three weeks. Furthermore, have enough cash to send your family away if necessary.
In these dramatic circumstances, taking care of your family will add a lot of stress, even more if any of them require medical attention. You will need to focus on getting your business back on its feet as soon as possible, the less distractions the better. From a business stand point, having cash is important as well.
Your employees are counting on their payroll and probably are not economically prepared for an emergency like this. You need them to show up to work, you want to be up and running before your competition. Paying your people will give you an edge and it is also the right thing to do.
- () -
After the discussion, our partner Oracle proceeded to explain their Cloud strategy in a presentation called; Journey to the Cloud. As we see it, the Cloud is here to stay and that's why we need to use the Cloud as a recovery tool.
Companies must understand all the alternatives out there and choose their own path. In a post María era, companies must use what the Cloud must offer in their favor to remain as much as “Business as Usual” during and after a catastrophic event. The following part summarizes the different journeys you can take, depending on your status, to use the Cloud as a recovery tool.
Journey to The Cloud as a recovery tool: Related terms
Stands for the equipment and software owned, operated and hosted at an organization facilities. These are meant to be used exclusively by the owner.
Gartner defines public cloud computing as a style of computing where scalable and elastic IT-enabled capabilities are provided as a service to external customers using Internet technologies—i.e., public cloud computing uses cloud computing technologies to support customers that are external to the provider’s organization.
Using public cloud services generates the types of economies of scale and sharing of resources that can reduce costs and increase choices of technologies. From a government organization’s perspective, using public cloud services implies that any organization (in any industry sector and jurisdiction) can use the same services (e.g., infrastructure, platform or software), without guarantees about where data would be located and stored. And there is another reason to use the Cloud as a recovery tool.
Gartner defines PaaS platform as a service (PaaS) offering, usually depicted in all-cloud diagrams between the SaaS layer above it and the IaaS layer below, is a broad collection of application infrastructure (middleware) services (including application platform, integration, business process management and database services).
Gartner defines Infrastructure as a service (IaaS) as a standardized, highly automated offering, where computer resources (i.e. hardware), complemented by storage and networking capabilities are owned and hosted by a service provider and offered to customers on-demand.
Gartner defines software as a service (SaaS) as software that is owned, delivered and managed remotely by one or more providers.
Now that we are clear on the terminology, let's explain the different journeys outlined in the meeting:
The Journey to the Cloud
This journey is about transitioning slowly, a more conservative approach. There are certain functionalities that you are way better of doing them on the Cloud, perhaps to lower cost, minimize complexity or lower risks. Some examples of these less critical services are; Backup as a Service, Archive Storage, test and/or training environments.
Also, this journey may include complementing your on-premise applications with those offered as a service. For example, you might have an ERP on premise but you need to update your inventory management process. In such an instance, you might want to explore implementing a cloud solution like Log Fire to implement the Cloud as a recovery tool.
Log fire is a cloud native warehouse management solution recently acquired by Oracle. Typical innovations are; implementing a cloud human capital management application integrated with your on-premise payroll system; implementing a cloud financial reporting solution over your general ledger or extending your order entry module with a Cloud CRM.
This second journey is one interesting choice and perhaps Oracle is the only mayor vendor out there offering it. In this instance, Oracle places its infrastructure and platform on the customer's premises.
However, both, the infrastructure and platforms are owned and managed by Oracle. Oracle sets up a Cloud just for you; and in such case, we can create a Cloud as a recovery tool plan for you. In this instance, the Cloud as a recovering tool at Customer can be replicated in Oracle’s Cloud for redundancy. This option seemed to be very interesting for participants because most of them lost communications over the internet after María.
Then we found the third journey, moving workloads. This allows you to process peak transactions at Oracle’s public cloud. Let’s say you are a Medicare health insurance provider and your period to process applications runs for a few months. You might want to move those peak processes to Oracle’s public cloud and pay form them as a service. When the peak is over, you turn off the service potentially reducing your costs.
4. Seize the opportunity
The fourth journey offers the possibility to develop new applications, or install prepackaged applications in the cloud. This is where most CIO’s need to focus on new projects.
Instead of overanalyzing hardware and infrastructure costs because of the significant dollar commitment it entails, business users can start quickly prototyping and implementing systems because infrastructure and platforms are now elastic, you can buy precisely what you need for the time you need sized for the stage the project is in.
Thus, you can activate a server with minimum capacity to start the project and increase the capacity later as the project progresses. Projects can now start immediately, and today, being fast is key to survive.
5. Upgrading and updating
The fifth journey is about modernizing your complete on-premise stack. For example, replacing your on-premise ERP with a cloud one. In this instance, you are obtaining a service, which is very different from acquiring a software.
A service, is a dynamic offering that is continually being managed, improved and modernized by the provider. It is a different experience, is the new way of doing business. Now, the small and medium can have access to solutions that were only available to large enterprises.
The SaaS offering, when compared to on premise, is like replacing an old eight-cylinder car with a new fuel efficient, yet powerful Tesla that gets updated via an internet connection. Take Fusionworks for example, we have the innovation ship ingrained in our culture and even though we were born before the Cloud age, we have already replaced all our software for SaaS offerings. Believe me, we are not going back.
The last one is no journey – simply put, newer companies are mostly cloud born. If you find a start up buying servers and on-premise software, take it as a signal. This means, do not invest in it. It will be out of business the day after you give them your money.
Whatever your stage is, what was clear during the panel is that we all must reassess our disaster recovery plans with what we have learned after María and that reassessment must include the Cloud as recovery tool.
Furthermore, with or without a disaster, the Cloud is the future and we all must have a Cloud as a recovery tool plan suitable for our organizations.