In the Event of a Disaster … Whats the Plan? Session 111 David Richard VP of Information Technology FARA Insurance Services. - презентация

1 In the Event of a Disaster … Whats the Plan? Session 111 David Richard VP of Information Technology FARA Insurance Services

2 © 2009 Progress Software Corporation. All rights reserved. Bad THINGS Can Happen Photo Courtesy of FEMA 2

3 © 2009 Progress Software Corporation. All rights reserved. Putting THINGS Into Perspective Event Severity Crisis Disaster Catastrophe 3 Risks Floods Fire Earthquakes Hail & lightning Tornadoes Power failures Hardware failures Communication failures Loss of DATA Political Unrest Terrorism & Sabotage Computer crimes & Malware Human Error Economic Sanctions Man Made Natural Disruptions

4 © 2009 Progress Software Corporation. All rights reserved. Failure Scenarios 4 Adapted from Blueprints for High Availability by Evan Marcus and Hal Stern (ISBN: )

5 © 2009 Progress Software Corporation. All rights reserved. Evaluating The Risks 5 "Half of U.S. corporations rate their internet downtime costs at more than $1,000 per hour." - Yankee Group

6 © 2009 Progress Software Corporation. All rights reserved. Business Continuity (BC) & Disaster Recovery (DR) 6 What is the difference? Data Business Continuity People Technology Process Disaster Recovery (Planned & Unplanned)

7 © 2009 Progress Software Corporation. All rights reserved. The Critical Components of a Business Continuity/Disaster Recovery Plan Phase 1Mitigation Assessing Events Risk Avoidance (Management, People, Process & Technology) Identifying Business Requirements Phase 2Preparedness / Roadblocks Determining Recovery Objectives Planning Testing Implementing Phase 3Response Calling it a disaster – Who makes the call ? Failing Over – is your DR site ready? Phase 4Recovery Recovering to full production Business returns to normal 7

8 © 2009 Progress Software Corporation. All rights reserved. Assessing Events Phase 1 - Mitigation 8 Customer Impact Value Probability Factor Scale High Impact / High Probability Low Impact / Low Probability Hurricane Tornado Workplace Violence Snow Storm Terrorist Attack Earthquake Virus Attack High Impact / Low Probability Low Impact / High Probability Computer Failure Staffing Issues Crisis Management: Planning for the Inevitable, (1986, 2002) (Stephen Fink)

9 © 2009 Progress Software Corporation. All rights reserved. Determining Recovery Objectives Phase 2 - Preparedness / Roadblocks 9 Im up and running in seconds, but Ive lost a days data I lost no data but it took me a week to get back up and running hrs days wks mthssecs mins Zero Downtime secs mins hrs days wks mths Freshness Recovery Point Objective Recovery Time Objective What are my disaster recovery needs?

10 © 2009 Progress Software Corporation. All rights reserved. When The Planning-tire Hits The Reality-road Phase 3 & 4: Response & Recovery Response Make the decision – Activate FOLLOW THE PLAN!!!! Utilize Failover Capabilities Document what you did along the way Recovery Fail back into production Perform a post-mortem -What worked? -What didnt work? -What could you have done better? 10

11 © 2009 Progress Software Corporation. All rights reserved. FARA – Business Continuity Case Study FARA.com Provides Insurance Services – including SaaS Web Apps Headquarters in Mandeville (near New Orleans) 18 US offices 400 employees 300 contractors $40 USD million - annual revenue 11 Head Quarters

12 © 2009 Progress Software Corporation. All rights reserved. FARA – Business Continuity Case Study Pre Hurricane Katrina Mitigation & Preparedness phase through 2004 Response -Tested thanks to Hurricane Ivan (2004) Recovery -Successful Failback from Houston 12

13 © 2009 Progress Software Corporation. All rights reserved. FARA – Business Continuity Case Study Response Phase Welcome Hurricane Katrina (2005) Aug 26: Friday Aug 27: Saturday Aug 28: Sunday 13 Enters Gulf of Mexico - Strengthens Evacuations start and the panic begins Decision to activate BC/DR plan given Payroll completed on Sat. HQ secured Tapes, Supplies & Hardware loaded Employee Hotline forwarded Essential Staff sent to Baton Rouge Tested telco had failed over to DR site Head Quarters

14 © 2009 Progress Software Corporation. All rights reserved. FARA – Business Continuity Case Study Response Phase Welcome Hurricane Katrina Aug 29: Monday 14 Katrina makes landfall (early am) Tropical storm winds in Baton Rouge 8 FARA company locations impacted 350 (88%) of workforce affected spurious telco & news communications US postal much slower than normal BUT Business as usual 968 New claims opened over 14 days uninterrupted claim check printing (+4,000) Head Quarters

15 © 2009 Progress Software Corporation. All rights reserved. FARA – Business Continuity Case Study Response Phase Welcome Hurricane Katrina Aug 30: Tuesday 15 Headquarters Damaged (not substantially) City power grid down (fix time = 3-6 weeks) Negotiated additional DR office space Ordered 200 new PCs Leased additional office space start processing 100s property claims Re- Focus on people and business Employee benefits call center Re housed 35 employees Redeploy 15 people to Houston office Lease new office space Hyatt Hotel 8 blocks from a FARA Office Location for Exchange 06

16 © 2009 Progress Software Corporation. All rights reserved. FARA – Business Continuity Case Study Recovery Phase Post Hurricane Katrina Sept 12: Monday Debrief Business Non Tech IT 16 W Resumed HQ in Mandeville Housing in Baton Rouge (family & pets) Better Executive re-location strategy Communications (voice) Direct Deposit of WC payments Plastic wallets, twitter, website, use USB drives, employee debit cards … Strength of key vendors such as Corp Express, AT&T, CDW, Progress Helped us truly define mission critical systems OpenEdge Replication Make adjustments & re-plan & re-test

17 © 2009 Progress Software Corporation. All rights reserved. 7% Probability of Unplanned Downtime Seems Like a Small Risk But Sometimes BAD THINGS Happen 17

18 © 2009 Progress Software Corporation. All rights reserved. Summary Mitigation Identify Business Requirements: RTO & RPO Planning Choose, Implement, Test & Document Response Whos in Charge? Execution & Documentation Recovery Execution & Documentation 18

19 In the Event of a Disaster … Whats the Plan? Session 111 David Richard VP of Information Technology FARA Insurance Services