What You Can Learn from Others Mistakes The Darwin Awards for Disasters Brian L. Bowman Sr. Solution Engineer, Progress Software Session 134. - презентация

1 What You Can Learn from Others Mistakes The Darwin Awards for Disasters Brian L. Bowman Sr. Solution Engineer, Progress Software Session 134

2 © 2009 Progress Software Corporation. All rights reserved. What Do These 3 Things Have In Common? 2

3 © 2009 Progress Software Corporation. All rights reserved. What Do These 3 Things Have In Common? 3 They all are extinct because they werent adaptable or didnt meet the needs of the user!

4 © 2009 Progress Software Corporation. All rights reserved. Who Is Darwin & How Do I Get an Award? 4 Charles Darwin (1809 – 1882) First of the evolutionary biologists …life evolves by a process … called natural selection" Sources: ~1998 – The Darwin Awards Created Honoring those who improve the species...by accidentally removing themselves from it!

5 © 2009 Progress Software Corporation. All rights reserved. Who Is Darwin & How Do I Get an Award? 5 Charles Darwin (1809 – 1882) First of the evolutionary biologists …life evolves by a process … called natural selection" Sources: ~1998 – The Darwin Awards Created Honoring those who improve the species...by accidentally removing themselves from it! Todays Goal: To learn from others mistakes and not make them ourselves!

6 © 2009 Progress Software Corporation. All rights reserved. The Three Candidates - Qualifications Candidates chosen for a specific reason! Names and some details withheld to protect individual careers All candidates running an OpenEdge application Applications are mission-critical to the business The Candidates Joe Beta (Software-challenged) John Dodo (Programmer-challenged) Mr. Camino (Power-challenged) 6

7 © 2009 Progress Software Corporation. All rights reserved. Candidate #1 – Joe Beta Occurrence: Early 2009 Application Partner - SaaS hosted application Database Backup Corruption Implemented 3 rd party product for complete system duplication -They said it wouldnt affect our application! 3 rd party product caused corruption to online backup An outage caused customer to try to go to backup… Recovery Disabled 3 rd party product for the time being Question internal database corruption caused by 3 rd party product Extensive testing with 3 rd party product inconclusive 7

8 © 2009 Progress Software Corporation. All rights reserved. Candidate #1 – Joe Beta Success & Failure Success – A Good Plan Online backup every 60 minutes (small databases) Transfer backups to separate machine for DR Failures Failed to test backup 3 rd party software causes online database backup corruption What we can learn from this… Test your backup! Test your backup process and strategy! There is a reason some 3 rd party products arent supported… 8

9 © 2009 Progress Software Corporation. All rights reserved. Candidate #2 – John Dodo Occurrence: June – 2009 Application Partner – SaaS hosted application Hardware malfunction Somewhere in the SAN configuration there was something bad... The Partner discovered -Corruption in the database – RAID Corruption -Last backup was not good -Backup script (written by Management) overwrote their previous backup Recovery is still going on Custom programs from the AP to re-create lost data from spreadsheets Financials impacted – cannot report properly Staff impacted – Overtime to recover data 9

10 © 2009 Progress Software Corporation. All rights reserved. Candidate #2 – John Dodo Success & Failure Success Making backups Failures Did not test backups or recovery process Overwrote backup media Didnt backup database before deleting it What we can learn from this… Test your backup! Test your backup strategy Dont let management write your critical system scripts! 10

11 © 2009 Progress Software Corporation. All rights reserved. Candidate #3 – Mr. Camino Occurrence: Early 2009 End User of AP Data Center located in headquarters office Power Outages April - Power and telephone to building were cut -UPS capacities had not been increased along with server growth May – El Camino ran into power pole Backup generators to building did not completely work Recovery Moved data center to true data center location Backup solution - Recovery time too slow 11

12 © 2009 Progress Software Corporation. All rights reserved. Candidate #3 – Mr. Camino Success & Failure Success No Database or application corruption Moved production systems to Data Center – consistent DR testing Implemented OpenEdge Replication to replicate off-site Failures Cannot recover lost revenue Business impact still to be measured – very competitive environment What we can learn from this… You have to test your complete plan Many outages are out of your control Your DR needs will change as your company grows 12

13 © 2009 Progress Software Corporation. All rights reserved. Alternative Choices Comparing All Three Candidates User Name Validate Backup TestingAfter Imaging OpenEdge Replication Professional Assistance Joe Beta XX XX John Dodo XXXXXXX Mr. Camino XXXX 13

14 © 2009 Progress Software Corporation. All rights reserved. And The Winner (or Loser) Is… 14 Candidate #1 – Joe Beta 3 rd party backup corruption… Candidate #2 – John Dodo Hardware corruption – data loss Candidate #3 – Mr. Camino External Power Outages – lost business

15 © 2009 Progress Software Corporation. All rights reserved. And The Winner (or Loser) Is… 15 Candidate #1 – Joe Beta 3 rd party backup corruption… Candidate #2 – John Dodo Hardware corruption – data loss Candidate #3 – Mr. Camino External Power Outages – lost business

16 © 2009 Progress Software Corporation. All rights reserved. Acceptance Video 16

17 © 2009 Progress Software Corporation. All rights reserved. How Can We Learn From These People? Test - Test - Test Test your backups – just because they finished doesnt mean they are good! Test your process – Technology Changes Test your DR plan – Requirements Change 17

18 © 2009 Progress Software Corporation. All rights reserved. How Can We Learn From These People? Test - Test - Test Test your backups – just because they finished doesnt mean they are good! Test your process – Technology Changes Test your DR plan – Requirements Change Often times you have the solution – you just have to implement it correctly 18

19 © 2009 Progress Software Corporation. All rights reserved. How Can We Learn From These People? Test - Test - Test Test your backups – just because they finished doesnt mean they are good! Test your process – Technology Changes Test your DR plan – Requirements Change Often times you have the solution – you just have to implement it correctly Dont be a Dodo bird! Disaster Recovery is important for all companies There are many options – determine which one is right for your business! 19

20 What You Can Learn from Others Mistakes The Darwin Awards for Disasters Brian L. Bowman Sr. Solution Engineer, Progress Software Session 134