75 million people are affected by fiber-cuts caused by a ship that tried to moor off the coast of Egypt in bad weather on last week. Phone and internet traffic has been severely reduced across a huge swath of the region, slashed by as much as 70% in some countries.

read more | digg story

I’m giving a talk at the Web2.0Expo on Wednesday called “Failures, Disasters, & Resilient Design“. I’ll be using some of the lessons-learned from our deployment and how they apply to managing technology.

Failures, Disasters, & Resilient Design
(Click here for ExpoCal!)

Jesse Robbins, ex-Amazon, OpenAid

Track: Web Operations
Date: Wednesday, April 18
Time: 4:30pm - 5:20pm
Location: 2008

This talk reviews real world examples of complex systems failure and how they relate to Web 2.0. Drawing on his work as the “Master of Disaster” at Amazon and using examples from Nuclear Power to his own experiences after Hurricane Katrina, Jesse will provide:

• A methodology for managing risk with Web 2.0 technologies.
• Best practices for building Resilient Systems with Web Services & Web Scale Computing
• Stories and pictures of things blowing up!

Here’s the full-version of one of my slides:

Date: August 25, 1999 10:04:50 PM PDT
From: Jesse Robbins
Subject: Mission College Technical Rescue Team, or “what happened to my beard and hair”

It seems that the little rescue group I was involved with has now officially evolved into the “Mission College Technical Rescue Team”. Complete with logos, patches, sponsors, jump-suits, and yes… even a secret handshake.

Some interesting personal changes have taken place since I last wrote. Most of which are a direct result of a 30 second encounter with the director of the Fire Science program. During the first indoor lecture of the class, he came into the room to discuss his pleasure with all the hard work we were doing, and explain his views of the future of the course.

He stopped for a second, stared at me, and said:
“But we won’t be representing the program with any Monkey-Faced-Little-Beards or girly little earrings now… will we…”

No, clearly we won’t. I am now regularly clean-shaven, with no earring to be seen, and definitely no “monkey faced little beard”. I am also now sporting a crew-cut and although my Breathing Apparatus makes a proper seal, I nowactually look my age… (scary thought)

The MCTRT has already done rescue demos for the City of San Jose and7 Chinese Generals who were visiting as part of some kind of Sister CityProgram. This included a confined-space-hazardous-materials rescue and the first public demonstration of our corporate sponsor SKEDCO’s new HAZMAT evacuation system. (All hail the sponsor!!!)

Most recently we have been working on the execution of the “Mid-air Pickoff”. This technique is for rescuing people stuck on a rope or on a ledge which allows us to transfer the patient from a failed rope system to our system and lower them to the ground.

We’ve been training at around 18 feet. Same level of difficulty, high degree of safety in case of a critical failure. It’s extremely technical rescue, and is easy to get yourself hung up… literally.

The biggest lessons learned so far are:

• On Ropes:
  1. Your rope is your friend.
  2. Don’t step on your friend.
• On Harnesses:
  • A bad harness is like a bad lover,
    Hang around together too long and both your legs fall asleep.
• On Equipment:
  1. Your equipment is your friend.
  2. Don’t drop your friend.
• On when equipment is dropped:
  1. Equipment will be dropped at the most critical time of the rescue.
  2. The captain will be filming you with a video camera.
  3. The captain will be zoomed in on your facial expression.
  4. The equipment will be expensive.
  5. The equipment will be marked with a yellow band.
  6. The yellow band will have special meaning.
  7. That meaning will not be “This equipment can be dropped”.
  8. That meaning will not be “This equipment can survive a drop”.
  9. That meaning will not be “This equipment is student loaner equipment, and is inexpensive”.
  10. That meaning will be “This equipment can’t be dropped.”
  11. That meaning will be “This equipment can’t survive a drop”.
  12. That meaning will be “This equipment is the captain’s PERSONAL in-service duty equipment, and is very expensive.”
  13. No matter how hard the person that dropped the equipment tries to apologize for the error and replace the equipment, no such remedy shall be accepted.
  14. The equipment, and a 8×10 glossy image will be displayed of the person who dropped the equipment at the exact moment the equipment was dropped. The face of the perpetrator will be one of grief and fear, having realized the error and trying desperately to grab the item before it bounces off the ground below.
  15. That display will be placed in a conspicuous location with the school.
  16. Ridicule for the drop will be administered immediately after the event occurs, and shall last no less than 1 week and no more than eternity.

More to come…

———————–
Jesse Robbins
San Jose, CA