I’m giving a talk at the Web2.0Expo on Wednesday called “Failures, Disasters, & Resilient Design“. I’ll be using some of the lessons-learned from our deployment and how they apply to managing technology.

Failures, Disasters, & Resilient Design
(Click here for ExpoCal!)

Jesse Robbins, ex-Amazon, OpenAid

Track: Web Operations
Date: Wednesday, April 18
Time: 4:30pm - 5:20pm
Location: 2008

This talk reviews real world examples of complex systems failure and how they relate to Web 2.0. Drawing on his work as the “Master of Disaster” at Amazon and using examples from Nuclear Power to his own experiences after Hurricane Katrina, Jesse will provide:

• A methodology for managing risk with Web 2.0 technologies.
• Best practices for building Resilient Systems with Web Services & Web Scale Computing
• Stories and pictures of things blowing up!

Here’s the full-version of one of my slides:

“You don’t choose the moment, the moment chooses you. You only get to choose how ready you will be when it does…”

I passed this Fire Academy lesson to my friend Ethan as I stepped onto a plane, returning home from vacation. I offered these words hoping he would find comfort, or at least company, while he wrote a letter to our many friends affected by the Blue House tragedy.

As the plane climbed out of San Jose I began writing down what I remembered from my crisis communication books, along with the great class taught by the King County Chaplains. Lost in sad thoughts, I almost missed the announcement the flight attendants were making: “May I have your attention. If there is a doctor or nurse onboard please ring your flight attendant call button.”

I rang mine and told the nervous looking flight attendant who came over that I was an EMT and happy to help. She asked me to come back where I found a 28 year old man, clutching his face in pain. The flight attendants said “He’s had eye surgery, and I guess he shouldn’t have flown.”

I began putting on my gloves, which I had instinctively grabbed from my bag as I left my seat. As I turned to face the patient a man abruptly stood up and said “I’m an eye surgeon”.

The doctor asked a few quick questions and then just as quickly… poked the patient in the eye. He stared intently for a moment, and told the flight attendants “This plane needs to come down, we need to reduce the pressure or this man will lose his vision in his only good eye”.

One of the flight attendants was communicating with the pilot and a medical dispatcher via a headset. I suggested that the doctor speak, and gave him the quick lesson in simplex communication. “Press, Pause, Speak, ‘Over’”.

While he did this, I asked for the medical bags and began inventorying them, suspecting that the kits were probably similar to an ambulance jump-kit without drugs or needles. I was happy to find that the aircraft “Enhanced Medical Kits” are really well stocked, and told the doctor what drugs and equipment were available to him.

A few minutes later the doctor was performing minor surgery on this man’s eye, with me holding a flashlight, setting up equipment, keeping a log, and whatever else needed to be done.

The pilots brought the plane down to 3,000 feet as the doctor finished de-pressurizing the man’s eye and saving his vision. I picked up the trash, asked the doctor if he wanted vitals or oxygen for the patient, and began writing a report.

In 2004 the FAA required commercial aircraft to carry an Enhanced Emergency Medical Kit, intended for use by medical professionals that might be on the plane. The flight-attendants can’t even open it, and most of the drugs are out of my scope-of-practice as an EMT. It’s a gift from those who have found themselves chosen by the moment… to those who are about to be.

-Jesse

(note: I migrated this post to here from my livejournal on September 22, 2007)

Christmas Eve in Palo Alto by my old fire station on Newell Road made me heartsick for firefighting. I spent most of the evening thinking at the coming expiration of my “5 year become a career firefighter plan” without having actually become a career firefighter.

I didn’t sleep well, and when my girlfriend Regina and I awoke (at 6AM) to drive to Fresno it was no farther from my mind. The drive was quiet, with me lost in thought and she lost in last-minute present wrapping.

The fog was dense on Pacheco Pass, and I was driving very carefully. I saw what looked like a serious accident and began to slow down even more. Several idiots swerved by me, nearly hitting a CHP officer who was setting out flares. Other than the cop, I didn’t see any emergency units on-scene and I could now see there were at least 6 blood covered patients huddled in the cold against the median.

I passed the accident, told Reg that I’d be stopping and found a safe spot to pull off. As I hustled toward the scene I was kicking myself for not carrying gloves on my keychain anymore. I found a guy who looked like an off-duty firefighter… crewcut, jeans, and a tattered CDF t-shirt, identified myself as an EMT and asked him if he needed a hand. He did.

Mr. CDF had already done the initial multi-casualty heavy lifiting by moving “walking wounded” patients to (relative) safety on the median, distributed basic bandages to put on minor wounds, and determined which patients would be the most critical. He must have been on scene for at least 10 minutes by himself.

We stole the first-aid kit from CHP officer’s car and began rifling through it for gloves (which was nearly impossible). This proved to be somewhat of a challenge, as it seems cops who primarily do traffic control don’t put a lot of thought into the utility of their jump-kits. They especially don’t seem to consider off duty and/or wannabe rescuers who might be rifling through their stuff. Regardless, I found a couple of glove kits, tossed one to the CDF guy and went to the car with patients he had identified as “criticals”.

The critical patients were a ~50 year old woman and her ~80 year old mother. I approached the car, introduced myself to the patients, and told them what was happening and what I was about to do.

“My name is J, I’m going to do some first aid until the fire department and ambulances arrive. There’s going to be a lot of commotion, and things will happen quickly, but everything we do is to help you…”

The daughter was pretty banged up with an obvious wrist fracture, some heavy bruising on her chest from where it looked like she had struck the steering wheel. She was complaining of severe pain in her wrist, was feeling dizzy, having difficulty breathing, and saying she felt like she was going to pass out. Grandma was looking much better, had no obvious trauma and was not having any trouble breathing. The Daughter was going to get treated first.

I had just started taking her vitals when the first Fire Engine arrived from about 10 miles away (They had to go through the same fog to get there so it took a while). I got the Captain’s attention, gave him the status for the two patients, and asked him for a better Stethoscope. (The CHP officer’s steth was missing the rubber earpieces, making it useless and uncomfortable) He tossed me a new one from their rig, and I got the vitals on the daughter just as the ambulances began to arrive.

I asked the Captain if he wanted me to start holding C-Spine on the patient, to which he responded “Yeah, if you’re up for being here a while”. I got behind her in the back seat of the car, told her that I was going to hold her head still and that it was important she move as little as possible and not shake her head to answer questions. I found a comfortable position, pulled the sleeves down on my jacket, and took control of her head.

(Almost every time I’ve held C-Spine for any extended duration I’ve managed to assume the most uncomfortable position possible, resulting in every limb going numb or cold. By the time we’re ready to move the patient onto the backboard, I have to be relieved because I’m a paralyzed and thoroughly frozen EMTsickle. After 5 years of doing this, I finally managed to get it right and was comfortable *before* I put my hands on the patient.)

The various responders began to assemble backboards and gurneys near the car as the Captain and one of the Firefighters began splinting the woman’s fractured wrist. Life-flight was on the way to a nearby parking lot (above the dense fog), we had her on high-flow oxygen, and things seemed to be going in the right direction.

I started talking to Grandma, who had insisted that I call her Grandma, and it was obvious that she was turn for the worse. She was starting to struggle a little for breath, and seemed like she was getting shocky. The paramedics had arrived and started treating her. She was going downhill pretty fast, and so a second helicopter was ordered.

We were able to move the daughter onto the backboard and get her transferred into the ambulance. I returned all the gear that I had absconded with and then asked if there was anything else I could do (there were now 3 fire engines on scene, 4 CHP units, 3 ambulances, two turtle doves, and a partridge in a pear tree). Mr. CDF and I were released and we scooted on our merry little way.

I suspect that Grandma had a hemothorax (blood in the lung cavity) and was probably in greater immediate peril than her daughter. I’ll never know how they fared at the hospital, but I did the best I could for both of them with the time and resources available. I hope they ended up with nothing more than a fractured arm for the daughter and rib for Grandma, that they were only a few hours late to open presents with their family, and are thankful for the gift that the accident wasn’t worse (it could have been much worse).

In that answering the call to help those in need I was given a gift too… An opportunity to write my next 5 year plan remembering why I wrote the first one. It was an unexpected gift, but one that I’m immensely grateful for.

Merry Christmas.

Date: August 25, 1999 10:04:50 PM PDT
From: Jesse Robbins
Subject: Mission College Technical Rescue Team, or “what happened to my beard and hair”

It seems that the little rescue group I was involved with has now officially evolved into the “Mission College Technical Rescue Team”. Complete with logos, patches, sponsors, jump-suits, and yes… even a secret handshake.

Some interesting personal changes have taken place since I last wrote. Most of which are a direct result of a 30 second encounter with the director of the Fire Science program. During the first indoor lecture of the class, he came into the room to discuss his pleasure with all the hard work we were doing, and explain his views of the future of the course.

He stopped for a second, stared at me, and said:
“But we won’t be representing the program with any Monkey-Faced-Little-Beards or girly little earrings now… will we…”

No, clearly we won’t. I am now regularly clean-shaven, with no earring to be seen, and definitely no “monkey faced little beard”. I am also now sporting a crew-cut and although my Breathing Apparatus makes a proper seal, I nowactually look my age… (scary thought)

The MCTRT has already done rescue demos for the City of San Jose and7 Chinese Generals who were visiting as part of some kind of Sister CityProgram. This included a confined-space-hazardous-materials rescue and the first public demonstration of our corporate sponsor SKEDCO’s new HAZMAT evacuation system. (All hail the sponsor!!!)

Most recently we have been working on the execution of the “Mid-air Pickoff”. This technique is for rescuing people stuck on a rope or on a ledge which allows us to transfer the patient from a failed rope system to our system and lower them to the ground.

We’ve been training at around 18 feet. Same level of difficulty, high degree of safety in case of a critical failure. It’s extremely technical rescue, and is easy to get yourself hung up… literally.

The biggest lessons learned so far are:

• On Ropes:
  1. Your rope is your friend.
  2. Don’t step on your friend.
• On Harnesses:
  • A bad harness is like a bad lover,
    Hang around together too long and both your legs fall asleep.
• On Equipment:
  1. Your equipment is your friend.
  2. Don’t drop your friend.
• On when equipment is dropped:
  1. Equipment will be dropped at the most critical time of the rescue.
  2. The captain will be filming you with a video camera.
  3. The captain will be zoomed in on your facial expression.
  4. The equipment will be expensive.
  5. The equipment will be marked with a yellow band.
  6. The yellow band will have special meaning.
  7. That meaning will not be “This equipment can be dropped”.
  8. That meaning will not be “This equipment can survive a drop”.
  9. That meaning will not be “This equipment is student loaner equipment, and is inexpensive”.
  10. That meaning will be “This equipment can’t be dropped.”
  11. That meaning will be “This equipment can’t survive a drop”.
  12. That meaning will be “This equipment is the captain’s PERSONAL in-service duty equipment, and is very expensive.”
  13. No matter how hard the person that dropped the equipment tries to apologize for the error and replace the equipment, no such remedy shall be accepted.
  14. The equipment, and a 8×10 glossy image will be displayed of the person who dropped the equipment at the exact moment the equipment was dropped. The face of the perpetrator will be one of grief and fear, having realized the error and trying desperately to grab the item before it bounces off the ground below.
  15. That display will be placed in a conspicuous location with the school.
  16. Ridicule for the drop will be administered immediately after the event occurs, and shall last no less than 1 week and no more than eternity.

More to come…

———————–
Jesse Robbins
San Jose, CA

Date: May 25, 1999 12:37:47 PM PDT
From: Jesse Robbins
Subject: Trauma: Jesse’s Life in the ER

In my last entry, I was describing the pride and confidence being in my EMT uniform inspired in me.  As I pulled into the parking area for the ER and passed 3 ambulances cleaning up after a call that confidence rather quickly changed to excitement and fear.  Over the past months of training we have been given a practical set of skills, most of which we have been told will be useless on a real patient until we get hands on experience.

What was quickly becoming apparent to me was that this was going to be when I get that experience.  Even if closely supervised, anybody that isn’t a little shaky when they realize that responsibility for the care of another human being is about to be placed in their inexperienced hands hasn’t quit grasped the fullness of the situation.  A lot of what-if’s and oh-shit’s go through your head, mostly questioning yourability to operate smoothly under stress.

I met with the charge nurse and another Mission college EMT just finishing up her rotation.  She had done 7am-3pm, I was set for 3pm-2am.  I got a quick rundown on her day, and then was sent off to triage in the ER.  Triage(French for “to sort”) is the process that you go through in the ER to determine treatment priority for walk-in patients.  If you arrive via ambulance you will be at least given a bed right away.  The nurse working triage that shift was a wonderful woman with an Irish accent.  After a quick acquaintance we let the first patient in.

Here’s what I learned in Triage:

• About Infants:
  1. Unlike the plastic infants we train on, and unlike the books description and instructions for taking vital signs on infants, they do not hold still nor can the be bribed or otherwise convinced to stop. Palpating a brachial pulse on a wiggling infant is a joke.
     To get around this, I learned to use my stethoscope to simply listen to the heart and get a pulse that way.
  2. Using a stethoscope on a crying infant to get a pulse is harder than getting a brachial pulse on a wiggling infant.  Infants also, apparently, wiggle and wave their arms about when crying.  Sometimes they can be convinced not to cry by giving them some part of your person, clothing, or more importantly stethoscope to examine or play with.  This does not  stop the wiggling.  If you have given them the piece of equipment needed to perform the assessment in order to bribe them to stop crying, you have made an especially large mistake.
  3. Once an infant has hold of any part of your person, clothing, or stethoscope dangling off of your neck, it may be necessary to use a vehicle extrication tool such as the jaws-of-life to get them to release it.  21,000 PSI hurst spreaders are recommended.
  4. There is nothing more appealing to an infant than a stethoscope dangling from your neck.
  5. There is nothing less appealing to an infant than having some mean man in a uniform try to take away the shiny thing with blue tubes coming out of it.
  6. Once an infant grabs the head of your stethoscope in it’s clutches, it will soon have the head of the scope firmly in it’s mouth.
  7. Replacement diaphragms for Sprauge-style dual head steths come in the little box it came in.  There are 2.
• About adults:
  1. The general level of calm exhibited by a patient with a feeding tube that has been unintentionally removed by a family member providing care is proportional to the level of hysteria exhibited by that family member.
  2. If at first your attempts at speaking Spanish to a patient fail, seek a translator.
  3. The term “Student” strikes fear into the souls of the ill and injured. Therefore:
     The proper way to answer the question: “Are you a doctor?” is “No, but I’m going to help you until a doctor can see you.” and NOT “No, I’m only an EMT student, but I can help you until a nurse or doctor can see you.”

I then did 8 hours in the actual ER.  Here’s what I learned…

• About the airway:
  • A human being, unlike the airway model we use to learn suctioning,    has fluids, objects, and a movable, muscular tongue.  Also, unlike the model we use, things don’t always just fit or slide in.
  • Don’t look into the mouth without a face mask on.
  • DON’T look into the mouth without a face mask on.
  • DON’T LOOK INTO THE MOUTH WITHOUT A FACE MASK ON.
• About humor:
  • Hard as it is, you must not laugh at or tease the drunk student with scalp lacerations who fell through a mirror while urinating.  Even when he asks you “You probably think I’m pretty stupid, Huh?”.
  • Nurses, on the other hand, can be as cruel to the patient as the situation warrants.
  • Hard as it is, you must not tell the friends/girlfriends/concerned frat-persons that despite the best efforts of the ER, their friend    will live.
• About the Police:
  1. The number of times a patient will explain to you that he didn’t do anything is directly related to the number of police officers in     the room.
  2. Once the second ambulance arrives, and the victim of the first patient is put into the vicinity of the first patient, his assertion that he did nothing will become an exponential function of the number of cops in the room.
  3. Although a nearly infinite number of Police Officers can cram themselves into an ER, there will never be enough x-ray shields to fit them all behind.
  4. Once the police have positioned themselves in a room, the doctor will then arrive and not have enough space to get to the patient.
  5. Once a patient has been arrested, the number of times he will say    “Thats f***ed up” is twice the number of officers in the room.    Also, every time an officer enters the room, he will say it again.
• About equipment:
  1. As a firefighter, we are trained to trust our equipment.  Hospital workers are trained not to trust their equipment.
  2. The more a patient is bleeding, the less time it will take to get a CT scan completed.
  3. The more critical the patient, the more likely the respirator will fail.
  4. I had a wonderful time in the ER.  I have a newfound respect for the people who work there every day.

Up next… How to remove the roof of a car in 2 minutes with only a hack-saw and a lift jack.
Hope this finds everyone well.

-jesse