I’m giving a talk at the Web2.0Expo on Wednesday called “Failures, Disasters, & Resilient Design“. I’ll be using some of the lessons-learned from our deployment and how they apply to managing technology.

Failures, Disasters, & Resilient Design
(Click here for ExpoCal!)

Jesse Robbins, ex-Amazon, OpenAid

Track: Web Operations
Date: Wednesday, April 18
Time: 4:30pm - 5:20pm
Location: 2008

This talk reviews real world examples of complex systems failure and how they relate to Web 2.0. Drawing on his work as the “Master of Disaster” at Amazon and using examples from Nuclear Power to his own experiences after Hurricane Katrina, Jesse will provide:

• A methodology for managing risk with Web 2.0 technologies.
• Best practices for building Resilient Systems with Web Services & Web Scale Computing
• Stories and pictures of things blowing up!

Here’s the full-version of one of my slides:

“You don’t choose the moment, the moment chooses you. You only get to choose how ready you will be when it does…”

I passed this Fire Academy lesson to my friend Ethan as I stepped onto a plane, returning home from vacation. I offered these words hoping he would find comfort, or at least company, while he wrote a letter to our many friends affected by the Blue House tragedy.

As the plane climbed out of San Jose I began writing down what I remembered from my crisis communication books, along with the great class taught by the King County Chaplains. Lost in sad thoughts, I almost missed the announcement the flight attendants were making: “May I have your attention. If there is a doctor or nurse onboard please ring your flight attendant call button.”

I rang mine and told the nervous looking flight attendant who came over that I was an EMT and happy to help. She asked me to come back where I found a 28 year old man, clutching his face in pain. The flight attendants said “He’s had eye surgery, and I guess he shouldn’t have flown.”

I began putting on my gloves, which I had instinctively grabbed from my bag as I left my seat. As I turned to face the patient a man abruptly stood up and said “I’m an eye surgeon”.

The doctor asked a few quick questions and then just as quickly… poked the patient in the eye. He stared intently for a moment, and told the flight attendants “This plane needs to come down, we need to reduce the pressure or this man will lose his vision in his only good eye”.

One of the flight attendants was communicating with the pilot and a medical dispatcher via a headset. I suggested that the doctor speak, and gave him the quick lesson in simplex communication. “Press, Pause, Speak, ‘Over’”.

While he did this, I asked for the medical bags and began inventorying them, suspecting that the kits were probably similar to an ambulance jump-kit without drugs or needles. I was happy to find that the aircraft “Enhanced Medical Kits” are really well stocked, and told the doctor what drugs and equipment were available to him.

A few minutes later the doctor was performing minor surgery on this man’s eye, with me holding a flashlight, setting up equipment, keeping a log, and whatever else needed to be done.

The pilots brought the plane down to 3,000 feet as the doctor finished de-pressurizing the man’s eye and saving his vision. I picked up the trash, asked the doctor if he wanted vitals or oxygen for the patient, and began writing a report.

In 2004 the FAA required commercial aircraft to carry an Enhanced Emergency Medical Kit, intended for use by medical professionals that might be on the plane. The flight-attendants can’t even open it, and most of the drugs are out of my scope-of-practice as an EMT. It’s a gift from those who have found themselves chosen by the moment… to those who are about to be.

-Jesse

(note: I migrated this post to here from my livejournal on September 22, 2007)

Christmas Eve in Palo Alto by my old fire station on Newell Road made me heartsick for firefighting. I spent most of the evening thinking at the coming expiration of my “5 year become a career firefighter plan” without having actually become a career firefighter.

I didn’t sleep well, and when my girlfriend Regina and I awoke (at 6AM) to drive to Fresno it was no farther from my mind. The drive was quiet, with me lost in thought and she lost in last-minute present wrapping.

The fog was dense on Pacheco Pass, and I was driving very carefully. I saw what looked like a serious accident and began to slow down even more. Several idiots swerved by me, nearly hitting a CHP officer who was setting out flares. Other than the cop, I didn’t see any emergency units on-scene and I could now see there were at least 6 blood covered patients huddled in the cold against the median.

I passed the accident, told Reg that I’d be stopping and found a safe spot to pull off. As I hustled toward the scene I was kicking myself for not carrying gloves on my keychain anymore. I found a guy who looked like an off-duty firefighter… crewcut, jeans, and a tattered CDF t-shirt, identified myself as an EMT and asked him if he needed a hand. He did.

Mr. CDF had already done the initial multi-casualty heavy lifiting by moving “walking wounded” patients to (relative) safety on the median, distributed basic bandages to put on minor wounds, and determined which patients would be the most critical. He must have been on scene for at least 10 minutes by himself.

We stole the first-aid kit from CHP officer’s car and began rifling through it for gloves (which was nearly impossible). This proved to be somewhat of a challenge, as it seems cops who primarily do traffic control don’t put a lot of thought into the utility of their jump-kits. They especially don’t seem to consider off duty and/or wannabe rescuers who might be rifling through their stuff. Regardless, I found a couple of glove kits, tossed one to the CDF guy and went to the car with patients he had identified as “criticals”.

The critical patients were a ~50 year old woman and her ~80 year old mother. I approached the car, introduced myself to the patients, and told them what was happening and what I was about to do.

“My name is J, I’m going to do some first aid until the fire department and ambulances arrive. There’s going to be a lot of commotion, and things will happen quickly, but everything we do is to help you…”

The daughter was pretty banged up with an obvious wrist fracture, some heavy bruising on her chest from where it looked like she had struck the steering wheel. She was complaining of severe pain in her wrist, was feeling dizzy, having difficulty breathing, and saying she felt like she was going to pass out. Grandma was looking much better, had no obvious trauma and was not having any trouble breathing. The Daughter was going to get treated first.

I had just started taking her vitals when the first Fire Engine arrived from about 10 miles away (They had to go through the same fog to get there so it took a while). I got the Captain’s attention, gave him the status for the two patients, and asked him for a better Stethoscope. (The CHP officer’s steth was missing the rubber earpieces, making it useless and uncomfortable) He tossed me a new one from their rig, and I got the vitals on the daughter just as the ambulances began to arrive.

I asked the Captain if he wanted me to start holding C-Spine on the patient, to which he responded “Yeah, if you’re up for being here a while”. I got behind her in the back seat of the car, told her that I was going to hold her head still and that it was important she move as little as possible and not shake her head to answer questions. I found a comfortable position, pulled the sleeves down on my jacket, and took control of her head.

(Almost every time I’ve held C-Spine for any extended duration I’ve managed to assume the most uncomfortable position possible, resulting in every limb going numb or cold. By the time we’re ready to move the patient onto the backboard, I have to be relieved because I’m a paralyzed and thoroughly frozen EMTsickle. After 5 years of doing this, I finally managed to get it right and was comfortable *before* I put my hands on the patient.)

The various responders began to assemble backboards and gurneys near the car as the Captain and one of the Firefighters began splinting the woman’s fractured wrist. Life-flight was on the way to a nearby parking lot (above the dense fog), we had her on high-flow oxygen, and things seemed to be going in the right direction.

I started talking to Grandma, who had insisted that I call her Grandma, and it was obvious that she was turn for the worse. She was starting to struggle a little for breath, and seemed like she was getting shocky. The paramedics had arrived and started treating her. She was going downhill pretty fast, and so a second helicopter was ordered.

We were able to move the daughter onto the backboard and get her transferred into the ambulance. I returned all the gear that I had absconded with and then asked if there was anything else I could do (there were now 3 fire engines on scene, 4 CHP units, 3 ambulances, two turtle doves, and a partridge in a pear tree). Mr. CDF and I were released and we scooted on our merry little way.

I suspect that Grandma had a hemothorax (blood in the lung cavity) and was probably in greater immediate peril than her daughter. I’ll never know how they fared at the hospital, but I did the best I could for both of them with the time and resources available. I hope they ended up with nothing more than a fractured arm for the daughter and rib for Grandma, that they were only a few hours late to open presents with their family, and are thankful for the gift that the accident wasn’t worse (it could have been much worse).

In that answering the call to help those in need I was given a gift too… An opportunity to write my next 5 year plan remembering why I wrote the first one. It was an unexpected gift, but one that I’m immensely grateful for.

Merry Christmas.

Date: Sun, 2 Jul 2000 21:26:33 -0700
From: Jesse Robbins
Subject: That’s Firefigher-Intern Peon to you… (Part 1)

I’ve told you all about my training, about the infamous Captain Roberts, about the members of my team, and about what it is like to stand 50 feet in the air on a freestanding ladder gin. I’ve told you about internships, and EMT training, and clinical time in the hospital… I’ve even suggested that I might tell you all about what going through an academy is like. The one thing that I haven’t told you about is what I do now that I’m an intern. This update will come in several sections…

My day begins on the day before the shift at around 11pm. Hours ago I realized that I needed to have all of my gear ready to go for the morning and then got caught up in reading and writing emails. At some point, the “oh shit” factor begins to click in my now tired head, grumbling as I start a load of laundry and begin ironing my shirt out. My pants are still dirty from the week before. At some point, the wash finishes and I put the wet clothes in the dryer. My uniform shirt is now hanging off the chin-up bar, and my pants are in the dryer along with my uniform T-shirts, shorts, and black boot socks.

I set the alarm for 6:30am, I set my other alarm for 6:31am, and I go to bed. I then get suffer some paranoia and get out of bed to check to make sure both alarm clocks are actually set properly. I have a terrible fear of oversleeping and being late for duty.

I wake up at 6:30 and get out of bed… starting the dryer again so that my clothes will be wrinkle free, grab a quick shower, get dressed, and then fold some of the clothes hot from the dryer and put them in my bag. I’m now wearing my Palo Alto internship uniform: Light brown shirt with patches, black “Sam Browne” belt, matching light brown pants, black socks, and polished zippered steel toe duty boots. On my belt I have a glove pouch, leatherman super-tool, radio belt clip, and my EMT shears ( EMT shears are the same as Paramedic shears but just work harder ;-). I take a brief look in the mirror to make sure that I am totally respectable, and then grab my bag, my sleeping roll (bag, pillow, down-comforter), my backpack with books and forms, and run to my truck.

I’m presently assigned to Engine 3. For those that know Palo Alto, Station #3 is located at Embarcadero and Newell, just before Middlefield Road. It is the oldest remaining station in Palo Alto. It is a single company station, housing one of the older open cab engines and the rescue boat. It’s 7:30 on the nose as I pull into the driveway and open the sliding gate. As an intern, I am supposed to arrive first in the morning and also get lowest priority in parking… so unless there are 4 spaces open, I park pressed to the fence by the gate. I leave the gate open for the shift change and walk in the door. The previous night’s crew is awake, coffee is usually brewing, and sometimes the newspapers are in. If not, I make the coffee and go outside to get the newspapers and lay them out neatly on the table.

By this point, I’ll probably speak with the firefighter on duty about their shift, find out if anything interesting happened, and if there is any special information that I need to know. Then it’s off to my hanger to get my helmet, turnouts (heavy protective clothing), and “red bag” containing my wildland firefighting equipment. It all must be checked to make sure that it is in proper working order, that nobody played any practical jokes and put weird things in it or hidden anything, that all the things I need are either in the pockets or otherwise attached to the jacket, suspenders, and pants. I carry a set of firefighting gloves, a NOMEX flash hood, straps to hold my gloves, heavy wire cutters, a hose-strap, earplugs, a few dollars in cash, and sometimes a small disposable camera. All this equipment gets stored on the engine or attached in various ways. Because the cab is open for the “tailboard” (rearward facing seats behind the driver and captain) getting all my equipment aboard is a bit tricky. I sit behind the driver/engineer on the left side of the engine. My helmet and coat go next to my seat on the motor compartment. My boots and red-bag get attached to the side of the engine on two separate folding foot holds. My box of latex gloves and at least one water bottle go to the right and left of the seat between the seatbelt equipment. Anything else gets crammed into whatever compartment I can find to fit it in where it is not in anyone way. The last thing I do is connect my headset to the engine’s communication system. It’s very noisy on the engine, especially sitting in an open cab with the engine running and the sirens on. The headset protects my hearing and allows the crew to communicate while we are driving around.

I’m now officially “on duty”. If there were an intern from the previous day on, I would relieve them by telling them that “I’ve got it from here”. As there is not yet an intern on every shift, it’s really mine from when I show up, but there is a certain feeling of the changing of the guard when I put on my equipment.

At this point I begin checking out the engine itself. I am responsible for each and every tool, hose, nozzle, radio, or medical supply on the engine. I start with the “green bag” containing all our basic first aid supplies and oxygen equipment as it is the most often used and abused tool on the rig. Sometimes things need to be restocked, recharged, or refilled, and almost always I have to rearrange the some of the gear so it’s how I like it. Much of my time on a medical call is spent grabbing things from the bag and giving them to whomever is the patient person so it’s extremely important that I can get at things when requested. Then I test out the defibrillator, suction unit, c-spine equipment, and burn kit to make sure everything is operational, stocked, and in good condition. I inventory the Self-Contained-Breathing-Apparatuses (SCBAs), spare air bottles, tools, ropes, lights, fire extinguishers, ladders, hoses, and hose appliances on the rig. At some point the firefighter shows up and begins his or her own check of the gear. It’s both of our responsibilities to make sure everything gets checked out. At 8:00am we run up the flags and shortly thereafter I finish checking out my equipment, the oncoming engineer and captain arrive and relieve their counterparts. We have a brief station meeting and then it’s time for the captain to do paperwork, the engineer to make sure the Engine is running properly, and for the firefighter and I to begin cleaning the station…

During this time, we are always ready to go on a call. Sometimes the bell rings and we are out for the rest of the day, but usually we make it at least to this point without screaming out of the station with sirens blaring and lights ablaze.

-Jesse

———————–
Jesse Robbins
San Jose, CA

Date: August 25, 1999 10:04:50 PM PDT
From: Jesse Robbins
Subject: Mission College Technical Rescue Team, or “what happened to my beard and hair”

It seems that the little rescue group I was involved with has now officially evolved into the “Mission College Technical Rescue Team”. Complete with logos, patches, sponsors, jump-suits, and yes… even a secret handshake.

Some interesting personal changes have taken place since I last wrote. Most of which are a direct result of a 30 second encounter with the director of the Fire Science program. During the first indoor lecture of the class, he came into the room to discuss his pleasure with all the hard work we were doing, and explain his views of the future of the course.

He stopped for a second, stared at me, and said:
“But we won’t be representing the program with any Monkey-Faced-Little-Beards or girly little earrings now… will we…”

No, clearly we won’t. I am now regularly clean-shaven, with no earring to be seen, and definitely no “monkey faced little beard”. I am also now sporting a crew-cut and although my Breathing Apparatus makes a proper seal, I nowactually look my age… (scary thought)

The MCTRT has already done rescue demos for the City of San Jose and7 Chinese Generals who were visiting as part of some kind of Sister CityProgram. This included a confined-space-hazardous-materials rescue and the first public demonstration of our corporate sponsor SKEDCO’s new HAZMAT evacuation system. (All hail the sponsor!!!)

Most recently we have been working on the execution of the “Mid-air Pickoff”. This technique is for rescuing people stuck on a rope or on a ledge which allows us to transfer the patient from a failed rope system to our system and lower them to the ground.

We’ve been training at around 18 feet. Same level of difficulty, high degree of safety in case of a critical failure. It’s extremely technical rescue, and is easy to get yourself hung up… literally.

The biggest lessons learned so far are:

• On Ropes:
  1. Your rope is your friend.
  2. Don’t step on your friend.
• On Harnesses:
  • A bad harness is like a bad lover,
    Hang around together too long and both your legs fall asleep.
• On Equipment:
  1. Your equipment is your friend.
  2. Don’t drop your friend.
• On when equipment is dropped:
  1. Equipment will be dropped at the most critical time of the rescue.
  2. The captain will be filming you with a video camera.
  3. The captain will be zoomed in on your facial expression.
  4. The equipment will be expensive.
  5. The equipment will be marked with a yellow band.
  6. The yellow band will have special meaning.
  7. That meaning will not be “This equipment can be dropped”.
  8. That meaning will not be “This equipment can survive a drop”.
  9. That meaning will not be “This equipment is student loaner equipment, and is inexpensive”.
  10. That meaning will be “This equipment can’t be dropped.”
  11. That meaning will be “This equipment can’t survive a drop”.
  12. That meaning will be “This equipment is the captain’s PERSONAL in-service duty equipment, and is very expensive.”
  13. No matter how hard the person that dropped the equipment tries to apologize for the error and replace the equipment, no such remedy shall be accepted.
  14. The equipment, and a 8×10 glossy image will be displayed of the person who dropped the equipment at the exact moment the equipment was dropped. The face of the perpetrator will be one of grief and fear, having realized the error and trying desperately to grab the item before it bounces off the ground below.
  15. That display will be placed in a conspicuous location with the school.
  16. Ridicule for the drop will be administered immediately after the event occurs, and shall last no less than 1 week and no more than eternity.

More to come…

———————–
Jesse Robbins
San Jose, CA