Velocity Web Performance & Operations Conference - Founding Chair

Amazon.com Seattle, WA

Mission: Build an “Always-On Amazon.com” by creating and improving a culture of operational and technical excellence.  

Technical Program Manager – Availability Program, Amazon Program Office

1/2006 – 11/2006

Led and participated in numerous projects and programs to improve availability for all websites bearing the Amazon brand.  Highlights included the following programs:

GameDay Program

Created an iterative program to improve company-wide availability through fault-injection into Amazon’s most critical systems.  This program continues to be effective in three ways:

• Preparation for GameDay drives the identification and mitigation of risks and impact from failure, reducing both the frequency of failure (MTBF) and duration of recovery (MTTR).
• Participation in GameDay builds individual confidence in recovering systems after failure and under stress.  This strengthens individual and cultural ability to anticipate, mitigate, respond to, and recover from failures of all types.
• The GameDay exercises trigger and expose “latent defects”, allowing the organization to decide when it will discover them, instead of letting that be determined by the next real disaster.

ScaleDay Program

Created a successful program to improve Amazon’s fourth-quarter availability through periodic stress testing on front-end websites.

Manager, Systems Engineering - System & Network Operations Center (aka: SNOC)

10/2004 – 1/2006

Managed team of systems engineers and network engineers as part of the 24x7 operations center for the Amazon.com platform.  Responsibilities included day-to-day supervision, project management, employee development, reviews, compensation, and hiring.

Leadership / Vision

• Wrote original proposal for creation and organization of the SNOC.   Proposal was adopted and has since expanded to include follow-the-sun coverage with a sister organization in Dublin, Ireland.
• Wrote and contributed to numerous policy and best-practices documents on a variety of subjects including systems architecture, distributed systems, fault-tolerant design, security, legal compliance, and disaster recovery.

Incident Management / “Master of Disaster”

• Call Leader Program - Created and participated a rotation of senior leaders engaged as the highest level of escalation to manage high-severity issues and events.
• Incident Management Initiative – Standardized high-severity event management across the company, enabling Amazon.com’s 4000+ technical employees to effectively engage and resolve problems
• Q4 “WAR Team” / Incident Management Center - Established, operated, and led the command center responsible for ensuring a successful holiday for Amazon’s customers and merchant partners.  The Q4 WAR team was comprised with senior leaders and executives from each 

Technical Contribution

• Frequent participation in product evaluation and selection, vendor management, and root cause investigation.  Technical consultation on issues ranging from software architecture to hardware lifecycle management.  Script writing and peer review.
• Sarbanes-Oxley (404) Compliance, UNIX Systems Lead – Compled Sarbanes-Oxley compliance documentation and training for all UNIX Systems and associated management controls.

Systems Engineer – Website Systems Engineering

8/2002 – 10/2004

Responsible for several key areas of the Amazon website, including over 10 teams and 25 distinct software applications.  Duties included day-to-day operation of over 400 servers, performance tuning, software benchmarking and profiling, design and architecture review, and server lifecycle management.  

Lead Backup Systems Engineer – Corporate Systems

8/2001 – 8/2002 

Participated in the design and led the implementation of Amazon’s Global Enterprise Backup System.  Mentored and supervised two systems engineers assigned to the project.   Initial daily backup size was over 12 terabytes with total capacity for 16 terabytes.  Overall archive size was over a petabyte with multiple datacenters and offsite vault locations throughout the US.

Angami Systems - Senior Network Engineer San Jose, CA

3/2000 – 1/2001

Provided senior level engineering and support for a high-end heterogeneous UNIX network.  Primarily responsible for short term projects such as script writing, application installation, system design and construction, application and OS upgrades, and documentation.  

Interliant - Systems/Network Engineer Burlingame, CA

3/98-12/1999 Employee - 12/1999 – 5/2000 Contractor

Provided senior level consulting and support for many clients located throughout the country.  Clients and projects included  Apple Computer, Lawrence Berkeley Labs, Mother Jones Magazine, Palo Alto Internet Exchange (PAIX), Ricoh Silicon Valley, and many others.

BLS Consulting, System Administrator Boulder Creek, CA

1/1997-2/1998

Worked for multiple clients building and maintaining corporate networks & systems both domestically and internationally.   Primary assignment was to deploy secure international network.  Acted as on-site liaison with governmental agencies in Panama and Jamaica and helped coordinate the construction of data centers in New York and Miami.   Managed servers and telco systems for clients at several co-location facilities in the bay area.

Publications:

Emergency Services Experience:

World Shelters Task Force One, Task Force Leader Hancock County, MS

8/2005 – 1/2006

Organized and led a 26 person task force as to assist under-served communities in the aftermath of hurricane Katrina.  Sixty- five shelters were set up, some taken down and moved, resulting in 80 deployed shelters.

Palo Alto Fire Department, Firefighter-Intern Palo Alto, CA

12/1999-6/2001

Completed California Firefighter-I certification while serving the City of Palo Alto working one 24 hour shift per week.  Duties included responding to fire and medical calls, routine station and equipment maintenance, public education and training, company drills, inspections, housekeeping duties and other tasks.