Thursday, 21 October 2010

Catalogue of Disasters proves performance engineering is essential

In my efforts to create impetus for performance engineering I have for the last few years delivered a talk at the UKCMG Forum in London, in my most recent talk I scoured the Internet with a goal to build a catalogue of disasters. So far these were the most notable. Please do add any more that you can think of in comments.

Catalogue of Disasters - By no means complete and definitely a work in progress, the information below is a reference of IT performance related crashes.

2002: Can't cope, won't copeNectar suspends Web registrations


By John Leyden • Posted in e-Business, 18th September 2002 16:20 GMT
Users flocking to sign up to the newly-introduced loyalty card scheme Nectar have flooded the site, forcing its backers to temporarily suspend Web-based signups.
As with the protracted delays in getting the 1901 Census Web site up and running, the backers of Nectar.com (Sainsbury's, Barclaycard, Debenhams and BP) have chronically underestimated demand.
The Nectar Web site has been receiving over 10,000 unique visitors an hour since Monday morning, according to Loyalty Management UK, which is running the programme.
That of course is not the main problem - The Reg get more visitors at peak times, for instance - no, the difficulty is that signing Nectar.com is a transaction heavy process that it maxing out the site's existing servers.
A notice on the site explains: "We are well on the way to becoming the biggest rewards programme in the UK and are currently experiencing very high volumes of traffic on the Nectar website. As a result you may experience difficulty in accessing our site. If you wish to register for the Nectar programme, please complete your registration form and post it to us in the envelope provided."
A spokeswoman for Loyalty Management UK said that Nectar is putting in additional servers to cope with extra demand, and expressed the hope that the site will be available later today.
She didn't known the platform on which Nectar.com runs but Netcraft reports that the site uses a Netscape-Enterprise/4.1 on Solaris 8 front end.
By signing for Nectar on-line, card holders get a bonus 100 points (worth 50p), and avoid the hassle of either ringing an 0870 number to register or using snailmail. This incentive, together with higher than expected early demand for the programme, sent Nectar titsup.com.
Perhaps the lack of Web access is a blessing in disguise. A Reg reader told us he was able to see another user's details when he managed, after a long struggle, to register on the site late last night.
Nectar is aware of the problem but describes it as an isolated incident, adding that it putting measures in place to prevent a repetition of the security breach.
Over six million registration packs are already with consumers and by the end of the week, Loyalty Management UK estimates over 10 million cards will be in circulation - enough to enrol more than 40 per cent of UK households.
Nectar has proved popular because it allows consumers to accumulate points at the four participating outlets and redeem credits against a wide variety of goods not just at those outlets but for other affiliates (such as Virgin, British Midland and Eurostar) as well. ®

2002: Don't meet your ancestors
1901 Census site closed for urgent repairs

By Tim Richardson Posted in Music and Media, 3rd January 2002 15:20 GMT
The UK's 1901 Census Web site that's been jammed with users since its launch yesterday has been taken off-line for some urgent maintenance.
At one point yesterday more than 1.2 million people were trying to access the site simultaneously far exceeding the site's day-to-day capacity.
Although the Public Records Office (PRO) expected the site to be popular it's been overwhelmed by the public's response and is taking measures to try and resolve some of the problems.
The site was pulled down at around 2.00pm GMT this afternoon and work is expected to last for around two hours.
A PRO spokesman told The Register that this should improve the site's performance and enable people to trace their ancestors who were alive at the turn of the twentieth century.
However, it seems the best advice is just to wait until all the fuss has died down and maybe take a look in a week or so.
The 1901 Census for England and Wales was taken on 31 March 1901 and contains the details of more than 32 million people.
Not only does the site allow Net users to records search by name and place, it also allows them to search by vessel. That's because more than 70,000 people were counted on merchant sea-going and coasting boats, inland barges and boats as well as naval vessels when the census was taken.
The site was created by the PRO and QinetiQ (formerly part of the Government's Defence Evaluation and Research Agency [DERA]). ®

2004: Christmas Spectacular

By Tim Richardson • Posted in Financial News, 3rd December 2004 10:11 GMTMarks & Spencer extends its "One Day Christmas Spectacular" offer into a second day after its website went belly up during its pre-Xmas sale.
The struggling high street retailer had tried to drum up trade with a 20 per cent-off bonanza in its stores and on its website yesterday. But so many people responded online yesterday, the M&S site fell over for several hours during the middle of the day.
As a result, the retailer has now extended its offer until this afternoon to appease frustrated shoppers.
A statement on the M&S web site reads: "For all online customers who may have had problems shopping online Thursday 2 December, we've extended our Christmas Spectacular Offers exclusively online until 2pm Friday 3 December." ®

2006: Gift-card shoppers overload iTunes Web site

The Associated Press Published: 12.29.2006
Swarms of online shoppers armed with new iPods and iTunes gift cards apparently overwhelmed Apple's iTunes music store over the holiday, prompting error messages and slowdowns of 20 minutes or more for downloads of a single song.
Frazzled users began posting urgent help messages Monday and Tuesday on Apple's technical forum for iTunes, complaining they were either not allowed into the store or were told the system couldn't process their request to download songs and videos.
It was not immediately clear how many people were affected by the slowdowns, and Apple Computer Inc. would not immediately comment Wednesday on what caused the slowdown and whether it had been fixed.
Analysts said the problems likely were the result of too many people with holiday iPods and iTunes gift cards trying to access the site at once.
Traffic indeed was heavy over the holiday, with more than four times as many people visiting the iTunes Web site on Christmas than at the same time last year, online market researcher Hitwise said Wednesday.
Some financial analysts said the interruption could be viewed as a sign that sales dramatically exceeded the Cupertino-based company's own forecasts.
"It's actually created more positive buzz among analysts. Traffic was so great it blew up the site," said Gene Munster, senior research analyst at Piper Jaffray. "If anything, it could be a positive. Demand was better than they were expecting."
Apple commands about 75 percent of the market for downloaded music, but could lose as much as 5 percent of that market share in 2007 because of increased competition from rival services, according to Piper Jaffray.
Dan Frakes, a senior editor at Macworld magazine and playlistmag.com, a Web site focused on digital music, said he and some colleagues were unable to access the iTunes store or received error messages when they tried to download songs.
However, others breezed through the process hassle-free, and Frakes successfully downloaded songs again on Wednesday. He said the problem likely was not as widespread as the discussion group chatter might indicate.
Analysts said they didn't anticipate a rash of iPod returns because of the delays.
Apple's stock price fell almost 5 percent Wednesday before rebounding to close up a penny at $81.52 on the Nasdaq Stock Market, then fell 65 cents to $80.87 at Thursday's close.

2007: Led Zeppelin reunion opens with Communication Breakdown
Website goes down like a...oh

By Chris Williams • Posted in Servers, 13th September 2007 13:12 GMT
The clamour for tickets for Led Zeppelin's reunion gig at the O2 in London in November has overloaded the registration website, frustrating thousands.
It's been down all morning and at time of writing we can't access the site. Organisers are appealing for patience and say fans have until midday on Monday to be in with a shot of a ticket.
The site registers would-be rockers in a lottery for the right to buy a £125 ticket. Led heads have been warned that eBay touts do not have tickets to sell.
The BBC reports that the organisers reckon 20 million people have tried to buy tickets. It seems more likely that 20 million attempts have been made to access it, but either way, the rush was fairly predictable.
The show at the former Millennium Dome will see John Bonham's son, Jason, sitting in on drums, and also features The Who's Pete Townshend. It has been organised as a tribute to Atlantic Records' co-founder Ahmet Ertegun, who died after a fall at a Rolling Stones concert last year. Profits will go to fund scholarships in the US and UK, and Ertegun's native Turkey. ®

2008: Debenhams

Debenhams web site, already known for availability issues during sales times is again unavailable over the Xmas sale period; other site such as Next used queuing systems to throttle the traffic to their sites and remained open for business.

2009: Obama’s Crash J.Crew web site.
During the inauguration of Barack Obama some viewers paid more attention to Michelle Obama's wife's attire it seems. So taken were they with the Gloves of the new first lady which were made by J Crew that they flocked en-mass to the ladies-wear section of the site.
By Tuesday afternoon the page that featured the gloves in question was unavailable, by Wednesday morning the whole women's section was down with a message saying: "Stay tuned…Sorry, we’re experiencing some technical difficulties right now (even the best sites aren’t perfect). Check back with us in a little while.".

It's an interesting story - there is a whole page about in the new york times here.

2009: 'Best Job In The World' Web Site Overloaded

Morning Edition, January 14, 2009
Morning Edition reported Tuesday that tourism officials in Queensland, Australia, are looking for an island caretaker. They bill it as the "Best Job in the World." The job involves swimming, snorkeling, strolling around the islands of the Great Barrier Reef and blogging about it. The salary for the six-month job is about $100,000. As more news organizations began reporting the story, applicants overloaded the Web site.

2011: Police Crime Site overloaded
AOL News, 2nd February 2011 by Hugh Collins - contributor
A British website showing block-by-block crime statistics crashed within a few hours of going online today after public interest overwhelmed servers.
The site,police.uk, crashed after receiving as many as 300,000 hits a minute, or 18 million an hour, The Press Association reported."Most popular gov website ever?" the British Home Office wrote on Twitter. "Demand for new #crime maps at around 300k a minute, equivalent to 18m hits an hr. Working hard to make sure everyone can access."
The site is intended to allow people to gauge levels of crime and police activity in neighborhoods in England and Wales. It breaks down crime into six categories: burglary, robbery, vehicle crime, violence, other crime and anti-social behavior,BBC News said.
When AOL News tried to access the site this afternoon, it displayed an error message, saying the site may be overloaded or down for maintenance.
There is no equivalent national map in the United States, according to Maggie McCullough, who heads up the Policy Map project at the nonprofit The Reinvestment Fund. The Policy Map project maps information such as crime statistics and mass transit access for different U.S. cities.
Individual U.S. cities including Chicago and Los Angeles currently offer crime statistics on a street-by-street level."Safety is the No. 1 thing," McCullough told AOL News. "People want to know what crime is happening in their neighborhood."
U.K. Policing and Criminal Justice Minister Nick Herbert specifically cited the Los Angeles example as a factor in creating the map of England and Wales. "Police.uk will make England and Wales world leaders in this field, with every citizen able to access details about crimes on their streets," Herbert said.
Not everybody is happy about the site. When it was first published, it showed Surrey Street in Portsmouth, on the south coast of England, as one of the most violent streets in the country, with 136 crimes in December alone.
In reality, the street is less than 350 feet long, and locals are amazed to hear they live in a crime hotspot.
"These maps are an utter joke. This is a quiet road tucked away and anyone can tell it's hardly Beirut," local Scott Mussen said, according to The Daily Mail.
Still, McCullough said that such maps do a lot of good, helping individuals and authorities make better-informed decisions.
"We need information to make decisions," McCullough said. "The public having information shouldn't be a bad thing. "

No comments:

Post a Comment