The MLS website panic switch

2010-04-14 13:18:00

when you see yourself in a crowded room
do your fingers itch are you pistol whipped?
will you step in line or release the glitch?
can you fall asleep with a panic switch?

-Silversun Pickups

Don Garber graced Grant Wahl with an interview last week and the story ran a couple days ago. You know the times are far from ordinary when I will set aside the talk of single entity and expansion to focus on what Garber said on page two in reference to the league's website.

When I posted the first followup to my original screed, I pointed out how MLS might respond to the widespread public unhappiness. I was pretty much right in all the places it counts, but I'm pleased to say that we got a lot more detail than I expected. Not only from Garber, but Greg Lalas got called on the carpet by BA Duane and Ben Rycroft last week on their podcast. So kudos to MLS for trying to get out in front of this a week or two after it all broke open.

Since Garber and Lalas aren't technical guys by trade and at least the former has never claimed to be one, I'm going to try real hard not to not rip them for any kind of technical gaffes. They're certainly not the ones out there coding, they're the damage control guys. Garber:

The challenges of the launch were all related to one very specific thing. We had a code that was developed that was managing a plethora of data streams that were coming in from hundreds of different places going down a single pipe coded and then sent out to the entire Web site. Five minutes after the site launched, that code got a bug and is broken. And it's still broken today.

"We had a code." Not "some code" or "code," but "a code". You know, if it were in the present tense, that's almost like a ah, fuck it here it is:

It's too late for me, save yourselves. And I'm not even going to start comparing the data streams going down "a single pipe" to the whole series of tubes thing, ah fuck it, Greg Lalas already did it, 17:20 in the podcast:

Not to make fun of the American senator who thought the internet being a bunch of tubes, but it seems like the internet is a bunch of tubes, and some of the stuff isn't getting into the right tubes the way it should.

You're. Not. Helping. Work with me.

Look, what's important is the plausibility of Garber's statement - that the web site issues spawned from the bit of code that aggregates game and stat data, code which worked before the system was turned on, but failed on site start up and is still broken. This statement we can evaluate, and even if Garber or Lalas don't understand exactly what's going on with the stats or aren't sharing all they know, we can try to figure it out on our own. And it's especially important that we do evaluate it, because it's been the party line going back to Chris Schlosser's original apology.

For what it's worth, I think the problem's existence speaks to bigger issues in the project. But since updating the stats is probably the most pressing technical issue that needs to be resolved, and has probably resulted in the most complaints, let's see what that fix might involve.

Elias Sport Bureau is the official statistician of MLS, so after I read Garber's interview I started looking at how Elias handled things with stats and game data. The fact that some random guy has documented a streaming game data API (for the non-techies, an API is a method to access data programmatically) for Major League Baseball, another sport handled by Elias, clued me into the fact that there was probably game data sitting somewhere on the MLS website.

I beat my head against the wall for a while trying to find the feeds before coming across a book that reminded me how easy they should be to identify. Baseball Hacks, a book put out by tech company O'Reilly Media, had what I needed in Hack 26, "Discover How Live Score Applications Work":

Today, most live score applications simply get data through HTTP requests to web servers. This is the same way your web browser retrieves web pages from the Internet. This makes it very easy for the baseball hacker to get what he wants from those servers.

After reading that I started poking around the DC-Philly Matchcenter page from the weekend, and within minutes had found the stat feeds for those games. As far as I can see, there's a roster feed, game tracker feed, play-by-play feed, and a very small final stats feed.

Inside the feeds are comma and bracket delineated lists of data relating to the game. From the play-by-play feed:

{
"id":"2",
"text":"-3rd Minute. Shot on Goal by 20-Roger Torres, 1st of match. Save by: 23-Troy Perkins, 1st of match.",
"time":"02:50",
"type":"Shot",
"team_abbr":"PHI",
"team_name":"Philadelphia Union",
"player_id":"STOR588828",
"x_cord":"0.0",
"y_cord":"0.0",
"player_name":"Roger Torres",
"player_number":"20"
}

What this means is that stats for the 2010 MLS season are currently sitting on the MLSsoccer website in a raw, unrefined form. The first weekend is hit or miss - only a couple games have the feeds. But starting with the April 1st games, it looks like they're all there. Given that Fox appears to have working stats, undoubtedly obtained from Elias as well, I suspect the problem that accounts for the missing information is internal to MLS.

The data that exists can be accessed programmatically or manually by anyone who can follow the naming convention of "http://www.mlssoccer.com/matchcenter/feed/game_tracker/MMDDYYYY_VTHT". If you were talented, you might be able to whip up a nicer interface than MLS has, although you might want to check out the MLS website terms of service and think long and hard before you shared that with anybody.

Allow me to pause for a moment and ruminate on how wonderful it would be for MLS to provide an open, supported API for their live game and historical data to anyone who wanted to aggregate it and package it in their own way. It would be a great way to leverage the technology already at hand and facilitate the spread of MLS-related information among the grass roots. I suspect what's more likely is that I'll be branded an anarchist hacker for suggesting something that would take the league message out of MLS hands and put a moderately bigger load and bandwidth burden on their server. Elias might not be happy, either.

At any rate, we know 2010 season stats are there and probably don't amount to more than a few megabytes of data. So far there's probably around 100 feeds that need to be aggregated, plus possibly player pictures and bios. Taking that data, dumping it into a database to sync with the prior seasons, and then spitting it back onto the standings, player profile and stats pages should not really be a big problem. I wouldn't be surprised if it needed nothing more than a script running at a predefined intervals or in response to a tickler on all the data feeds. It is certainly not a millions-of-dollars-budget, three-weeks-after-it-got-out-the-door-it-still-doesn't-work kind of thing. Especially when it looks like a lot of the formatting on the front end has already been done.

They'd really have to start totally over with a new implementation for it to take this long. And even then a good coder could probably push out an intermediate fix sooner than that.

Not only that, but Garber, Lalas and Schlosser have all said the code worked before launch and that they expected it to work after launch. If the whole website relied on this code, and MLS was purportedly aware of the issue within 5 minutes of launch, why was the problem not dealt with immediately? In my experience, the turnaround time on something like this is minutes, hours, or in extreme cases, days - not weeks. And if it was working before the launch, why was the person who wrote it fired before they could fix it on production?

So something does not add up for me. The three week long broken state of the live statistics seems to indicate deeper, more structural issues than just a problem with the code that syncs the new data. Maybe it has to do with the people working on the project, maybe the underlying system architecture is tying some hands. I really don't know. But what I can say with reasonable certainty is the broken code that's sitting out there somewhere is a sign of the overall project problems, and not a root cause.

What it does appear to be the cause of is more bad decisions among the MLS and/or Rocketfuel teams. For example, earlier this week the stats and player pages disappeared, and the main site navigation links to those pages sat broken for a couple days before the links themselves were removed. Then a couple days later, coincidentally after I expressed my opinion on Twitter that such a move showed how far things were gone, the pages slunk back up, still apparently broken.

What kind of decision making process is in place for something like that to happen? Who made the call to pull the pages down, only to put them back up? Or did the pages break on their own and no one on the dev team noticed for a day or so? I'm willing to entertain the idea that the move was planned in the process of fixing the site, but the broken nav links, the two day turnaround time, and the continued lack of populated data give the appearance of a web team in total turmoil, a team running on panic and fear since the switch was flipped and community opinion turned completely against them.

So for me, the broken live data code is only a small, but significant, part of the overall problem, which is this: The new website was poorly planned, poorly tested, and poorly implemented. The broken code does not account for the numerous markup and css issues, the lack of quality control on the other content that pre-populated the site (news stories and photo galleries), random pages dropping off the site, the performance problems on the web server itself (in testing for the data feeds, I discovered that as many as a third to half of my http requests to mlssoccer.com are being dropped), the over-reliance on Silverlight for the streaming video, the poor layout design of most pages, or the failure to have fixed the most important problems a full three weeks after launch.

The disclosure that people are updating things manually is an indication that Drupal, the content management system, might not be working, either. On one of the news tabs on the home page yesterday, a headline inexplicably appeared next to the "Latest News" tab, an indication that humans are editing the raw HTML on the home page and making mistakes. The mood can't be pretty back there right now. It's as if nothing is working at all. At least today the search engine stopped saying "Results not found" when it would in fact return results.

Don Garber's public admission that there was no backup plan if things did go wrong is shocking. The term "gross professional negligence" of the implementing team comes to mind. Greg Lalas saying that his only regret was not being able to yell at the development team more effectively betrays a total lack of technical understanding in the executive team that continues to show itself as the new website froths and convulses on the floor.

On the topic of accountability and failed leadership, Garber had this to say:

The answer to solving problems is not rolling heads. The answer is get it right, figure out what's the best way to manage this going forward, see if the systems we have in place are proper, that the technology is right. And we will continue to analyze as we have even if we didn't have this issue. For anybody who has ever run a business, the last think you do is start firing people when you have problems. That's the worst kind of leader in crises. The best kind of leaders in crises are empowering those people who are devastated by where they are to try to give them the opportunity to lift up, solve their problems and be able to use this as a learning experience so it doesn't happen again.

While that last bit clearly didn't happen for the code developer, it's apparently happening for MLS Director of Digital Media Chris Schlosser. I'd like to think the opportunity to solve problems means Schlosser is on notice to fix this mess or he's back to being an ad salesman, but it's probably wishful thinking on my part.

To be fair the situation is not entirely Schlosser's fault. The guy who, as many commenters to this blog have pointed out, hasn't yet correctly entered the URL on his twitter home page to point at a vacant smugmug site should never have been handed the keys to the MLS web project. When Garber talks about figuring out the best way to manage this going forward, that should have been done a year ago, and it should have been done better than it was. MLS can not continue to "analyze as we have," because that's what got it into this mess. Radical cultural change needs to take place at many levels.

And no matter how much pity I feel for a management team totally unprepared for the task at hand, they absolutely need to be called out and held accountable for the failures in planning and their lack of understanding. Garber shouldn't know how to code stuff, but he should know enough to select a person to head the project who has the actual tools to successfully carry it off. It is a failure of leadership to not recognize when the people put in charge of projects are unequipped to lead those projects. It is a failure of leadership when arbitrary deadlines are insisted upon in the absence of a backup plan in the event of failure. And it would be a failure of leadership to not put people in positions where they able to succeed after ample evidence that they are not currently in those positions.

The time to chop or reassign people, if that time does come, is not now. MLS needs to extinguish fires that are still burning before it can sort itself out. Ultimately someone with a more technical background and deeper understanding of the issues involved needs to be made Director of Digital Strategy. Maybe Schlosser can figure things out, but I think he'll be lucky to still be with MLS in a year's time.

******

Epilogue

When researching this post, I took a trip down memory lane and was reminded of just how bad MLS has traditionally been at web sites. What we have here is nothing new. The site from around 2001-2002 wasn't exactly great even though pinstripes were hot that year. In fact it was so bad that the launch of the new MLSNet in 2004 (i.e., the site that was just abandoned) was greeted with much enthusiasm even though at launch it was missing a lot of the same features as MLSsoccer. And while the sheen eventually came off, it was functional enough that most people could get by.

I think there were two key differences between then and now. First, the old MLSNet worked quickly to restore the missing data. It didn't sit around for weeks frothing at the mouth and spewing Euro transfer rumors. It got down to the business of reporting on MLS, and while it could have been faster and better, it was good enough.

Second, the level of expectation is different. Let's not forget that in 2003 fan-run sites were the ones providing the streaming Superdraft coverage. It's a different world in 2010, many promises were made, and a lot of fans expected MLS to actually deliver. The semi-positive response to the last redesign in 2004 may have actually helped lull MLS management into a sense of complacency with the current redesign.

It is interesting to note that in 2004 MLS was bringing all the websites under one banner, and now it's bringing the technical portion in house, while opening up content creation to the teams. No matter what they do, they struggle to get it right. This web ineptitude didn't happen overnight, and it won't go away without a hard look in the mirror and an influx of people who know what they are doing.

A closing note to the MLS people who are reading this: tell John or whoever does it to go ahead and close the secure port on MLSNet. It's still indexed on Google, and now that I know how the live feeds work, I don't think I'll need it to get to the old site anymore. At this point all it does is make me sad. Thanks.

-FS

Go to the homepage, browse the archive by month, topic, league, international and college competition, or explore relevant keywords: