Paul McAleer

Someone is Wrong on the Internet: Jay Greene

Paul McAleer

Jakob Nielsen recently released his findings on usability testing with Windows 8 and they were brutal. Just brutal. He uncovered lots of issues with the new OS.

So I was intrigued when I found an article by Jay Greene at cnet about how Nielsen is wrong in this case. In fact, the headline calls him out for “old-school thinking”.

Why Jakob Nielsen’s Windows 8 critique is old-school thinking

The usability expert branded Microsoft’s new operating system “disappointing.” But his approach would lead to the kind of incremental improvements that have dogged Microsoft for years.

Not the most promising start. But let’s dive in.

Apple co-founder Steve Jobs was once asked what market research went into the creation of the iPad.

“None,” Jobs replied, in one of his most celebrated quotes. “It’s not the consumers’ job to know what they want.”

I was reminded of that when I read usability guru Jakob Nielsen’s recent criticism of Windows 8.

Problem: usability testing is not market research. Market research tends to cover more opinions about existing things, or potential things, than actual task-based usability testing. Usability testing is design research, and while I’m presuming there’s a bit of opinion that comes out in Nielsen’s usability testing… task-based testing isn’t a game of, “Huh, I like this in blue!”


Usability testing is designed to answer the question, “Does this work for a person?” That’s it. Opinions, when gathered, may be valuable but that’s the land of focus groups. Nielsen wasn’t asking customers what they want. He was asking customers to use something. Huge difference.

I’ll leave it to others to decide whether 12 experienced PC users is a large enough sample size to gain meaningful insight. 

Someone had research on that. Hm. Wonder who?

The problem with watching people try any product that’s undergone a significant change is that they will struggle. It’s human nature.

No disagreement here, but Greene seems to dismiss Nielsen’s work just a bit, or perhaps devalue it, because it’s something new. But this is precisely why you want to get something like this in front of users: it’s a big change. How will they react? Can they still get their work done? Can they do anything at all? It’s not a simple, “Eh, people will struggle.” We know that. We seek to know what they’re struggling with, and then we can (if we choose) make it less of a struggle.

All right. So now we get to the quote that drove the headline.

“It’s a very old-school approach to usability,” said Tom Hobbs, the creative director at the Seattle design firm Teague and an interaction design specialist. ”For Microsoft to support incremental change around a new paradigm would be suicide,” Hobbs said.

Old-school! Okay. Hm. So, incremental change. So did Greene talk with Nielsen about that?

“Every time you do incremental change, you acquire crud,” Nielsen said. “At some point in time, you’ve got to make a break from the past,” Nielsen said. “But you can’t do it very often. That’s why it’s so important to make sure you get it right.”

Say, doesn’t that directly conflict with the entire premise of the article? Here is Nielsen admitting that incremental change is problematic. “Crud” is right. And yet, if you simply put these quotes up against Hobbs’s, you can see that there’s actually no disagreement here. At all. Nielsen is actually not advocating an old-school approach! He’s instead saying that if you’re going to make a large design change, you’ve got to make sure it’s right.

The disagreement point is over “getting it right”. But I feel that point is not even explored in this piece. Nielsen’s got data that says Windows 8 is tough on users. And while Microsoft conducted usability testing (and presumably boatloads of other insight gathering), it’s in the realm of possibility that this testing data was used, partially used, or fully ignored. We don’t know. We might never know.

“I don’t think [Microsoft has] got it all right by any sense,” Hobbs said. “But I’d argue that Microsoft is doing some of the most interesting user interface work on the planet.”

Interfaces can be subjectively interesting and be objectively unusable.

People can differ on whether they think the new Windows 8 user interface is worthy. But, for a change, Microsoft seems to have a distinct point of view.

Totally don’t get this ending. It feels like a soft, “Well, I guess we’ll just have to agree to disagree!” stance. And that’s unfortunate. There’s an interesting article waiting to be written: Microsoft, a very large company, has conducted extensive testing on this new product. Jakob Nielsen, a well-respected usability expert, has conducted testing raising serious concerns about this new product.

This is not that article.