We test with different OS/browsers. But the main advantage of Panera’s (restaurant) testing is getting users on their computer.

This surfaces plenty of bugs that are unlikely to be discovered with “QA testing.” Watching how people react and recover puts an entirely different perspective on which “bugs” are important. We find far more QA problems in an hour compared to days of brain dead “Does every feature work on XP/Firefox? Yes, Vista/IE, yes…”.

Also you find a lot of problems that people won’t report because the situation made them feel stupid or somehow responsible.

Use testing is more realistic than a lab. The user is completely comfortable with their machine. That means testing is on our software in a real-world environment. They don’t stumble, “where is the delete key”, “how do I open the browser again”, “I hate this mouse”, “where did I save that download?”

You can see if they find our shortcut and tray icons amid a cluttered desktop (as compared to a sparkling clean Vista install).

You see problems like the Skype IE “phone number dialer” screws up layout (no one would report that).

All kinds of surprises come up.

Most important for our product, we see how users react to the software that depends on browser and media players installed. A big part of the test “that we consistently flunk” is helping them get the machine ready.

We have tested on our machines. My experience is developers especially don’t like the approach above because it’s not focused and there are too many variables. Precisely. The same was even true with the last design/UX team I had. They wanted to focus on “how does this look in Safari”, and tightly controlling testing of a use case. I agree there is a role for that. But what I found is our main problems have been outside of an unrealistic starting point “OK from this screen how would you LISTEN to the program.” “they pressed listen, it worked!”

Another use testing trick that worked great for us was recruiting from our building. We setup a table in the lobby with coffee and donuts. Then we had a schedule form and offered $25 for 30 minutes. Everyone was in the building, so it was an elevator ride for them. We had a controlled environment and the developers could watch (but most would find an excuse not to). About 50% are no-show.

We’ve not done any of this in a year and I plan to with the site updates. It takes some time but is priceless.

Why worry about having the users test in their own environment (PC)? The focus of the test is on usability. Testing to see if the site works properly across different systems should be part of QA work and can be done separately.