Siempre he creído el test debe estar dentro de un protocolo de cara a los clientes pero cuando es un proyecto propio o de la empresa en cuestión, el protocolo es la agilidad y el argumento las métricas. Cuando digo agilidad me refiero a la capacidad de hacer cambios a tiempo real con el menor coste posible.

Creo que la usabilidad viene dada en un 5% del prototipo y el otro 95% por el método. El cuándo me tengo que plantear las preguntas y cuándo tengo que hacer conclusiones.

Let me clarify some points here:

- I said it was a rant.
- I said “most” (users, u-tests, usability engineers) but never “all”, which leaves some space for guys hwho do and get it right.

I know the benefits of a good usability test, nothing that I haven’t done before (a few dozens, at least). There is always something to learn from them, yeah.

BUT:

- I get pissed off by those reports that only “point to problems” and don’t offer solutions (usually because the “usability engineer” doesn’t know shit about good design)

- I find it completely naïve (even offending) to test something like an iPhone in such a cold setup: ignoring user motivation and user context. It’s like picking me for testing the usability of some Bricolage-DIY tool: I just don’t give a damn about it, I know nothing, I have no clue about the whole thing and It’s not likely that I will ever own such tool.

- I fully agree with Astudillo when he says that user research is far more useful than usability testing. Right! It’s what gives you the good info and sets the proper constrains.

- I find that doing this u-test on the iPhone is totally oportunistic: some guys trying to catch the wave of something they would not even dream of designing themselves. You say I’m not humble, I say coming out of nowhere just to point these stupid mistakes on the iPhone (THE IPHONE MAN!!) is nothing but humble. It’s like the football-coach-wannabes that populate the Spanish bars criticizing all the decisions made by the big guys at the big clubs.

First-usage usability and trained-usage usability are two faces of the same coin. The first emphasizes effectivity, and the second emphasizes efficiency. To disregard any of them is detrimental to a product. To evaluate first-time usability is not everything, but it can be extremely useful sometimes, since many users may abandon an application if they have a bad first-usage experience. This is proportional to the intrinsical risk of abandonment of the task/product, and may not be the case in the iPhone, where most of the users are motivated enough to do trial-and-error until they become reasonably proficient with the interface. Still, I think you are overestimating the designer’s capacity to foresee first-usage difficulties from the drawing board, and underestimating their capacity to ignore some details that may lay beyond their attention or interests. As they said in American Beauty, “don’t underestimate the power of denial”.

At the end of the day, most of the usability issues unveiled by a u-testing could had been identified by a competent interaction designer, yes, but certainly not all. I confess I’ve never observed a u-testing that has not led me to look at one or two things I had overlooked as a designer. I guess this makes me fall into the category of “stupid”, but Iet me prefer the more politically correct term “humble” :-) Designers are not unlike developers: they have the right to make errors, and the duty to detect and correct them with the means at their disposal.

And the more the user profile is different from your own profile as a user, the more u-testing proves itself useful. This said, in cases where a project’s user reserach budget is too narrow, I usually tend to conserve pre-design user research at the expense of post-design u-testing. I guess this is because u-testing tends to unveil problems that are of a more universal nature and therefore better detectable by designers themselves, whereas good pre-design user research tends to give you a wealth of information about needs and expectations that, in my experience, challenge the designer’s preconceptions and lead to a much better-informed design.

As for a qualitative analysis using expressions such as “some users thought” instead of “some of the participants declared they had thought”, well, it’s not too rigorous, but aren’t you getting a little picky, man?

Pienso que te pones en un postura extrema hacia el otro lado. Una cosa es pensar que existe un curva de aprendizaje y otra aprovechar el impacto informativo que te puede dar un usuario en una prueba… claro, sólo te puede hablar de un uso en su carácter funcional o perceptivo (sobretodo en el “diseño”), pero no te muestra por supuesto lo que el usuario sí hace con el producto y cómo lo hace… no es lo mismo hacer la prueba que ver al usuario usando el producto.

Saludos.

Regarding your questions:
- I agree with Pi that first usage tests can help you with intuitive interfaces and general consumer products. First impression on an interface can lead you to stop using an application if it is not vital for you.
- Anecdotic remarks (“one participant said…”) helps exemplifying potential interaction problems (with no statiscal value, obviously).

And regarding your feelings I’d say:
a) most users do not like learning a new interface, they want to get the job done… and most of them are not computer savvy and have been doing ok ever since
b) i have not found statistical difference in stupidity among usability engineers than that found in other guilds like developers, graphic designers, cab drivers, university teachers or single mothers…
c) most u-tests tell mostly what you should have known from the start if you were a non-stupid usability engineer of sorts, but they also help you convince your customer to rework his overly complex interface, and sometimes they bring useful insights as a bonus

I do not like unconstructive ranting, so… thing is, I assume you are one of those non-stupid usability engineers too… what do you think that could have been a better example of u-report for this same issue? I still do not get your point…

But still… sometimes I think that all a) b) and c) premises you said are totally right. And then it’s when I start thinking about breaking with everything and growing tulips in Aalsmeer!!!