In a research project for my current employer, Dependable Systems Lab, a graduate student and I investigated what makes a configuration interface error-prone. We interviewed users and observed them to identify errors they make when setting file permissions in the Windows XP file permissions interface. We classified the results of our observations by user experience and Norman's human error taxonomy, established design goals to alleviate the errors, and created an alternate UI. We then designed a scenario-based think-aloud to compare user errors across classes of errors. We captured the screen and the user's speech (think-aloud) in addition to task completion scores. We coded the users' behavior using a detailed version of Norman's error taxonomy to document what errors they made and why. We had 24 users in the pilot stage and another 24 in the final experiment. This work was reported in the following publications: http://portal.acm.org/citation.cfm?
id=1090415&dl=ACM&coll=&CFID=15151515&CFTOKEN=6184618 http://www.cs.cmu.edu/~maxion/pubs/ReederMaxionDSN05.pdf http://www.cs.cmu.edu/~maxion/pubs/ReederMaxion06.pdf