TQR — Sorting Out Card Sorting

Steven Hannah’s Sorting Out Card Sorting isn’t real­ly about card sort­ing. It’s an exam­ple of using a par­tic­u­lar method­oloy to do aca­d­e­m­ic lit­er­a­ture review and then a pro­pos­al for cre­at­ing a tool that can be used to aug­ment and extend the knowl­edge dis­cov­ered dur­ing the review.

But even if the phras­es ground­ed the­o­ry and con­stant com­par­a­tive method make you feel a bit woozy it’s worth hav­ing a look at Chapter 4: Analyzing the Data. Hannah dis­cov­ers twelve char­ac­ter­is­tics that are used to describe card sort exer­cis­es which rough­ly reflect how card sort­ing is done and discussed.

These char­ac­ter­is­tics make a decent basis for a “things you need to con­sid­er while plan­ning your card sort­ing exer­cis­es” check list. I’ve pruned a cou­ple of redun­dant items and rearranged the rest into a use­ful order for planning.

  1. Define the infor­ma­tion domain
  2. Select a tar­get audience
  3. Choose between open and closed sort
  4. Choose between indi­vid­ual or group exercises
  5. Choose a num­ber of cards to be used
  6. Select the objects to be used on the cards
  7. Set a time lim­it for the exercise
  8. Choose analy­sis methods.

Two things emerge as con­sis­tent recommendations:

  • Investigators should pro­vide a lit­tle guid­ance to the test sub­jects as pos­si­ble. Only what is nec­es­sary to get peo­ple pil­ing the cards up.
  • Card sorts should be lim­it­ed to less than 100 objects (cards), or what ever will take less than one hour.

The ques­tion of how to ana­lyze the results of the user sorts is, well… murky. The usu­al dichoto­my of qual­i­ta­tive vs., quan­ti­ta­tive exists here as every where. The dif­fi­cul­ty is that while design­ers (and oth­ers of the edi­to­r­i­al bent — this by JJG is food for thought on the sub­ject) are will­ing to “eye­ball” the con­clu­sions there are oth­ers who want some­thing more rig­or­ous. They jus­ti­fi­ably point out that the mass­es of data brought to light by card sorts over large sets or the responces of of many par­tic­i­pants can yield results that are too large to wrap one’s mind around.

Surely quan­ti­ta­tive analy­sis is a good thing — espe­cial­ly if there is a per­ceived need for “facts and fig­ures” to jus­ti­fy design deci­sions. There is how­ev­er as of yet lit­tle infor­ma­tion on the spe­cif­ic types of quan­ti­ta­tive analy­sis that are appro­pri­ate and how the results would be used in the design process.

One aspect of card sort­ing that is not con­sid­ered in Hannah’s sur­vey is the ques­tion of com­put­er aid­ed sort­ing exer­cis­es. This soft­ware did­n’t exist when most of the lit­er­a­ture sur­veyed in this paper was pub­lished. These tools exist now and whether or not to use them is an impor­tant test design question.

Oh… about the pre­sen­ta­tion on the web page. I have not been able to make it work. It won’t even attempt to load in Firefox. It tries to load under IE7/Vista but enters a loop and takes up all my band­width. I’d say that I’ll go try it on the lap­top with XP and IE6 but, you know that I won’t; it’s too much trou­ble. Too bad.