In my work on Games in Learning, I was wrestling with how to properly organize all of the various terms I had been encountering (this very problem is the core of my upcoming publication), and so I decided to do a thought exercise. I printed all of the different terms' definitions onto slips of paper, and then I sorted them based on similarity at a semantic level.
Definitions that seemed to describe the same principles and ideas would get grouped together. Once I had sorted them all (50 definitions in total across 13 terms!), I looked at how the terms were distributed. Not terribly surprisingly, I found that there was often a lot of breakup of a term across multiple different groupings. What this meant, at an anecdotal level, was that the definitions for any given term were so disparate from each other that they were more like definitions from other terms than definitions of the same term. A real canary in the coal mine, if you will (it's worth noting that miners actually made sure to rescue and revive their canaries, they were sorta like the miners' pets).
I had already been writing a rather semantics and philosophy heavy paper about the problem that resulted from there being all these terms and definitions, and I decided that I could possibly use my thought experience to gather some data for this otherwise dataless paper. So I made a few more sets of definition slips, and came up with a general instructions and procedure for other people to do this same thing. I knew that if I collected other people's intuitive semantic sorting, I could use it somehow to show that anyone who stumbles into my field from off the street would be confused the same way I was at the start, because the whole field is sorta a mess.
So I had these stacks of slips, and an instruction sheet, and I thought to myself, with a moment of dread, "Is this anything? Does this have any value as data?". So I hopped into my literature search, and much faster than I had expected, I came across the method of card sorting (a fairly broad method that encompasses lots of different types of methods that involve sorting cards into piles following various metrics). And lo and behold, the exact way I had done it in my thought exercise, and had begone to do with colleagues within the department, was precisely the same as one of the types of card sorts (repeated single-criterion card sort). I was rather pleased at my luck, that I'd independently reinvented this method, and had done it just like it's done elsewhere. So I started my initial data collection, and filled out my IRBs, and decided to start plugging my data into excel (I'm doing all of the data collection, entry, and analysis by hand due to a lack of budget for the expensive programs that could do it for me). And as I started filling out my first 50x50 matrix, I panicked.
I had in my head an idea of how to do this, of what it would look like, of how I'd stack all of the matrices I'd collect on top of each other to get final values, and then do some sort of clustering, but I didn't really know exactly how that'd work. I had a picture in my head, but nothing on paper.
So I hit the literature again. And again, I found that the exact way I was planning on doing my data analysis was exactly how lots of other card sorts had been analyzed. The more I dig, the more variations I can try out to see how they'd affect my final results, but yet again I'd reinvented the wheel, and not only that but the wheel I'd made was a nice circular one, not a clunky octagon that'd need some honing.
I got lucky. I got lucky that I independently invented card sorting, and figured I'd analyze it the same way that everyone else analyzes it. I'd like to say that it's not because I'm smart, but because the method is intuitive, but I have no way of gauging that. But what I am glad, is that even though card sorting has been around for a few decades, that it is a robust method with a healthy background, and that it lets me gather data for research that is otherwise mostly just philosophy, logic, and semantics.