A few months ago I faced a dilemma: I had planned hours of user interviews as a part of my research and design process (mental modeling) but hadn’t accounted for the time and effort it would take to transcribe the audio to text.

I started researching options that were economical and fast and came upon Andy Baio’s post about his experience. It was encouraging and I decided to give Amazon Mechanical Turk (MTurk, for short) a try.

I started with five audio files, each 45-60 minutes long. I took Andy’s template for a HIT (a unit of work on MTurk), modified it, paid the money, and sent them off into the ether. I wasn’t sure what to price the things, so I started with $10. I allocated a few hours of time to work on this. (In retrospect, I admit this is a really crappy rate.)

By the next day I started seeing that my HITs were being fulfilled, but I also started getting email from workers and potential workers:

One simply said my price was too low for the amount of work. Fair.
Two people sent me emails saying they could do the work, but never accepted the HITs. I took this to be the equivalent of a potential eBay buyer sending an email and saying, “I’ll give you $x for this!” I told these people to accept the hits.

Once the HITs were accepted, I got feedback from the workers. Mostly, the time turned out to be a limitation for them - I severely underestimated the amount of time it would take to transcribe this audio. However, all but one worker completed the transcriptions with time extensions.

Regarding the quality of work:

I specified a format in the HIT, and only one person followed it. Others used the names of the interviewer (me) and the interviewee. I did not reject these, and should have.
While Andy’s template provided a textarea for the completed transcription, only one person used this. All others found ways of contacting me on the web and did it that way. (The audio files were hosted at my personal domain, so finding me was perhaps too easy.)
There were definite glitches with all of the transcriptions, either in character formatting or the actual transcription (typos, missing words.) However, there were also issues on my end with the audio files - some were echo-y, and thus, I take the blame for this.

Would I use MTurk for this again? No. While the allure of having a low-cost transcription service is strong, the quality of work and the sheer pain of the HIT management process is just too time-consuming for me. While one might be able to get better results with a more reasonable reward and timeframe, I’m not willing to take that chance.