Jay’s Not Here

It’s been a while since I blogged about my study so here’s a quick update:

• Acquired a current-gen iPod Touch to give away to one randomly-selected participant.
• Signed up 3 people to come into the lab between now and Monday.
• Tentatively signed up 2 more over the next few weeks.
• Started writing tools to automatically collect statistics about previous sessions:
  • One participant saved their work 53 times per hour.
• Learned that a dual-computer setup makes video transcription much much faster. I’ve got one computer playing back the video on a 40″ monitor using MPlayer (awesome keyboard shortcuts for pausing and navigating video); the other one has an Excel spreadsheet with pre-labeled cells for storing the transcribed data. This setup saves me from having to Command-Tab between apps each time I want to type something. Plus, I get to see the video and the transcript at the same time.
• Discovered that using raw sugar instead of granulated increases the likelihood of caramelization during cookie baking.

I heard back from the ethics review board today about my study. They weren’t happy about how I used the word “remuneration” so I decided to look it up on Wikipedia.

Apparently, someone was able to sneak this in:

Renumeration can include:
…

• Eating boiled eggs to gain usage of supernatural powers

I had my first participant come into the lab for my first qualitative study. The session went fairly smooth except for a few wrinkles. I’m studying expert software developers—i.e. people who code to put food on the table. I learned quite a lot:

• An interruption-free environment lets a coder focus on their task so much they can lose track of time and let 3 hours pass. (Ahh, memories of undergrad assignments…)
• My idea of easy can be very, very different from someone else’s.
• Coders don’t like think-aloud experiments. They just want to do things their way.
• I need to make the task even shorter.
• Even though your screen recording software has a crash recovery feature, DO NOT TRUST IT!^[1]
• The last GO Bus going home leaves Union Station at 0040h. Not a good idea to miss that one, lest I want to spend the night downtown…

Now it’s time to go over all the data. Luckily I have all 3 hours worth of source code deltas, browser history, and IDE metadata, despite losing 2 hours worth of video. Some data is better than no data.

^[1]
I made the awful mistake of thinking my data was safe after recording the session. The program was in the middle of transcoding the capture data to a standard movie file and I stupidly decided to close the lid of my laptop to put it to sleep. Of course, when it woke up, the screen recorder crashed, but I knew it had a crash recovery feature that allows transcoding to resume. Well, it did continue, but the last 2h10 of the recording was all green screen. It only salvaged the first 50 minutes.

Now that I’m back in Toronto (though not quite on EST), I’ve had a bit of time to think over my research. Emerson was kind enough to review my study design and share his literature. Thanks very much for that!

One issue that arose from our discussion is my usage of the term “expert”. In research literature, expert programmers are generally those in industry who are paid to program. In all the software firms I’ve worked for, I’ve observed huge differences among these “experts” to the extent that I don’t feel right calling them all “experts”. I can’t call these “lesser experts” novices because that term refers to inexperienced programmers, such as most CS undergrads.

I ended up choosing a term used by Sonnentag’s study of software design expertise[1]: high and low performers. So now I can talk about high- and low-performing novices, as well as high- and low-performing experts.

My new problem is how to operationalize “performance”.

[1] S. Sonnentag. Expertise in professional software design: a process study. The Journal of applied psychology, 83(5):703–715, October 1998.

This morning, I made my final pitch at the conference to Thomas Setzer, a postdoc from TU München.

I walked over to Banana Leaf for a great 7 course Malaysian tasting menu for lunch. The highlights of the meal were the chicken in gulai sauce and the fried banana dessert.

The rest of the day, I went to Granville Island; tried the Taproom Taster, which is a 4-course beer sampler, got a screaming monkey, took the aquabus across False creek, watched the local dragonboat teams practice and gawked at the $1.5M waterfront condos. Yes, that’s half a million per bedroom. Ouch!

A quick summary of what I took away from the talks today:

Carlo Ghezzi: We should measure quality via the impact of our work

Emerson Murphy-Hill: Refactoring toolsmiths don’t refactor much; even less for normal people. (Ready, normal people? / Ready!–ready!–ready! / Let me hear it!)

Jorge Aranda: Software repositories don’t give us the full picture.

Steve: We are seriously hosed if we don’t do something drastic in the next 5-10 years.

Tonight’s reception was at the Vancouver Aquarium. I got to the sushi table too late so I missed out on that. I found it oddly disturbing to see people eating raw fish at the aquarium… Anyway, I saw the belugas, dolphins, frogs, caecelians, jellies, and of course, sashimi. Err, fish.

During the course of the evening, I pitched my research ideas Karl Reed, an academic gypsy from Australia who’s currently in Italy (I wish I had his job); Shivkumar Shivaji, an engineer at Yahoo!; and Adrian Kuhn, a PhD student from Berne, Switzerland. I felt humbled yet enthusiastic to be surrounded by so many smart and well-spoken people.

Serendipity struck once again. I was trying to recruit people to go to a little Japanese dessert bar prior to dinner. Somehow, Emerson Murphy-Hill ended up coming along–I’ve been keeping an eye out for him because Gail told me we had similar research interests. I had no idea it was him until I introduced myself after we ordered our parfaits. He’s currently at UBC studying super geniuses. I gave my pitch and it seems like our approaches are very similar. If we can arrange a three-way study replication with Chris Parnin, I think we’d have something definitely worth presenting at the next ICSE. Oh, the black sesame parfait I ordered was quite interesting. Mmm…

Finished off the evening at a pool side poster reception; roasted animals carved on tables, open bar and food being carried by wait staff on fancy trays–yes, that type of reception. They had porter cheese there! Beer, samosas, turkey, roast beef; first time I ate sushi standing up. Despite all the alcohol, I somehow did not end up falling into the pool as I toured the posters.

I started off the day attending talks given by Tony Hoare, John Cameron, Daniel Jackson (and later on in the afternoon, Michael Jackson himself). It was very interesting to see the interplay between big names in the field; and even more interesting to see Steve from our research group challenging their views.

I pitched my research ideas to Andrew Ko who gave me quite a bit of good feedback about how I should focus my efforts. I need to reflect on these further before I can write about it.

I went over to the legendary Tanpopo all-you-can-eat sushi restaurant on Denman for lunch. I haven’t been there for 11 years but the food was just as good as I remember. The layout of the place changed a bit. It seems like their kitchen is much bigger. Anyway, lunch went really well except for the final bit of awkward discussion. That’s all I’ll say about that.

On the way back to the conference, we had a bubble tea craving and decided to hike up Robson to Bubble World. They have a special machine that shakes the tea so much it forms lots of tiny bubbles resulting in a very creamy ethereal taste. Milky Taro tea with pearl–Mmm mmm mmm!

Sukanya taught me how to speak and read some Thai. All the interesting stuff at this conference really does happen outside of the regular program.

I bumped into Chris Parnin on my way to dinner and ended up chatting about the empirical research he’s doing. He shared his insights on the difficulties of collecting data during studies and strongly cautioned me to draw upon existing bodies of knowledge whenever I can. Our research interests are similar enough that we’re thinking about replicating each other’s studies and seeing what results we get. He also told me that the cognitive models of memory I’ve been using to frame my research questions are very outdated. He shared a number of pointers about current models (e.g. episodic memory) and how to find relevant literature on the subject. I found out he’s quite an egg.

At the IBM Jazz reception, I met another member of the UC Irvine group. Joel Ossher is studying how people look for code. From what I could gather, he wanted to be able to do a natural language query and get back the software artifacts that implement the answer to the query. For example:

Q: How do I make Python say ‘Hello world’?

A: print(’Hello world’)

Jon proposed a contextual help system based on mining and indexing various project artifacts. For example, the system would provide realtime links from within an IDE to related artifacts such as other bits of code, mailing list postings and BugZilla tickets. We think we can pull it off in 2 weeks and get Jon his thesis topic. Coincidentally, at the reception Apple Viryakattiyaporn presented a poster on very similar work she did recently at UBC.

I finished off the night with sweet potato fries, root beer and a bellini for dinner at White Spot. Seven days left.

I met Gail Murphy from UBC. She offered invaluable advice and insight into my research idea. She’s tried my approach before and strongly cautioned me against doing between-subjects experimental design because of the wide variance in my experimental unit (i.e. software practitioners). She also suggested I should use pair programming–with the less experienced of the pair driving–in my study to force the programmers to explicitly communicate their thoughts to one another as they complete their task. I’m thinking about planting a pseudo-novice as the one driving so that they can prompt their partner to fully explain their intentions.

On the bus ride to Grouse Mountain, I had an interesting chat with Gyeuk at ETRI about software security and attack vectors. It was really foggy tonight so we didn’t get a very good view outside the gondola. As we passed the two towers, our car rocked slowly back and forth. The feeling was somewhere between riding the subway during rush hour and The Bat at Canada’s Wonderland.

Grouse Mountain is a beautiful facility. Unfortunately, the fog robbed all 92 of us of all its grandeur. Open bar and 3 course dinner thousands of metres above sea level. Mmm…

On the bus trip back, I was talking to Daqing Hou, an ECE professor from Clarkson University in New York, about software practices involving code cloning. I found it interesting because code cloning occurs frequently within the projects in the Eclipse ecosystem due to API policies.

I had cheesecake for dessert paired with a Grouse Mountain Cream Ale. Mmm…

Tom Ball gave an inspiring history of version control tool evolution and its relation to the present field of mining software repositories. What surprised me is that in the mid 80’s, there existed tools for editing code across versions similar to how Apple’s Time Machine provides access to historical versions of files; of course, minus the fancy 3D animations.

I met Susan Sim and her student Sukanya Ratanotayanon–both from UC Irvine–at the ICPC poster session. Susan was presenting an interesting visualization of concept location activities as a graph similar to an EKG. It reminded me of body position graphs used by polysomnographists which indicate whether a person is sleeping on their side, back or stomach. Frequent changes in the graph indicate whether they’re uncomfortable while they sleep; however the graph alone is not enough of an indicator of whether they’re breathing is obstructed–which is the primary reason for conducting a polysomnographic study.

Sukanya presented a semi-automatic method for concern map recovery. I’m curious how the contents of the concern maps she generates compare to the task contexts built by tools like Mylyn when used by experienced developers. If her method can reliably predict which files a developer would probably need to review or modify for a given task, that would certainly help users of Mylyn prepopulate a task context to guide their navigation around a code base.

I also met Jim Buckley from the University of Limerick. He’s studying the information seeking behaviours of open source developers. His study found that for certain open source projects, the most prevalent questions asked on developer mailing were about development processes (e.g. How do I access the bug tracker?). He also found more than half the questions asked went unanswered. I’m wondering whether those questions weren’t answered because the answer existed elsewhere on the project portal such as a wiki page.

Rain is in the forecast for the next two days. I spent much of the afternoon outdoors wandering along the sea wall, through Stanley Park, and around downtown Vancouver. I finished off the evening with some uni and mint-infused shochu. It was quite smooth and didn’t taste like it was 25% alcohol by volume–although I certainly felt the 25% a short time later.