Had our first paper discussion session of the year over lunch today. I think I focused too much on the negatives this time. It’s hard to overlook poorly formulated arguments, especially when it’s backed by inconclusive data.
I should have tried harder to tune into the signal while filtering out the noise. Would’ve been more constructive to think about some of the new contributions (if any—see? Negative again) those studies brought to the field.
Those papers did shed some insight on the difficulties of conducting experiments in software engineering. Solid numbers about the practical aspect of software engineering is hard to find. The interesting bits are hard to measure (e.g. productivity, reliability, correctness, efficiency).
Even if we could come up with a way to do it, many factors would get in the way of collecting those metrics faithfully and reliably (e.g. how do we get test subjects to “collect” data for us without bogging them down with extra documentation work?).
Round two—fight!