I know it when I see it
How can you measure familiarization? How can you tell how familiar someone is with a software system? It’s not enough to say you know it when you see it.
What does it mean to be an expert in something? At my research group meeting yesterday, we collectively decided we can operationalize1 expertise in at least three ways:
- Timely responsiveness
- Experts can perform tasks within their domain faster, more precisely, and more accurately than novices.
- Social consensus
- You’re an expert if most other people agree that you’re an expert.
- Predictive power
- The experts’ crystal balls are bigger, shinier, more reliable and more accurate than novices’.
Of course, those definitions presuppose that there are such things as experts. Greg suggested that I measure expertise relatively: Wei is more of an expert at Eclipse plugin development than Tom but less than Fiona.
1Learn a new word every day 
Yay, school!