Friday, October 23, 2009

Benchmarking Metrics: What and Why?

In his post entitled "Top 10 theory schools?", Jonathan Katz gives a list of what he considers to be the top 10 theory departments in the US. (Read the post and the comments for an update on the discussion.) One of the comments reads as follows:

I would agree about:


MIT (Silvio, Shafi, Ron, Michel, …)
Cornell (Rafael, Eva, Jon, Bobby,…)
Berkeley (Luca, Christos, Umesh,…)
CMU (Venkat, Manuel, Avrim, Ryan,…)
Princeton (Boaz, Sanjeev, Moses, Bernard,..)

The others are a bit flakier:

GA Tech (Santosh, Chris, Vijay, Sasha, …)
UT Austin (Adam, David, Brent)
UCSD (Mihir, 1/4 Russell, Daniele, maybe Hovav)
U Washington (Anna, Paul, Anup)

I would say at least he following schools are VERY comparable to above:

Stamford (Dan, Serge, Tim, Amin,…)
NYU (Subhash, Assaf, Yevgeniy, Richard, …)
Harvard (Salil, Michael(s), Leslie)
Columbia (Mihalis, Rocco, Tal)

I would say there are top 5, and then top 10 following them.
So this commentator measures the quality of a theory group using the perceived research quality of the very best researchers at a given institution. Another possible metric would be the size of the group of active researchers with high international visibility. Yet others could be the number of peer-reviewed publications in top-class journals and conferences, or the average such number per staff member.

What criteria are most useful and why (bearing in mind that, at the end of the day, we are always making subjective judgements)? I am interested in this topic since the School of Computer Science I am working at is presently undertaking a benchmarking exercise. The aim of the exercise is to find three to four departments in the Nordic countries with which we aim at comparing ourselves in the very short term, within five years and within 10-15 years. One of the interesting aspects of this exercise is that our school is substantially smaller than most of the departments elsewhere. So, what do you think would be good metrics for the benchmarking exercise? Standing of the top scientists within the school? Average number of peer-reviewed publications and citations? Or what?

3 comments:

Derrick Stolee said...

If the list is supposed to be "Top 10 Theory Schools to attend as a graduate student" a likely benchmark would be average number of theory students graduating and receiving jobs similar to that student's goal.

For instance, a list for top 10 theory schools for tenure-track positions could be different than top 10 theory schools for placement at research labs.

ryanw said...

Probably one of the most important factors for a grad student (if not the most important) is the potential for a good advisor. This is one reason why it might make sense to focus on the top 5 researchers at each school. If a group has 30 researchers but none of them is a good advisor fit, this may be worse than a group of 1 researcher who is a perfect fit.

So it really depends on who/what you are benchmarking for.

Do you want to measure your scientific impact? Look at some function of citations, awards, etc.

Do you want to measure your influence on the community? Look at the numbers of program committee memberships, editorial boards, invited articles/talks, former students hired into faculty, etc.

Do you want to measure whatever that anonymous poster was measuring? Compute some function of the above, then take only the top 3-5 ranked people into account.

Anonymous said...

I'm really surprised Brown didn't make it: Upfal? Mathieu? Klein? Tamassia? Preparata? ...