Monday, November 21, 2016

Academic evaluation and hiring in Italy: The curious incident of Giovanni Sambin in the ASN 2016

Disclaimer: This post might contain imprecisions about the ASN, since I have never worked at an Italian university myself and I have never applied for the Italian ASN. I welcome corrections from whoever reads this post and has experience with this Italian evaluation exercise. Let me state at the outset that what I write pertains to fields such as computer science and mathematics. I do not know what is done in the humanities. 

An Italian law dated 30 December 2010 specifies a procedure for academic hirings in Italy at the level of associate and full professor. According to that law, recruiting for those positions should be "based on scientific qualification criteria. A national commission evaluates and assesses the candidates scientific qualification." See this outdated web site, which should be compared with the one in Italian.  Only candidates that have obtained the so-called Abilitazione Scientifica Nazionale (ASN, National Scientific Qualification) can then apply for a professor position at an Italian university, if and when such positions are advertised.

One can wonder why Italy uses this two-step system for academic hirings, whose need is not felt in any of the countries where I have worked so far. I guess that the first step is meant to filter out potential candidates who do not meet minimum requirements for being a professor at any of the 63 public universities in Italy. 

According to the regulations a "national commission evaluates and assesses the candidates scientific qualification." In fact, there is one national commission for each of the many scientific areas considered in Italy. Many of these commissions have to examine hundreds of applications, and their members play the role of gatekeepers and paladins of quality in the Italian university system. I can only imagine how much work is needed to do a thoughtful job in one of those committees and how easy it is to make enemies regardless of how considerate one is in justifying one's opinions. The evaluation is partly based on bibliometric criteria, which are known beforehand, should simplify the work of the commissions and should give a look of objectivity to their decisions. However, as far as I know, the commissions can also base their decisions on a qualitative analysis of the applicants.

Given the crucial role played by the members of the evaluation committees, one would expect that their members are chosen by taking the candidates' scientific profile and experience carefully into account. As it turns out, however, the qualifications of candidates for the committee are evaluated using only the following three bibliometric criteria:
  1. Number of publications in the period 2006-2016 (threshold 9);
  2. Total number of citations  in the period 2001-2015 (threshold 80);
  3. H-index in the last 15 years (threshold 5). 
In order to be eligible, one has to meet the thresholds in at least two of the above criteria. This might even seem reasonable. Note, however, that only publications indexed in Web of Science or Scopus count. In particular, journal papers published in outlets that are not indexed by Web of Science/Scopus are not taken into consideration (regardless of their content and impact) and conference papers don't count at all. Books and monographs don't count either, regardless of how influential they might be. Web of Science/Scopus are also used for calculating citations and the h-index. Again, this provides a smaller coverage than the one offered by Google Scholar, say.

By way of example, recently Giovanni Sambin, one of the most famous, currently active Italian logicians and an expert academic  one would trust to lead a national evaluation committee for Mathematical Logic, was considered to be ineligible as an evaluator because he met only one of the above-mentioned criteria. His Google Scholar profile is here.

This kind of decisions makes me wonder whether there is an overemphasis on bibliometric evaluations in Italian academia. If experience over a long and distinguished academic career plays second fiddle to fairly arbitrary thresholds calculated using only Web of Science and Scopus, I wonder how reliable the decisions of the evaluation committees will be considered by Italian academics. Most importantly, having so many people spend a lot of time seeking the holy grail of the national qualification and small committees devote endless hours examining their qualifications looks like a huge waste of energy and resources. I cannot help but think that that energy and time would be best used for research, teaching and all the other tasks that make up our work.

Tuesday, November 15, 2016

Workshops at LICS 2017

The following six workshops will be co-located with LICS 2017 and will take place on Monday, 19 June 2017, on the premises of Reykjavik University:
  • WiL: Women in Logic. Proposers: Valeria de Paiva, Amy Felty, Anna Ingolfsdottir, Ursula Martin
  • LCC: Logic and Computational Complexity. Proposers: Norman Danner, Anuj Dawar, Isabel Oitavem, Heribert Vollmer
  • LMW: Logic Mentoring Workshop. Proposers: Anupam Das, Valeria Vignudelli, Fabio Zanasi
  • LA: Learning and Automata. Proposers: Borja Balle, Leonor Becerra-Bonache, Remi Eyraud
  • LOLA: Syntax and Semantics of Low-Level Languages. Proposer: Matija Pretnar, Noam Zeilberger
  • Metafinite model theory and definability and complexity of numeric graph parameters. Proposers: Andrew Goodall, Janos A. Makowsky, Elena V. Ravve.
More details on this workshops will be available from the conference web page in due course. I hope that you'll consider attending them and submitting excellent papers to the conference. 

Wednesday, November 09, 2016

What makes a research institution excellent?

Some time ago I stumbled across the video of the panel discussion "IST Austria: On the Way to the Top: What Makes a Research Institution Excellent?". (There is also a much shorter, 11-minute version of the video here.) I watched the discussion with great interest, and found it inspirational and thought-provoking.

The panelists were Patrick Aebischer (president of EPFL until the end of 2016), Jonathan Dorfan (president of the Okinawa Institute of Science and Technology), Peter Gruss (former president of the Max-Planck-Gesellschaft from 2002 till 2014), Helga Nowotny (former President of the European Research Council), Rolf-Dieter Heuer (Director General of CERN from 2009 to 2015), Haim Harari (President, from 1988 to 2001, of the Weizmann Institute of Science) and Olaf K├╝bler (former president of ETH Zurich). It doesn't get much better than this, in terms of experience about the subject matter and, if you are interested in the topic or even just in hearing experienced academics discuss it, I'd encourage you to have a glass of your favourite beverage, relax and have a look. It is remarkable how much agreement there was in isolating the key ingredients leading to research excellence.

Here is my quarter-baked summary of some of the contributions, with apologies for not covering the whole discussion, possibly biased reporting and for any error I might have made.

Patrick Aebischer stated that Europe lacks super-brands such as Berkeley, CalTech, CMU, Harvard, MIT and Stanford. One needs elite universities to attract talents. The US attracts the best graduate students, the best young researchers with their tenure-track system and also people in high-ranking management positions. He also mentioned that to foster excellence, it is useful to have some competition between public and private universities. He said that integration of research and education is key to achieve excellence, as are attracting and keeping the best faculty, and giving early independence to young individuals. 
In order to achieve excellence, funding must be significant. A flexible organizational structure is needed to be able to compete at the highest level.
In this era, one can rise fast, but one can also fall faster than before.
Peter Gruss started by asking a fundamental question: what makes creative research possible? In his words, it is amazing how easy the answer is and how difficult it is to achieve it: "Hire the most brilliant minds and give them everything they need to stay brilliant." That's it.

He referred to the work of the historian Rogers Hollingsworth who isolated the following ingredients for excellence in research institutions:
  1. Excellence in research and leadership. On this point, Gruss said that is critical that one hires top people because top people hire people who are better than themselves. To get them, one has to do head hunting plus advertising. One should strike a good balance between tenured and non-tenured people to maintain flexibility.
  2. Small research settings.
  3. Small group size, but large context.
  4. Multidisciplinary contacts. One has to install interfaces between different disciplines. (Examples: Have only one coffee room.)
  5. Independence as early as possible. Give young people stability for a certain period of time to allow them to unfold their creativity. Coaching and mentoring of young researchers must be provided.
  6. Core institutional and flexible funds. There should be a balance between high-trust and low-trust funding. When handing out high trust funding, an agency must trust the funded institution: Give them the money that you can afford and let them do what they want with it. Trust them to make the most of the received funding. 
He also mentioned that a study in the US pointed out that 75% of citations in patents are to papers funded by public money. Of this 75% of papers most of them belong to the top10% of the papers cited within the scientific community. Hence one should invest in research, in fact, in top research. Hire the best without compromise. Put a lot of weight on top scientists.

Rolf-Dieter Heuer mentioned the importance of "taking society with you." One has to promote science in society.

For research, one needs to continuously develop a vision, which will drive innovation and technology, partnership with industry and feed back to research. Every excellent institution must keep this virtuous circle.  One must think strategically and long term.

All staff needs to have intellectual challenges, including administrative staff. Excellence can be in individuals, but also in cooperation. Excellence must allow for failure, for some research that might fail. This is doing science at the edge. What one can guarantee is that the path will be fruitful.

Olaf Kuebler stated that the strategy to create a leading research institution is deceptively simple: "Search, appoint and retain world-leading scientists. All else will follow."

The reputation of a university is made by the people who leave the university students, graduate students, assistant professors etc.

He also stated that an excellent research institution must:
  1. Make significant contributions to themes of global importance.
  2. Identify and develop new themes of global importance. 
  3. Harmonize its portfolio with its funders.
Helga Nowotny said that being open towards the future is the key aspect of excellence. Invest in excellent young people, who are competent rebels and understand that scientific knowledge is always preliminary. One has to bear in mind that excellence is always a multi-dimensional concept.
One should provide the best possible working conditions. This involves
  • a space component: space that makes it almost obligatory to run into each other and discuss, as ideas emerge by talking to each other, and 
  • a time component: give time for the unexpected, for the unforeseen, for serendipity.

Jonathan Dorfan mentioned that one should establish a setting that is conducive for inter-disciplinary research, where researchers from different fields can cooperate and exchange ideas.

Haim Harari closed the meeting with an articulate and thought-provoking short address. He started by pointing out what he considers to be key ingredients for an excellent research institution. 

Funding must be versatile and come from many sources. Only if one is versatile one can have the right mix. Government funding leads inevitably to egalitarianism  and democracy. However, science is not democratic. Still there has to be a balance between the power of the president and the faculty.

A research institution should be as international as possible and as national as possible. It should give something back to the taxpayers: education and touching society. Technology transfer is the other thing one return to society.

The Weizmann Institute put all the different subjects in the same campus, which leads to inter-disciplinary research that cannot be done by any single subject alone. 

Harari also said that the excellence of a research institution should be evaluated according to  three different measures:
  1. its best ten people,
  2. the average quality of its professors and
  3. its worst professor.
The quality of the worst professor says what the threshold of the institution is for hiring and is a very important indicator of the standards of the research institute/university.

In Harari's opinion, the president of a research institute/university should regularly ask herself/himself: If I could fire some of my professors, how many would I fire?" If the number is a non-trivial fraction of the faculty, then the threshold of the institution is not high enough.

So, in your opinion, what makes a research institution excellent?