NSF Funded Workshop on Scholarly Evaluation Metrics

A one-day NSF-funded workshop entitled "Scholarly Evaluation Metrics: Opportunities and Challenges" will take place in the Renaissance Washington DC Hotel on Wednesday, December 16th 2009. The 50 available seats were filled the day that the workshop was announced. I would have loved to be in attendance, given my role as a P&T chair, but I heard about it four days after the announcement.

The focus of the workshop is the future of scholarly assessment approaches, including organizational, infrastructural, and community issues. The overall goal is to:

"identify requirements for novel assessment approaches, several of which have been proposed in recent years, to become acceptable to community stakeholders including scholars, academic and research institutions, and funding agencies."

Panelists include Oren Beit-Arie (Ex Libris), Peter Binfield (PLoS ONE), Johan Bollen (Indiana University), Lorcan Dempsey (OCLC), Tony Hey (Microsoft), Jorge E. Hirsch (UCSD), Julia Lane (NSF), Michael Kurtz (Astrophysics Data Service), Don Waters (Andrew W. Mellon Foundation), Jevin West (UW/eigenfactor.org), and Jan Velterop (Concept Web Alliance).

A summary of the goal of the workshop:

The quantitative evaluation of scholarly impact and value has historically been conducted on the basis of metrics derived from citation data. For example, the well-known journal Impact Factor is defined as a mean two-year citation rate for the articles published in a particular journal. Although well-established and productive, this approach is not always best suited to fit the fast-paced, open, and interdisciplinary nature of today's digital scholarship. Also, consensus seems to emerge that it would be constructive to have multiple metrics, not just one.

In the past years, significant advances have been made in this realm. First, we have seen a rapid expansion of proposed metrics to evaluate scientific impact. This expansion has been driven by interdisciplinary work in web, network and social network science, e.g. citation PageRank, h-index, and various other social network metrics. Second, new data sets such as usage and query data, which represent aspects of scholarly dynamics other than citation, have been investigated as the basis for novel metrics. The COUNTER and MESUR projects are examples in this realm. And, third, an interest in applying Web reputation concepts in the realm of scholarly evaluation has emerged and is generally referred to a Webometrics.

A plethora of proposals, both concrete and speculative, has thus emerged to expand the toolkit available for evaluating scholarly impact to the degree that it has become difficult to see the forest for the trees. Which of these new metrics and underlying data sets best approximate a common-sense understanding of scholarly impact? Which can be best applied to assess a particular facet of scholarly impact? Which ones are fit to be used in a future, fully electronic and open science environment? Which makes most sense from the perspective of those involved with the practice of evaluating scientific impact? Which are regarded fair by scholars? Under which conditions can novel metrics become an accepted and well-understood part of the evaluation toolkit that is, for example, used in promotion and tenure decisions?

I look forward to the twitter stream..

Sphere: Related Content

Have Life Science Researchers Removed Themselves from the Mainstream Library User Population?

A report Entitled Patterns of Information Use and Exchange: Case Studies of Researchers in the Life Sciences has been released by the British Library and the Research Information Network.

The report was developed by capturing the day-to-day patterns of information use in seven research teams from a wide range of disciplines. The study, undertaken over 11 months and involving 56 participants, concluded that ‘one-size-fits-all’ information and data sharing policies are not achieving "scientifically productive and cost-efficient information use in life sciences"

Skip past all of that and jump to page 47 of the report. There, they state (I'll let the report speak for itself) :

"Conventional university library facilities rank low as a vehicle for accessing published information. The traditional role of professional information intermediaries has been largely replaced by direct access to online resources, with heavy reliance upon Google to identify them. Given the limitations of generic search engines such as Google, measures to reconnect researchers with IIS professionals could bring improvements in information retrieval, and benefits to the research process.

"Researchers also tend to use services that have been ‘proven’ by colleagues, or to interrogate websites they regard as authoritative and comprehensive in their field. When they use such services, researchers tend to take the results on trust: the specificity and the breadth of the information retrieved do not appear to require further enquiry.

"The result of all these developments is that many life science researchers have removed themselves from the mainstream library user population. They do not even use the library catalogue. Library-based services can replace the services researchers do use only by demonstrating that they can improve retrieval capability, and deliver results within a timeframe that corresponds to researchers’ own patterns of work. This is a significant challenge when researchers are driven by a desire for immediate online access to specific resources of interest, at a time convenient to them, and from a known and trusted source."

Overall they found that the groups that they studied use a narrow range of search engines and bibliographic resources, for three reasons:

• lack of awareness and time to achieve or build a broader suite
• the ‘comfort’ that comes from relying on a small set of familiar resources, usually endorsed by peers and colleagues, and
• the cost in time and effort needed to identify other resources, and to learn to use them effectively.

They detail what would appear to be emerging roles of the library in a researcher's information seeking patterns:

"The challenge for institutional information services is thus to develop and provide online services geared to the needs of their research groups and thereby to add value to the research process, facilitating the use of new tools, providing individuated professional support, as well as advice, training and documentation on a subject or discipline basis. Any such strategy would have to be proactive: as noted by our regenerative medicine group, researchers are reluctant to adopt new tools and services unless they know a colleague who can recommend or share knowledge about them."

"Library and information service providers in the higher education sector need to come to a clearer view of their structures and roles.. some of our groups expressed a desire for better portals and tools to identify the information resources relevant to researchers working in their domain. Some of the specialised repositories that are emerging (e.g. in neurophysiology) may help to develop such services."

"Re-establishing a lively and sustained dialogue with their research communities is a key challenge for the library and information services in many universities. Such dialogue is essential if libraries are to provide the publications, other information resources and services that their researchers need."

"Better engagement between information professionals and researchers could add to the efficiency and effectiveness of research, with specialist support facilitating the use of new tools, and providing individuated professional advice, training and documentation on a subject or discipline basis."

"Such a strategy would have to be proactive, for researchers are reluctant to adopt new tools and services unless they know a colleague who can recommend or share knowledge about them. And it would have to meet the challenge of delivering results that correspond to researchers’ patterns and timetables of work."

Sphere: Related Content