, , , ,
"Quantitative Methods in Research Evaluation: Citation Indicators, Altmetrics, and Artificial Intelligence" by Mike Thelwall from the University of Sheffield critically examines the use of citation data, altmetrics, and artificial intelligence in research evaluation. The book discusses various indicators that aid in evaluation at different levels such as articles, scholars, departments, universities, countries, and funders. It also explores the strengths and weaknesses of these indicators and their broader implications for research assessment. Thelwall presents evidence comparing citations and altmetrics against expert judgments from the UK Research Excellence Framework 2021 to showcase their value across diverse academic fields. Furthermore, he delves into the potential applications of traditional artificial intelligence and large language models for research evaluation with supporting evidence of their effectiveness. Ultimately, Thelwall concludes that while citation data can offer valuable insights in certain research fields for specific purposes, they should not be considered as measures of research quality on their own. He also argues that artificial intelligence may have a role in limited circumstances for certain types of research evaluation.
Introduction
The evaluation of research has always been a crucial aspect of academia, with the need to measure and assess the quality and impact of scholarly work. In recent years, there has been a growing interest in using quantitative methods for research evaluation, such as citation data, altmetrics, and artificial intelligence (AI). These methods offer new ways to analyze and understand the influence and reach of academic publications. However, their use has also sparked debates about their reliability and validity as measures of research quality.
In his book "Quantitative Methods in Research Evaluation: Citation Indicators, Altmetrics, and Artificial Intelligence," Mike Thelwall from the University of Sheffield critically examines these methods' strengths and limitations. He presents evidence from various studies to support his arguments while also discussing their implications for research assessment.
Citation Indicators
Citation indicators have long been used as a measure of research impact. They are based on the number of times a publication is cited by other researchers in their work. Thelwall discusses how citation data can be used at different levels – individual articles, scholars, departments, universities, countries, and funders – to evaluate research performance.
One significant advantage of citation indicators is that they provide a standardized way to compare publications across different fields. However, Thelwall highlights that this approach may not be suitable for all disciplines due to variations in citation practices among fields. For example, some fields tend to cite more frequently than others or have longer reference lists per publication.
Moreover, Thelwall argues that relying solely on citations as an indicator of research quality can lead to biases towards established researchers or popular topics rather than innovative or niche areas. He also notes that self-citations can inflate an author's citation count artificially.
To showcase the limitations of using citations alone for research evaluation purposes accurately,Thelwall compares them against expert judgments from the UK Research Excellence Framework 2021. The results show that while citation data can be useful in certain fields, they should not be considered as the sole measure of research quality.
Altmetrics
Altmetrics, or alternative metrics, are a relatively new method for measuring research impact. They track online mentions and discussions of scholarly work on social media platforms, news outlets, blogs, and other non-traditional sources. Thelwall explains how altmetrics offer a more comprehensive view of the reach and influence of publications beyond traditional citations.
One significant advantage of altmetrics is their ability to capture immediate attention and engagement with research compared to citations that may take years to accumulate. However, Thelwall also acknowledges that altmetrics have limitations in terms of coverage and reliability. For example, not all publications receive online attention or are mentioned on social media platforms.
To demonstrate the value of altmetrics in research evaluation,Thelwall presents evidence from studies comparing them against expert judgments from the UK Research Excellence Framework 2021 across various academic fields. The findings show that altmetrics can provide valuable insights into the societal impact and public engagement aspects of research but should not be used as a sole indicator of quality.
Artificial Intelligence
The use of artificial intelligence (AI) in research evaluation is still in its early stages but has gained significant interest due to its potential for automating processes and analyzing large amounts of data quickly. Thelwall discusses how traditional AI techniques such as machine learning algorithms can aid in identifying patterns and relationships between publications based on citation data or text analysis.
He also explores the emerging use of large language models like GPT-3 for evaluating research by automatically generating summaries or abstracts based on an article's content. While these methods have shown promising results in some cases,Thelwall argues that they may not be suitable for all types of research evaluation due to their limited understanding capabilities compared to human experts.
Conclusion
In conclusion, "Quantitative Methods in Research Evaluation: Citation Indicators, Altmetrics, and Artificial Intelligence" by Mike Thelwall provides a comprehensive analysis of the strengths and limitations of using citation data, altmetrics, and AI for research evaluation. The book offers valuable insights into the potential applications of these methods at different levels while also highlighting their caveats.
Thelwall's evidence-based approach to comparing these methods against expert judgments from the UK Research Excellence Framework 2021 adds credibility to his arguments. He concludes that while citation data can offer valuable insights in certain research fields for specific purposes, they should not be considered as measures of research quality on their own. Similarly, altmetrics can provide additional perspectives on research impact but should not be used as the sole indicator of quality.
The book also highlights the need for caution when using AI in research evaluation due to its limited understanding capabilities compared to human experts. Overall,Thelwall's work serves as a thought-provoking read for anyone interested in quantitative methods for research evaluation and raises important questions about their use and implications in academia.
