{rfName}
Ev

Indexed in

License and use

Icono OpenAccess

Citations

9

Altmetrics

Analysis of institutional authors

Dentella, VittoriaAuthor

Share

Publications
>
Article

Evaluating the Language Abilities of Large Language Models vs. Humans: Three Caveats

Publicated to:Biolinguistics. 18 e14391- - 2024-01-01 18(), DOI: 10.5964/bioling.14391

Authors: Leivada, Evelina; Dentella, Vittoria; Guenther, Fritz

Affiliations

Humboldt Univ, Inst Psychol, Berlin, Germany - Author
Inst Catalana Recerca & Estudis Avancats ICREA, Barcelona, Spain - Author
Univ Autonoma Barcelona, Dept Catalan Philol, Barcelona, Spain - Author
Univ Autonoma Barcelona, Dept Filol Catalana, Barcelona 08193, Spain - Author
Univ Rovira i Virgili, Dept English & German Studies, Tarragona, Spain - Author
See more

Abstract

We identify and analyze three caveats that may arise when analyzing the linguistic abilities of Large Language Models. The problem of unlicensed generalizations refers to the danger of interpreting performance in one task as predictive of the models' overall capabilities, based on the assumption that because a specific task performance is indicative of certain underlying capabilities in humans, the same association holds for models. The human-like paradox refers to the problem of lacking human comparisons, while at the same time attributing human-like abilities to the models. Last, the problem of double standards refers to the use of tasks and methodologies that either cannot be applied to humans or they are evaluated differently in models vs. humans. While we recognize the impressive linguistic abilities of LLMs, we conclude that specific claims about the

Keywords

Artificial intelligenceGrammaticalityLarge language modelsProbabilitieProbabilities

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal Biolinguistics, Q3 Agency Scopus (SJR), its regional focus and specialization in Linguistics and Language, give it significant recognition in a specific niche of scientific knowledge at an international level.

Independientemente del impacto esperado determinado por el canal de difusión, es importante destacar el impacto real observado de la propia aportación.

Según las diferentes agencias de indexación, el número de citas acumuladas por esta publicación hasta la fecha 2025-06-19:

  • WoS: 3
  • Scopus: 6

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2025-06-19:

  • The use, from an academic perspective evidenced by the Altmetric agency indicator referring to aggregations made by the personal bibliographic manager Mendeley, gives us a total of: 6.
  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 6 (PlumX).

With a more dissemination-oriented intent and targeting more general audiences, we can observe other more global scores such as:

  • The Total Score from Altmetric: 5.85.
  • The number of mentions on the social network X (formerly Twitter): 12 (Altmetric).

It is essential to present evidence supporting full alignment with institutional principles and guidelines on Open Science and the Conservation and Dissemination of Intellectual Heritage. A clear example of this is:

  • The work has been submitted to a journal whose editorial policy allows open Open Access publication.
  • Additionally, the work has been submitted to a journal classified as Diamond in relation to this type of editorial policy.
  • Assignment of a Handle/URN as an identifier within the deposit in the Institutional Repository: http://hdl.handle.net/20.500.11797/imarina9369657

Leadership analysis of institutional authors

This work has been carried out with international collaboration, specifically with researchers from: Germany.

There is a significant leadership presence as some of the institution’s authors appear as the first or last signer, detailed as follows: Last Author (Guenther, Fritz).