A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments

  • Christiane Behnert, Dirk Lewandowski
  • Journal of Documentation, May 2017, Emerald
  • DOI: 10.1108/jd-08-2016-0099

A framework for designing retrieval effectiveness studies of library information systems

What is it about?

Purpose: This paper demonstrates how to apply traditional information retrieval evaluation methods based on standards from the Text REtrieval Conference (TREC) and web search evaluation to all types of modern library information systems including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources. Design/methodology/approach: We apply conventional procedures from information retrieval evaluation to the library information system context considering the specific characteristics of modern library materials. Findings: We introduce a framework consisting of five parts: (1) search queries, (2) search results, (3) assessors, (4) testing, and (5) data analysis. We show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs. printed monographs and what issues need to be considered for retrieval tests in the library context. Practical implications: The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context. Originality/value: Although a considerable amount of research has been done on information retrieval evaluation, and standards for conducting retrieval effectiveness studies do exist, to our knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-firstcentury library information systems. We demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

Read Publication


The following have contributed to this page: Christiane Behnert and Dirk Lewandowski