What is it about?
Technology-Assisted Reviews (TAR) aim to expedite document reviewing (e.g. medical articles or legal documents) by iteratively incorporating machine learning algorithms and human feedback on document relevance. Continuous Active Learning (CAL) algorithms have demonstrated superior performance compared to other methods in effciently identifying relevant documents. We address one of the key challenges for CAL algorithms -- deciding when to stop displaying documents to reviewers.
Photo by Markus Winkler on Unsplash
Why is it important?
In this paper, we handle the problem of deciding the stopping point of TAR under the continuous active learning framework by jointly training a ranking model to rank documents, and conducting a “greedy” sampling to estimate the total number of relevant documents in the collection. We prove the unbiasedness of the proposed estimators under a with-replacement sampling design, while experimental results demonstrate that the proposed approach, similar to CAL, eectively retrieves relevant documents but it also provides a transparent, accurate, and eective stopping point.
Read the Original
This page is a summary of: When to Stop Reviewing in Technology-Assisted Reviews, ACM Transactions on Information Systems, October 2020, ACM (Association for Computing Machinery), DOI: 10.1145/3411755.
You can read the full text:
Technologically Assisted Reviews in Empirical Medicine
Evidence-based medicine has become an important strategy in health care and policy making. In order to practice evidence-based medicine, it is important to have a clear overview over the current scientific consensus. These overviews are provided in systematic review articles, that summarise all evidence that is published regarding a certain topic (e.g., a treatment or diagnostic test). In order to write a systematic review, researchers have to conduct a search that will retrieve all the documents that are relevant. This is a difficult task, known in the Information Retrieval (IR) domain as the total recall problem. With medical libraries expanding rapidly, the need for automation in this process becomes of utmost importance. The goal of this lab is to bring together academic, commercial, and government researchers that will conduct experiments and share results for a high recall task that specialises in the medical domain, and release a reusable test collection that can be used as a reference for comparing different retrieval approaches in the field of clinical systematic reviews.
The following have contributed to this page