KAPE: Automatic performance estimation of deep code search

What is it about?

KAPE is an algorithm that can automatically estimate the performance of deep code search models on unseen data without manually matching code snippets to test queries. The main idea is to use semantically similar training data to perform the evaluation.

Featured Image

Why is it important?

Code search is a common yet important activity of software developers. An efficient code search model can largely facilitate the development process and improve the programming quality. Given the superb performance of learning the contextual representations, deep learning models, especially pre-trained language models, have been widely explored for the code search task. However, studies mainly focus on proposing new architectures for ever-better performance on designed test sets but ignore the performance on unseen test data where only natural language queries are available. The same problem in other domains, e.g., CV and NLP, is usually solved by test input selection that uses a subset of the unseen set to reduce the labeling effort. However, approaches from other domains are not directly applicable and still require labeling effort.