SignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors

What is it about?

Search Engines such as Google and Bing have revolutionized how we interact with the cyber world. However, search engines that support sign language queries in native form do not exist currently. While translating a sign language to a spoken language and using existing search engines might be one possibility, this can miss critical information because existing translation systems are either limited in vocabulary or constrained to a specific domain. In contrast, this paper presents a holistic approach where ASL queries in native form, as well as ASL videos and textual information available online, are converted into a common representation space. We designed search engines for sign languages. Such a joint representation space provides a common framework for precisely representing different sources of information and accurately matching a query with relevant information that is available online. Evaluated over a set of Deaf users with native ASL fluency, the accuracy is comparable with state-of-the-art recommendation systems for Amazon, Netflix, Yelp, etc., suggesting the usability of the system in the real world. For example, the recall@10 of our system is 64.3%, i.e., among the top ten search results, six of them are relevant to the search query. Moreover, the system is robust to variations in signing patterns, dialects, sensor positions, etc.

Featured Image

Photo by Viktor Forgacs on Unsplash

Photo by Viktor Forgacs on Unsplash

Why is it important?

In today's digital age, where information is at our fingertips, it's crucial to acknowledge the significant inequities that marginalized groups like the Deaf community face in accessing the vast wealth of knowledge on the internet. The internet often relies on textual, spoken, or auditory content, creating a formidable barrier for Deaf individuals who primarily use sign language. This inequality may contribute to language deprivation among Deaf individuals and limit their educational, social, and employment opportunities. To address this issue, enabling search inquiries using sign language, in one's native language, ensures that Deaf individuals can access information and participate fully in the digital world, thus promoting equity and reducing serious issues such as language deprivation.