DuoLingo-AutoDiff: In-Database Automatic Differentiation with MLIR

What is it about?

Automatic differentiation is a core algorithm for training models for machine learning. It is responsible for calculating the gradients of the model function, which are then used to incrementally improve the model's parameters. There are two ways to implement this algorithm: forward and reverse. Forward mode evaluates the function and computes the derivative in one pass once for each independet input variable, while in reverse mode, two passes are need to calculated all derivatives simultaneously. In-database machine learning enables machine learning models to be trained within a database system. This reduces latencies, such as network latency, and enhances data security. Further, code-generating database systems offer an compiler environment to generate arbitrary code. Therefore, this paper discusses the first open-source implementation of automatic differentiation with MLIR and LingoDB. LingoDB is a code-generating database system that uses the MLIR compiler infrastructure to generate machine code from user queries. The MLIR infrastructure is designed to facilitate the development of domain-specific intermediate representations. These representations are temporary and are necessary for generating machine code tailored to the currently used processor architecture. This work demonstrates that MLIR's optimization improves automatic differentiation's runtime by reusing frequently used sub-expressions, thereby minimizing the generated code. Additionally, the results show that the forward mode algorithm generates more instructions and is therefore slower than the reverse mode algorithm.

Featured Image