AN AUTOMATED DATA QUALITY AND DRIFT REMEDIATION PIPELINE FOR PRODUCTION MACHINE LEARNING SYSTEMS

Abstract

Machine-learning pipelines deteriorate without warning. Once deployed, the statistical characteristics of the input data differ from the training distribution, even as the input data itself degrades in terms of quality, with missing values, outliers, schema conflicts, and duplicates. Current tools address each of these issues independently, with validation tools able to recognize constraint violations without fixing them, drift detectors being able to detect problems without taking action, and monitoring dashboards being able to visualize the decline without addressing it. In this work, we present a fully automated end-to-end MLOps pipeline, which integrates validation, drift detection, alerting, automated remediation, storage, and retraining. The pipeline takes batches of production data as inputs, validates them against a Great Expectations expectations suite, evaluates the distance between the batch distribution and a reference window with Evidently AI using the Kolmogorov-Smirnov, Wasserstein, Chi Squared, and Jensen-Shannon distances, aggregates feature-level results in a drift score, and uses a two-step threshold strategy to decide on the future of the batch. Remediation involves median and mode imputation, k-nearest-neighbours imputation, interquartile-range and Isolation Forest outlier correction, deduplication, and refitting of encoders. The resulting data is stored in partitioned Parquet format with lineage and validation metadata stored in PostgreSQL, with Apache Airflow managing the directed acyclic graph and retraining queue based on a promotion gate. Experiments were run using three real-world drifting datasets along with two synthetic drift generators, obtaining an accuracy of 94.2 percent in detecting drifts with F1 score of 0.931 and mean detection delay of 2.4 batches; median end-to-end latency of 36.2 seconds for a 50,000-row batch; and mean gains in F1 between 0.054 and 0.103 after remediation. Compared to a static deployed model, the pipeline lowers peak accuracy degradation by 12.8 percentage points (from 16.7 to 3.9) or 76.8 percent reduction in peak accuracy degradation, while reducing the number of retraining requests by 33.3 percent.