Building Reproducible ML Pipelines with DVC | Hands-On Tutorial

CML (Continuous Machine Learning) is an open-source tool that enables git flow for machine learning with auto-reporting features for pull requests and merge requests in GitHub, GitLab, or Bitbucket. It integrates seamlessly with cloud services like AWS, Azure, Google Cloud Platform, or Kubernetes. In a typical CI/CD workflow, engineers push changes to remote git servers, triggering jobs in CI/CD tools like GitHub Actions. CML can launch cloud VMs in AWS (potentially spot instances for cost savings), run pipelines, push metrics and reports, and automatically post customizable reports as comments in pull requests.

The UK Hydrographic Office demonstrates enterprise ML operations using DVC and CML for processing hundreds of gigabytes to terabytes of noisy marine sensor data. Before DVC, data preparation took 10-15 hours with manual server transfers between on-premise systems. DVC synchronizes data across environments and provides reproducibility for managing multiple model candidates with numerous experiment branches. CML enables automated GPU training by allowing local experiment design, GitHub code pushes, and AWS GPU runner creation. Results are saved via DVC to S3 and returned as metrics in Pull Requests. Teams transition from branch-per-model architectures to monorepos with subdirectories, using DVC's monorepo support for shared components. Key challenges include maintaining git hooks in subrepos, persisting model directories iteratively, and determining what constitutes an experiment versus local iteration.

Deploying CML involves creating a virtual machine using the OVA package. Users name the VM, select storage location, and configure network settings (typically DHCP for automatic IP assignment). The hypervisor reads default configurations from the OVA file and automatically sets up the VM. After deployment, CML boots and requires initial configuration through a wizard. Users must select network type (DHCP), configure system administrator credentials, and set up the CML controller username and password. The system applies all settings before allowing access to the CML GUI. This process prepares CML for network simulation operations.

CML installation requires VMware Workstation Pro and two files: a .ova file and a reference platform ISO. After importing the .ova, configure the VM with at least 4 CPUs, 24+ GB RAM (recommended 28 GB), and 150-180 GB disk. Set network adapter to Bridged mode with 'Connect at Power On'. The installation process involves accepting the license, setting admin credentials, and copying reference platform images from the ISO to the VM disk. The system assigns an IP address (e.g., 10.10.10.59) for dashboard access via HTTPS.

Continuous Integration (CI) practices from DevOps can be adapted for Machine Learning projects by extending CI systems to version data like code using tools like DVC, allocate cloud resources for model training, and provide feedback through metric reports in pull requests using libraries like CML (Continuous Machine Learning). This approach addresses the unique challenges of ML workflows where data changes significantly impact model outcomes, requiring specialized handling of large datasets, hardware allocation, and multi-metric evaluation beyond simple pass/fail tests.

This section covers the complete workflow for tracking and comparing deep learning experiments. PyTorch Lightning automatically integrates with TensorBoard for real-time visualization of training metrics. Configuration files enable automatic logging of all metrics in the required format. Each experiment run is saved as a new version, allowing side-by-side comparison of different models. Alternative logging methods include CSV files, which can be read with pandas for custom analysis. The lightning_logs folder stores all training data during the training process, enabling continuous monitoring and comparison of model performance across different training runs.

Experiment tracking systems record and visualize the results of machine learning experiments, including model performance metrics, training curves, and hyperparameter configurations. These systems enable data scientists to compare different experiments, identify successful configurations, and document the development process. Visualization tools help communicate results to stakeholders and support decision-making about model selection and deployment.

Determined provides comprehensive experiment tracking and visualization capabilities. The platform tracks data versions, model configurations, hyperparameter settings, training checkpoints, and evaluation metrics. Visualization tools like TensorBoard integration allow analysis of per-class validation metrics to diagnose model performance issues. In the demo, TensorBoard revealed that the 'dog' class had significantly lower accuracy (0.35-0.45 mAP) compared to other classes, indicating a data imbalance problem where the training dataset contained only 50 dog images compared to thousands of people images.

Experiment tracking enables easy comparison of multiple runs through visualization tools. The instructor explains that tracking systems can plot metrics from different runs on the same graph, calculate statistics, and highlight significant differences. The session covers tagging runs with labels for organization, smoothing metrics to identify trends, and customizing visualization options. The instructor emphasizes that comparison tools save time and reduce errors in manual analysis, enabling researchers to quickly identify which configurations perform best.

After model training, all information is captured in log files using the Machine Learning Flow Logger. Logs record metrics, experiment names, run IDs, and tracking information for error diagnosis and performance tracking. Metrics are accessed via run.get_metrics() or through the workspace experiments section. For visualizations like box plots (used to detect outliers and anomalies), the log_image() method renders plots as images. This systematic logging enables comprehensive experiment tracking and analysis.

MLflow's Model Registry is a centralized system for managing machine learning models through versioning and stage transitions (None, Staging, Production, Archived), enabling organizations to track model lineage and ensure consistent deployment across environments. Models can be logged and registered using mlflow.log_model() or mlflow.register_model(), with model URIs following the format 'models:/{model_name}/{version}'. Deployment options include serving models locally via mlflow model serve, deploying to Amazon SageMaker using mlflow sagemaker deploy, or creating flexible Docker images that can run on any cloud platform. The model registry supports both framework-specific models (Keras, PyTorch) and generic Python functions through mlflow.pyfunc.PythonModel wrappers, making it a versatile tool for the entire ML lifecycle from experiment tracking to production deployment.

This section demonstrates the complete ML deployment pipeline from training to production. It covers registering models in SageMaker Model Registry, organizing models into package groups (e.g., 'credit card fraud detection'), and setting approval statuses (pending, approved, rejected). The workflow shows how best-performing hyperparameter configurations identified through parallel tuning can be registered as model packages. Model registration includes defining input/output content types, allowed instance types, and approval workflows requiring ML operations team review before production deployment. This integration creates a complete governance loop from model development through deployment-ready registration.

Model registry is a centralized versioned storage for models, similar to Docker registry for containers. Register models using 'mlflow.register_model(run_id, name='model_name')' or through the UI. Models get version numbers (v1, v2) and can be tagged with metadata. Load models from registry using 'mlflow.sklearn.load_model(model_uri)' where URI follows 'models:/{model_name}/{version}'. Make predictions using the loaded model's predict method. This completes the MLOps workflow: data preparation, model training, experiment tracking, model validation, model registration, and deployment-ready prediction. The registry enables tracking different versions, managing deployments, and ensuring reproducibility.

After starting the tuning job, you can monitor its status in the Google Cloud console. When the tuning job completes, you'll see the tuned model in the Vertex AI model registry. From there, you can deploy it to an endpoint for serving or further test it in Vertex AI Studio. This workflow enables deploying customized models to production environments.

This section covers model lifecycle management and production deployment. MLflow enables logging models along with hyperparameters, metrics, and artifacts like SHAP explanations. The Model Registry provides a central place for tracking model versions and their lifecycle states, tracking which version is currently in production, which is a staging candidate, and who is authorized to move models between states. Before promotion, models typically undergo validation through automated tests or manual review. SHAP (Shapley Additive Explanations) provides model interpretability by showing which features most influenced each prediction, identifying top features affecting predictions, and revealing how feature effects change over time or across different groups. Moving models to production is challenging because development environments differ significantly from production systems. MLflow addresses this by providing standardized ways to export models for production use, including as Spark UDFs for batch and streaming processing.