End-to-end Machine Learning Model Monitoring Pipeline With NannyML and Grafana
Overview
Machine learning projects don’t stop at a successful model inside a Jupyter notebook. They don’t even stop when you deploy it to cloud and have an inference endpoint. ML is an iterative process, never-ending. Even after deployment, you have to constantly babysit it so that it works just as well as it did during the development phase.
Zillow’s scandal is a perfect example of what happens if you don’t. In 2021, Zillow lost a stunning 304 million dollars because of their machine learning model that estimated house prices. Zillow overpaid for more than 7000 homes and had to offload them at a much lower price. The company was “ripped off” by its own model and had to reduce its workforce by 25%.
These types of silent model failures are common with real-world models, so they need to be constantly updated before their production performance drops. Failing to do so damages companies’ reputation, trust with stakeholders, and ultimately, their pockets.
To demonstrate my skills in this area of machine learning, I’ve written a two-part tutorial that covers how to set up a monitoring pipeline with NannyML and an alerting system with Grafana so that I get Discord notifications when model performance drops:
- An End-to-End ML Model Monitoring Workflow with NannyML in Python
- Grafana Tutorial: A Beginner’s Guide to Monitoring Machine Learning Models
Key Skills Demonstrated
- Model monitoring and performance tracking
- Data drift detection and analysis
- Setting up monitoring dashboards
- Configuring alerting systems
- Python integration with monitoring tools
- MLOps and model maintenance
Tools & Technologies
- NannyML
- Grafana
- Python (pandas, scikit-learn, XGBoost)
- Docker
- Discord webhooks
- PostgreSQL
- Prometheus