Member-only story
Metrics for ML Model in Production
The dominant approach to creating ML systems is to collect a dataset of training examples demonstrating correct behaviour for a desired task, train a system to imitate these behaviours, and then test its performance on independent and identically distributed (IID) held-out examples. These metrics say Cross-validation score are not the only metric that matters in model production. There are a variety of metrics you want to track that includes software, model input, model performance and user interaction.
𝟭. 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 — 𝗰𝗼𝗺𝗽𝘂𝘁𝗲 𝗲𝗻𝗴𝗶𝗻𝗲 𝗮𝗻𝗱 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 𝘂𝘀𝗲𝗱 𝘁𝗼 𝘀𝗲𝗿𝘃𝗲 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹
The metrics tracked could include memory usage, compute utilization and prediction latency. If there are ever errors or over-usage of resources, then alerts must be set up to notify the team via email.
𝟮. 𝗠𝗼𝗱𝗲𝗹 𝗜𝗻𝗽𝘂𝘁 — 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗼𝗳 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮 𝗶𝗻 𝘁𝗵𝗲 𝗱𝗼𝘄𝗻𝘀𝘁𝗿𝗲𝗮𝗺 𝗽𝗿𝗼𝗰𝗲𝘀𝘀
The metrics to be tracked included null rate, outliers, and trend shifts. If there are shifts and performance declined greatly, then it’s important to consider re-designing the model
𝟯. 𝗠𝗼𝗱𝗲𝗹 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 — 𝗼𝗻𝗹𝗶𝗻𝗲 𝗺𝗼𝗱𝗲𝗹 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲
