Interpretability of Machine Learning models

If a machine learning model performs well, why do we not just trust the model and ignore why it made a certain decision? Well, the problem is that a single metric, such as classification accuracy, is an incomplete description of most real-world tasks.

Let us dive deeper into the reasons why interpretability is so important. When it comes to predictive modelling, you have to make a trade-off:

• Do you just want to know what is predicted? For example, the probability that a customer will churn or how effective some drug will be for a patient.
• Do you want to know why the prediction was made and possibly pay for the interpretability with a drop in predictive performance?

In some cases, you do not care why a decision was made, it is enough to know that the predictive performance on a test dataset was good. But in other cases, knowing the ‘why’ can help you learn more about the problem, the data and the reason why a model might fail.

Some models may not require explanations because they are used in a low-risk environment, meaning a mistake will not have serious consequences, (e.g. a movie recommender system) or the method has already been extensively studied and evaluated (e.g. optical character recognition).

The need for interpretability arises from an incompleteness in problem formalization, which means that for certain problems or tasks it is not enough to get the prediction (the what). The model must also explain…