Calculating the Information in Probability distributions

Instead of directly learning the concept of what these fancy terms convey, let's start by taking some example which will suffice us in digging the concept deep.

• Say there is an URN having balls of colors Red, Blue and White and You manually have put these balls in the Urn. The true probability distribution of random variable X taking the value R, B or W be [0.25, 0.35, 0.45].
• Now one of your friend comes to you and you gave him this game to predict the number of balls of various colors. He peeps a little into the jar and predicts the same as [0.35, 0.45, 0.2].
• Overall aim of yours is to let your friend predict the correct probability and now since his prediction is wrong, you say him that you will give him a hint of how wrong he is in his prediction.
• Say there was tournament of one your favourite game which you and Anant(your friend) use to watch together and predict what will happen in upcoming matches.
• Now say Final match happened and you watched that. Out of 4 teams A,B,C,D the team B won the match. So the actual probability of event X taking random values can be presented in a probability distribution as [0, 1, 0, 0] where all the probability mass is concentrated on event B happening.
• But unfortunately Anant had seen…