Tweet
I think YOhio is going to like this post. TopUte brought up Bronco's record in games decided by less than 7 points:
PAC points out that this is not necessarily evidence of being lucky:
Moliere pointed out that 18-2 would be a low a very low probability outcome by random chance (what luck implies ... a 50/50 chance of winning close games). Moliere is right in isolation. The probability (in isolation) of going 18-2 or better (assuming no skill) is 0.02%. Moliere and PAC's intuitions are actually quite good but this issue has some complications. This situation is complicated by at least two potentially important first order issues:
1. There is plenty of empirical evidence in the cross-section about persistence in close game performance by coaches. There is very little evidence of persistence (consistent with luck). However, Indy does point out that there may be evidence that a small number of coaches have "close game" skill. We need to take that into account. I am going to take this issue into account and test whether we can say if Bronco's close game record is consistent with luck or skill based on his close game record given it appears to be a rare skill.
2. There are a host of statistics that might be interesting and reveal something about luck versus skill. But if those are not interesting or not in the tails of the distribution we tend to ignore them. If you look at enough different statistics one of them is likely to look like an outlier. I am not going to deal with this issue, but I think it is a bit less important here.
First, I am going to ignore some other issues. Not all close games are created equal. For example, games where a few points were scored late and closed the gap probably shouldn't be treated the same as games where the lead change back in forth. Issues like this could make Bronco's record less (as in the example) or more remarkable.
The big issue we need to take into account is that based on the empirical evidence, it is very unlikely that a coach is skilled in winning close games. How can we take this "prior" into account? The answer is we use Baye's rule.
Suppose only 1/100 coaches has "close game skill". Is Bronco's record inconsistent with luck given that or is this evidence that he is one of those 1 out of 100 coaches?
Setup:
Let P(U) = 0.99 ... This is our prior as a probability that Bronco is unskilled.
Let P(S) = 0.01 ... This is our prior as a probability that Bronco is skilled.
Now for the conditional probabilities:
P(W=18|U) = 0.00018 ... The probability that Bronco has a record of 18-2 conditional on being unskilled is 0.00018 (this comes directly from the probability of winning each game being 0.5).
No let's suppose that skilled coaches win 0.80 of their close games (we could have various degrees of skill but I don't think that would change things much). Given that, the following would be true:
P(W=18|S) = 0.1369 ... The probability that Bronco has a record of 18-2 conditional on being skilled
Using Bayes Rule:
P(U|W=18) = P(W=18|U)*P(U)/(P(W=18|U)*P(U) + P(W=18|S)P(S)) =
P(U|W=18) = 0.00018*0.99/(0.00018*0.99 + 0.1369*0.01) = 11.5%
P(S|W=18) = 0.1369*0.01/(0.00018*0.99 + 0.1369*0.01) = 88.5%
So based on the evidence and what I think is a reasonable set of parameters it looks like Bronco is likely a skilled close game coach.
What would change this conclusion? If you think "close game skill" in coaches is really rare (like a once in a generation thing).
Suppose only 1/1000 coaches has close game skill:
Let P(U) = 0.999 ... This is our prior as a probability that Bronco is unskilled.
Let P(S) = 0.001 ... This is our prior as a probability that Bronco is skilled.
P(U|W=18) = P(W=18|U)*P(U)/(P(W=18|U)*P(U) + P(W=18|S)P(S)) =
P(U|W=18) = 0.00018*0.999/(0.00018*0.999 + 0.1369*0.001) = 56.8%
P(S|W=18) = 0.1369*0.001/(0.00018*0.999 + 0.1369*0.001) = 43.2%
So as long as you don't believe "close game coaching skill" is incredibly rare (on the order of it is unlikely that any current coach has close game coaching skill but maybe a handful have had coaching skill in the history of time), then it is likely that Bronco is a skilled "close game" coach.
Note this doesn't mean that BYU didn't experience overall good luck during these games. Luck may have played some role.
In short, PAC was right.
Originally posted by Top Ute
View Post
Originally posted by PaloAltoCougar
View Post
1. There is plenty of empirical evidence in the cross-section about persistence in close game performance by coaches. There is very little evidence of persistence (consistent with luck). However, Indy does point out that there may be evidence that a small number of coaches have "close game" skill. We need to take that into account. I am going to take this issue into account and test whether we can say if Bronco's close game record is consistent with luck or skill based on his close game record given it appears to be a rare skill.
2. There are a host of statistics that might be interesting and reveal something about luck versus skill. But if those are not interesting or not in the tails of the distribution we tend to ignore them. If you look at enough different statistics one of them is likely to look like an outlier. I am not going to deal with this issue, but I think it is a bit less important here.
First, I am going to ignore some other issues. Not all close games are created equal. For example, games where a few points were scored late and closed the gap probably shouldn't be treated the same as games where the lead change back in forth. Issues like this could make Bronco's record less (as in the example) or more remarkable.
The big issue we need to take into account is that based on the empirical evidence, it is very unlikely that a coach is skilled in winning close games. How can we take this "prior" into account? The answer is we use Baye's rule.
Suppose only 1/100 coaches has "close game skill". Is Bronco's record inconsistent with luck given that or is this evidence that he is one of those 1 out of 100 coaches?
Setup:
Let P(U) = 0.99 ... This is our prior as a probability that Bronco is unskilled.
Let P(S) = 0.01 ... This is our prior as a probability that Bronco is skilled.
Now for the conditional probabilities:
P(W=18|U) = 0.00018 ... The probability that Bronco has a record of 18-2 conditional on being unskilled is 0.00018 (this comes directly from the probability of winning each game being 0.5).
No let's suppose that skilled coaches win 0.80 of their close games (we could have various degrees of skill but I don't think that would change things much). Given that, the following would be true:
P(W=18|S) = 0.1369 ... The probability that Bronco has a record of 18-2 conditional on being skilled
Using Bayes Rule:
P(U|W=18) = P(W=18|U)*P(U)/(P(W=18|U)*P(U) + P(W=18|S)P(S)) =
P(U|W=18) = 0.00018*0.99/(0.00018*0.99 + 0.1369*0.01) = 11.5%
P(S|W=18) = 0.1369*0.01/(0.00018*0.99 + 0.1369*0.01) = 88.5%
So based on the evidence and what I think is a reasonable set of parameters it looks like Bronco is likely a skilled close game coach.
What would change this conclusion? If you think "close game skill" in coaches is really rare (like a once in a generation thing).
Suppose only 1/1000 coaches has close game skill:
Let P(U) = 0.999 ... This is our prior as a probability that Bronco is unskilled.
Let P(S) = 0.001 ... This is our prior as a probability that Bronco is skilled.
P(U|W=18) = P(W=18|U)*P(U)/(P(W=18|U)*P(U) + P(W=18|S)P(S)) =
P(U|W=18) = 0.00018*0.999/(0.00018*0.999 + 0.1369*0.001) = 56.8%
P(S|W=18) = 0.1369*0.001/(0.00018*0.999 + 0.1369*0.001) = 43.2%
So as long as you don't believe "close game coaching skill" is incredibly rare (on the order of it is unlikely that any current coach has close game coaching skill but maybe a handful have had coaching skill in the history of time), then it is likely that Bronco is a skilled "close game" coach.
Note this doesn't mean that BYU didn't experience overall good luck during these games. Luck may have played some role.
In short, PAC was right.
Comment