PDA

View Full Version : Why isn't the computer ranking the most accurate?



SeattleUte
12-04-2008, 09:08 AM
Meant to say most credible. Substitute credible.

TripletDaddy
12-04-2008, 09:14 AM
Two nights ago, the Pacers beat the Lakers on a last second tip in. They actually outrebounded the Lakers and scored more points in the paint.

The Lakers are odds-on favorites to win the NBA title, but they lost a game.

The Pacers aren't even predicted to make the playoffs in the Eastern conference.

If you were to ask a computer, the computer would tell you that the Pacers are 1-0 vs the Lakers this season. Yet nobody in their right mind would claim that the Pacers are actually the better team. Good teams have off nights. Bad teams have good nights. A computer cannot consider intangibles.

Put it this way....when you were up for admission into the partnership, would you have wanted a computer to generate statistics and the admissions committee make its decision strictly based on your work statistics, or do you want a chance to plead your case in person and add context to those statistics?

BlueHair
12-04-2008, 09:20 AM
The computers are less accurate at the beginning of the season and more accurate at the end.

il Padrino Ute
12-04-2008, 09:22 AM
Perhaps because computers can only do what humans tell them to do.

ERCougar
12-04-2008, 09:27 AM
Two nights ago, the Pacers beat the Lakers on a last second tip in. They actually outrebounded the Lakers and scored more points in the paint.

The Lakers are odds-on favorites to win the NBA title, but they lost a game.

The Pacers aren't even predicted to make the playoffs in the Eastern conference.

If you were to ask a computer, the computer would tell you that the Pacers are 1-0 vs the Lakers this season. Yet nobody in their right mind would claim that the Pacers are actually the better team. Good teams have off nights. Bad teams have good nights. A computer cannot consider intangibles.

Put it this way....when you were up for admission into the partnership, would you have wanted a computer to generate statistics and the admissions committee make its decision strictly based on your work statistics, or do you want a chance to plead your case in person and add context to those statistics?

No computer would tell you the Pacers are a better team either. They all deal with probabilities of winning, accepting the possibility that a worse team could win. Given the choice, I'll take computers over people any day, although I think the ideal is a mix (a la BCS standings).

TripletDaddy
12-04-2008, 09:28 AM
Perhaps because computers can only do what humans tell them to do.

I'm Sorry, EPU. I'm afraid I cannot agree with that.

This conversation can serve no purpose anymore. Goodbye.

http://cr4.globalspec.com/PostImages/200803/hal_C7E74921-0C75-0D6F-A5B2FE8D41F73CA2.jpg

TripletDaddy
12-04-2008, 09:30 AM
No computer would tell you the Pacers are a better team either. They all deal with probabilities of winning, accepting the possibility that a worse team could win. Given the choice, I'll take computers over people any day, although I think the ideal is a mix (a la BCS standings).

The BCS ranking differential between SC and Utah is minimal.

What do you think the human odds would be in a game with SC v Utah? A push? No way. USC would be heavy favorites.

The computers tell us that some teams are really comparable. In reality they are not.

What was the computer ranking for BYU and for TCU the day before both teams played each other? What happened there?

SeattleUte
12-04-2008, 09:32 AM
Perhaps because computers can only do what humans tell them to do.

The algorithms are based on abstract principles evenyone agrees on as being relevant to determining the pecking order. They function without bias deploying data with pure objectivity. Computers are not crystal balls. But it seems to me they're more credible than the bias-fraught human polls. Human polls are fraught with the very bias you decry.

il Padrino Ute
12-04-2008, 09:37 AM
The algorithms are based on abstract principles evenyone agrees on as being relevant to determining the pecking order. They function without bias deploying data with pure objectivity. Computers are not crystal balls. But it seems to me they're more credible than the bias-fraught human polls. Human polls are fraught with the very bias you decry.

I realize that. But again, the algorithms are what humans have agreed upon. There is still that human element.

The computers are not biased, but the humans feeding the data are.

ERCougar
12-04-2008, 09:41 AM
The BCS ranking differential between SC and Utah is minimal.

What do you think the human odds would be in a game with SC v Utah? A push? No way. USC would be heavy favorites.

The computers tell us that some teams are really comparable. In reality they are not.

What was the computer ranking for BYU and for TCU the day before both teams played each other? What happened there?

Rankings should be a combination of both achievement on the season and quality of the team. An underperforming 9-3 bunch of superstars shouldn't be #1, even if they could blow anyone away if they decided to show up. I'm not convinced USC would be "heavy" favorites anyway, even if that did matter. Utah deserves a reward for beating everyone they played, particularly given their common opponent.

I don't recall the exact computer ratings of BYU and TCU at gametime, but their computer ranking differential was much less than the human ranking differential, which would seem to argue against your point, right?

SeattleUte
12-04-2008, 09:42 AM
I realize that. But again, the algorithms are what humans have agreed upon. There is still that human element.

The computers are not biased, but the humans feeding the data are.

Whatever bias you're talking about isn't like the popularity contest of the polls.

il Padrino Ute
12-04-2008, 09:46 AM
Whatever bias you're talking about isn't like the popularity contest of the polls.

Probably so, but my point still stands that computers can only do what they're told to do by humans.

SeattleUte
12-04-2008, 09:47 AM
Probably so, but my point still stands that computers can only do what they're told to do by humans.

Of course. I said the most credible. We're talking about relative credibility.

il Padrino Ute
12-04-2008, 09:48 AM
Of course. I said the most credible. We're talking about relative credibility.

Fair enough. And as much as I hate to agree, the computer rankings are most likely more credible than anything else.

Flystripper
12-04-2008, 09:52 AM
Meant to say most credible. Substitute credible.

The computer rankings in the BCS are a sham because they do not allow the statisticians use MOV in any way. Any computer model that does not have a factor for MOV is pretty weak IMHO. The most accurate poll is the Vegas poll that combines proprietary statistical models with human observation.

Flystripper
12-04-2008, 09:54 AM
Rankings should be a combination of both achievement on the season and quality of the team. An underperforming 9-3 bunch of superstars shouldn't be #1, even if they could blow anyone away if they decided to show up. I'm not convinced USC would be "heavy" favorites anyway, even if that did matter. Utah deserves a reward for beating everyone they played, particularly given their common opponent.

I don't recall the exact computer ratings of BYU and TCU at gametime, but their computer ranking differential was much less than the human ranking differential, which would seem to argue against your point, right?

Rankings are a reward and not a measure of who is best? I disagree with your definition of rankings, but you are entitled to your opinion.

Flystripper
12-04-2008, 09:55 AM
Fair enough. And as much as I hate to agree, the computer rankings are most likely more credible than anything else.

The vegas poll is the most credible poll IMHO.

ERCougar
12-04-2008, 09:57 AM
Rankings are a reward and not a measure of who is best? I disagree with your definition of rankings, but you are entitled to your opinion.

So do you think Utah deserved the MWC championship? Do you think they were the best team? Do you think Vegas would pick them against TCU?

My answers: Yes, No, No.

If we're ranking quality of talent, I can see your point, although I would still argue that there's something to be said for consistent winning. But to me, the point of rankings is to determine either seedings for a tournament or a national champion, and in either of those cases, there should be some reward for on-the-field achievement and not just on-paper talent.

Flystripper
12-04-2008, 09:59 AM
So do you think Utah deserved the MWC championship? Do you think they were the best team? Do you think Vegas would pick them against TCU?

My answers: Yes, No, No.

If we're ranking quality of talent, I can see your point, although I would still argue that there's something to be said for consistent winning. But to me, the point of rankings is to determine either seedings for a tournament or a national champion, and in either of those cases, there should be some reward for on-the-field achievement and not just on-paper talent.

Standings are not rankings. Utah was MWC champion because of their conference standings not ranking. It may sound like a quibble but I think the difference is meaningful.

ERCougar
12-04-2008, 10:02 AM
Standings are not rankings. Utah was MWC champion because of their conference standings not ranking. It may sound like a quibble but I think the difference is meaningful.

Kind of like civil union or marriage.

I see your point, I guess. There are certainly exceptions to computer rankings providing an accurate predictor, but I'll take them over any poll (except like you say, the Vegas oddmakers, whose livelihoods depend on it) to predict outcomes. DDD's BYU-TCU game is a perfect example of how bad voters are at picking quality.

CJF
12-04-2008, 11:58 AM
No computer would tell you the Pacers are a better team either. They all deal with probabilities of winning, accepting the possibility that a worse team could win. Given the choice, I'll take computers over people any day, although I think the ideal is a mix (a la BCS standings).

The problem is not the computers but who writes the computer program. The computers have built in biases also. If they didn't, we would only need one computer for the BCS but there are several different models used. Who is to say there aren't less biased versions out there not being used?

SeattleUte
12-04-2008, 12:20 PM
The problem is not the computers but who writes the computer program. The computers have built in biases also. If they didn't, we would only need one computer for the BCS but there are several different models used. Who is to say there aren't less biased versions out there not being used?

How amazing that despite all the built in biases about which 3D likes to run off about ad nauseum the Utes are still ranked no. 5 in the computer.

TripletDaddy
12-04-2008, 12:38 PM
How amazing that despite all the built in biases about which 3D likes to run off about ad nauseum the Utes are still ranked no. 5 in the computer.

I don't think I talked about bias. Did I?

Your stats show why the computer is not always the most accurate.

As I stated, no human is going to look at that ranking and think it is accurate. Vegas won't give those odds on the Utes, not by a longshot.

What is that ranking, btw? It isnt the overall BCS ranking.

pelagius
12-04-2008, 12:52 PM
If you were to ask a computer, the computer would tell you that the Pacers are 1-0 vs the Lakers this season. Yet nobody in their right mind would claim that the Pacers are actually the better team. Good teams have off nights. Bad teams have good nights. A computer cannot consider intangibles.


LOL, everybody's an econometrician.

DDD, Computer models have problems but not really the one your describe. You are not taking about intangibles really. You are just saying the Lakers had a bad game: bad luck. Models can approprately model the role of luck in sports outcomes; that's not particularly difficult. For example, my guess is that no model in its right mind would rate the Pacers better than the Lakers. Why? The model can figure out the Pacers got lucky too.

Certainly games like the preceding might, at least slightly, affect the precision of the inferences you can make but really that's a critique of a model's power rather than a critique about models leading to misleading inferences (and, of course, a game like that should affect the precision of the inferences we can draw a little or a tiny bit). Computer models have their limitations, and they are used in a poor way by the BCS (not just becuase the MOV issue) but they can and do take into account luck and bad bounces (in fact most of the time I suspect models do that part better than humans).

pelagius
12-04-2008, 12:58 PM
How amazing that despite all the built in biases about which 3D likes to run off about ad nauseum the Utes are still ranked no. 5 in the computer.

Technically a computer model reall doesn't say this. It just like a political poll. They should start reporting the confidence interval. The computer model actually says its best estimate is that Utah is five but that model can't tell with any precision if Utah is actualy probably the 1 best team to probably the 20th best team (just guessing but I am probably close). And the 5 comes from a bad model to begin with but even then the model really is claiming that it thinks the Utes are problably anywhere from the 1st best to the 20th.

This is a huge problem in general in the way the BCS uses computers. They move way beyond the inferences that the models themselves say they should actually make with the numbers. Its fine to still use them but people should understand their limitations with respect to this problem.

SeattleUte
12-04-2008, 01:00 PM
lol, everybody's an econometrician.

Ddd, computer models have problems but not really the one your describe. You are not taking about intangibles really. You are just saying the lakers had a bad game: Bad luck. Models can approprately model the role of luck in sports outcomes; that's not particularly difficult. For example, my guess is that no model in its right mind would rate the pacers better than the lakers. Why? The model can figure out the pacers got lucky too.

Certainly games like the preceding might, at least slightly, affect the precision of the inferences you can make but really that's a critique of a model's power rather than a critique about models leading to misleading inferences (and, of course, a game like that should affect the precision of the inferences we can draw a little or a tiny bit). Computer models have their limitations, and they are used in a poor way by the bcs (not just becuase the mov issue) but they can and do take into account luck and bad bounces (in fact most of the time i suspect models do that part better than humans).

smack down!!!!

TripletDaddy
12-04-2008, 01:03 PM
Technically a computer model reall doesn't say this. It just like a political poll. They should start reporting the confidence interval. The computer model actually says its best estimate is that Utah is five but that model can't tell with any precision if Utah is actualy probably the 1 best team to probably the 20th best team (just guessing but I am probably close). And the 5 comes from a bad model to begin with but even then the model really is claiming that it thinks the Utes are problably anywhere from the 1st best to the 20th.

This is a huge problem in general in the way the BCS uses computers. They move way beyond the inferences that the models themselves say they should actually make with the numbers. Its fine to still use them but people should understand their limitations with respect to this problem.

smack down!!!!!

pelagius
12-04-2008, 01:07 PM
Kind of like civil union or marriage.

I see your point, I guess. There are certainly exceptions to computer rankings providing an accurate predictor, but I'll take them over any poll (except like you say, the Vegas oddmakers, whose livelihoods depend on it) to predict outcomes. DDD's BYU-TCU game is a perfect example of how bad voters are at picking quality.

I have no doubt that the Vegas poll is the most accurate. But there would be a problem with incentives if the BCS actually used it. There would certainly be an incentive to put two teams against each other that would generate more betting action rather than necessarily the best two teams. Yes, this would be limited to some degree by reputational effects but I think the Vegas poll would have significant room to manipulate the poll to their advantage (at least among the top three or four teams). This is not to suggest there are no incentive problems with the other polls. The coaches poll is a nightmare from an incentive point of view.

pelagius
12-04-2008, 01:23 PM
smack down!!!!


smack down!!!!!

Sorry you two ... both of my posts came out harsher than I really meant.

CJF
12-04-2008, 01:47 PM
How amazing that despite all the built in biases about which 3D likes to run off about ad nauseum the Utes are still ranked no. 5 in the computer.

You won't see me knock the Utes or their ranking for being too high this year in either human or computer polls. There are biases inherent in the system though. I guess that's why there are three deciding factors in determining the BCS ranking. I still think the system is broken and the only way to truly fix it is to have a playoff.

ERCougar
12-04-2008, 02:01 PM
I have no doubt that the Vegas poll is the most accurate. But there would be a problem with incentives if the BCS actually used it. There would certainly be an incentive to put two teams against each other that would generate more betting action rather than necessarily the best two teams. Yes, this would be limited to some degree by reputational effects but I think the Vegas poll would have significant room to manipulate the poll to their advantage (at least among the top three or four teams). This is not to suggest there are no incentive problems with the other polls. The coaches poll is a nightmare from an incentive point of view.

Agree on all counts. I don't think the Vegas poll should be used to rank or seed teams in a playoff, for both the reason you cited and that, as I mentioned earlier, there should be some reward for performance on the field through the whole season. Thus, Florida may be the best team on the field, but we shouldn't ignore their loss to Ole Miss.

I also agree the coaches poll potentially has all sorts of bias, the AP less so, and I guess the Harris is a mix. No computer poll is perfect, so my ideal situation would be a mix of Harris, AP, and several computer polls to seed teams for a playoff. It would be interesting to see the confidence intervals on the computer ratings.

pelagius
12-04-2008, 02:10 PM
It would be interesting to see the confidence intervals on the computer ratings.

I will do it if I have a chance to do a little programming tonight. The intervals are, of course, different for each team and it makes sense to do it in terms of rankings. IOW, find the range of rankings where the hypotheses tests of ranking equality are statistical insignificant (technically this should be done taking into account a multiple hypotheses test inference problem but we won't worry about that). Not exactly the same as a typical 95% confidence interval but it makes more sense in context of a computer poll.

pelagius
12-04-2008, 04:22 PM
A It would be interesting to see the confidence intervals on the computer ratings.

For an example I did Utah



W/L Computer Model

Confidence Interval
Est. Rank Low High
4 1 45

MOV Computer Model

Confidence Interval
Est. Rank Low High
13 5 38


As you can see, there is a lot of sorting on noise when it comes to computer models. On the other hand, its not like the human polls aren't doing their share of sorting on noise.

ERCougar
12-04-2008, 04:35 PM
For an example I did Utah



W/L Computer Model

Confidence Interval
Est. Rank Low High
4 1 45

MOV Computer Model

Confidence Interval
Est. Rank Low High
13 5 38


As you can see, there is a lot of sorting on noise when it comes to computer models. On the other hand, its not like the human polls aren't doing their share of sorting on noise.

Is this based on your own computer models?

pelagius
12-04-2008, 04:38 PM
Is this based on your own computer models?

Ya, those are mine; those are the only one I can do confidence intervals for but I think they are fairly representative of most W/L and MOV based models. The other models definitely would not have more statistical precision than mine.

ERCougar
12-04-2008, 05:27 PM
Ya, those are mine; those are the only one I can do confidence intervals for but I think they are fairly representative of most W/L and MOV based models. The other models definitely would not have more statistical precision than mine.

Interesting. Is your MOV model straight up MOV or does it have some sort of discount for large MOV's?

pelagius
12-04-2008, 05:39 PM
Interesting. Is your MOV model straight up MOV or does it have some sort of discount for large MOV's?

That one was RAW anadjusted MOV. I just did those two because they represent the two extremes. An adjusted MOV is easy to implement, though. When I do adjust I do something like the following:



Adjusted MOV = MOV if MOV <= 21
= 21 + (MOV-21)^0.5 if MOV > 21


When you do that the Utes move from 13 to 12 [but the confidence interval would be basically unaffected].


Someday, I will clean up my code (and move it all to perl for portability) and make it available to anybody who wants it.