Heaps' Passing Efficiency: A matched sample approach

[MarkGrace]

I'm not saying what you outline above isn't a real objection, but yes, I would say to some degree I am hoping my methodology is controlling for you central objection. That's why I suggested, it's not that I misread or misunderstood you, but rather that you don't think my identifying assumption overcomes your objection.

You use the following to illustrate your objection:


This is certainly a problem with passing efficiency particularly when making comparisons (although if I had to guess I think they are rarely first order objections when dealing with averages and controlling for opponent quality). I also agree that it can be a problem for cross-sectional comparisons or really any comparisons that involve significantly different offensive design, opponent quality, or supporting cast talent and execution level.

What I am assuming is that the non-quarterback talent level and non-quarterback execution level for this year is fairly similar to the average of BYU from 1993, 1997, 2004, and 2007. Some of those years probably had above average talent and execution level and some probably had below average. As long as this year isn't significantly worse than the average of those years over these dimensions than it's a decent way to do it. Specifically, we should end of up with that notion that the receivers had a similar probability to gain yards after a catch, drop a ball (this may be the most objectionable of them), and generally make a play when playing against teams of similar quality. This doesn't need to be perfectly true; just true enough that it is not a first order concern.

I would say what you highlighed is only part of my objection. To me my central objection is this:

A quarterback can't do a single thing without relying on his teammates. So I look at PER and wonder why we place such value on it in terms of measuring QB's in a 1:1 comparison. Sizzle breaks out these Heaps vs. Beck stats, and Jay says "oh, big difference" and then cites their PER. I see that the result is a big difference, but I don't see anything about it that screams big difference in their play, because I don't know what's gone into the resulting performances in terms of what the QB could control. If a QB throws all five balls on target and all of them get dropped, then what? I don't know. If a RB or lineman whiffs on his assignment and the QB goes down instead of getting a chance to make a play, then what? I'm not really sure. If a receiver lets a perfect pass go through is hands and get picked? If receivers are dropping passes in the endzone (which is by far the biggest way to kill this thing)?

I guess I'm less interested in arguing the probability of what would have happened and more interested in what actually happened.

[MarkGrace]

If you're a stats guy, I can't see why you would possibly not like college pass efficiency as a stat to evaluate college QB's. It takes important elements: pass completion %, yardage (normalizing for attempts), and TD:INT ration and combines them all together in one stat.

Of all the stats in all the sports to pick on, this seems like the last culprit.

I still think I gave you the best answer to this.

Look at this list.

http://web1.ncaa.org/mfb/natlRank.js...&div=IA&dest=O

or this

http://web1.ncaa.org/mfb/natlRank.js...&div=IA&dest=O

Look at the list and who's at the top and the middle and the bottom. Look at the sprinkling of teams, some teams with good talent still with a QB in the middle or bottom, and some with lesser talent still with a QB at the top.

I think I explained it pretty sufficiently in what you quoted. Of all the composite stats around, to me it seems by far the most team dependent (ie., not quality of opponent or talent level, but what the team is actually doing), which is why I don't find it exceptionally convincing in measuring individual performance. Like I said, great to look at, part of the equation, whatever.

[pelagius]

I
I guess I'm less interested in arguing the probability of what would have happened and more interested in what actually happened.

No, you cannot easily fall back on that distinction. That distinction washes out in averages (at least as a separate issue from the identifying assumption issue that I explained previously which is a real and reasonable objection even if I don't think it is a first order concern); that's why we deal with averages in empirical work. It's a reasonable objection to a single game or even a few games, but not really to a season average and its a stretch for a 1/2 season average.

[jay santos]

I think I explained it pretty sufficiently in what you quoted. Of all the composite stats around, to me it seems by far the most team dependent (ie., not quality of opponent or talent level, but what the team is actually doing), which is why I don't find it exceptionally convincing in measuring individual performance. Like I said, great to look at, part of the equation, whatever.

I might be misunderstanding you. Do you think all stats comparing individuals are useless? Or just QB stats? Or just pass efficiency?

[MarkGrace]

I might be misunderstanding you. Do you think all stats comparing individuals are useless? Or just QB stats? Or just pass efficiency?

No, I don't. But on a scale of the individual stats I'm most familiar with (and as Pelagius is showing, I'm not very familiar with this one ), it seems the least individual in nature, which I guess is why I'd rather see some discussion of what's actually happening rather than jump to a conclusion based on a result. I get that this is not really possible by virtue of the sport, but I guess that's why I'm discounting it more than the others. If I could rank sports based on how well they lend themsevles to isolated performance, it would look something like this:

1. Baseball
2. Basketball
3. Football (but well behind #2).

Like I said earlier, Indy said PER has the strongest correlation to offensive output, and I totally see that and it's importance there. But again, that doesn't make it any less team dependent to me. A pitcher can throw a strike without anyone's help (the catcher doesn't even have to catch it to be a strike), a basketball player can make a J on his own, but a QB can't really do anything in that formula without someone else's help. You say it's the best measure of QB performance -- that may very well be the case. But to me that doesn't make it foolproof in evaluating a QB. I'm never going to look at the thing and assume one QB is better than another just because of it, and a lot of people seem to want to treat it that way.

[jay santos]

No, I don't. But on a scale of the individual stats I'm most familiar with (and as Pelagius is showing, I'm not very familiar with this one ), it seems the least individual in nature, which I guess is why I'd rather see some discussion of what's actually happening rather than jump to a conclusion based on a result. I get that this is not really possible by virtue of the sport, but I guess that's why I'm discounting it more than the others. If I could rank sports based on how well they lend themsevles to isolated performance, it would look something like this:

1. Baseball
2. Basketball
3. Football (but well behind #2).

Like I said earlier, Indy said PER has the strongest correlation to offensive output, and I totally see that and it's importance there. But again, that doesn't make it any less team dependent to me. A pitcher can throw a strike without anyone's help (the catcher doesn't even have to catch it to be a strike), a basketball player can make a J on his own, but a QB can't really do anything in that formula without someone else's help. You say it's the best measure of QB performance -- that may very well be the case. But to me that doesn't make it foolproof in evaluating a QB. I'm never going to look at the thing and assume one QB is better than another just because of it, and a lot of people seem to want to treat it that way.

I think I understand your position, but I don't think it's rational. All statistics, all metrics, all data have imperfections. But it would be pretty silly to toss them all out and go though life and business and everything blind. Pass rating is pretty much universally accepted as the best stat to compare college QB's.

[MarkGrace]

All statistics, all metrics, all data have imperfections.

Yes, agree.

But it would be pretty silly to toss them all out and go though life and business and everything blind.

Yes, agree.

Pass rating is pretty much universally accepted as the best stat to compare college QB's.

Yes, agree.

[Sizzle]

I think I understand your position, but I don't think it's rational. All statistics, all metrics, all data have imperfections. But it would be pretty silly to toss them all out and go though life and business and everything blind. Pass rating is pretty much universally accepted as the best stat to compare college QB's.

Basically what Mark Grace is saying is that Aaron Rodgers playing with a team of receivers and OLinemen from the local HS team would not look like an All-world QB if we were playing in the NFL. It's not because Rodgers sucks, but has a lot to do with the talent around him.

Obviously this is a very extreme example, and I don't think anybody is making that argument for Heaps in this case.

FWIW, I went back and watched part of the Utah State game. The ESPN delayed recording cut out a good portion of the first quarter (so I ended up not watching it). Basically I only saw Heaps performance inside the 10 on the first two series. The first series he was bad, nearly throwing an INT and vastly overthrowing Apo, neither time he was under significant pressure. I did think that the pass that he threw over JJs head was impacted though by a very poor snap that threw the timing of the play off. If the snap is on target, I think it is a very makeable throw and goes in for an easy 6. Snap is horrible, and he air mails it over his head.

I'll try to remember to set the DVR this time for the SJSU game so I can pay closer attention... though it is looking like that may not matter since Riley very well could be the guy.