Heaps' Passing Efficiency: A matched sample approach

[pelagius]

Very interesting. You didn't mention any controls for OC, but did you see any trends b/w Chow, Anae and Doman?

I actually, can do that but I have to make the empirical design different in that case. I would need to use all the years (say 1976-2011), and then impose more assumptions on the data. If I have time, I might do it. The comparison between Chow, Crowton, and Anae would be useful but the comparison involving Doman is basically statistically a joint comparison with Heaps.

[jay santos]

Also noteworthy to say Heaps had the most experience of any of those QB's entering his soph season.

Walsh started a couple games as a soph before medical redshirting and redoing his soph season.

Feterik and Hall's first season starting was the soph season.

Beck started only a few games his frosh year.

Heaps started most of the season his frosh year.

As for surrounding talent: Feterik probably had the worst followed by Heaps.

[U-Ute]

Just as I thought: red is better than blue.

:fighton3:

[pelagius]

Not really. Pelagius, I remember thinking that based on your response at the time you had read my post wrong, even though I didn't comment.

http://cougaruteforum.com/showthread...242#post664242

No I don't think I read your post wrong. This does sharpen the measure relative to the objections you raise. However, it certainly does not completely overcome them and relies on an assumption that you may find unpalatable.

[pelagius]

Also noteworthy to say Heaps had the most experience of any of those QB's entering his soph season.

Walsh started a couple games as a soph before medical redshirting and redoing his soph season.

Feterik and Hall's first season starting was the soph season.

Beck started only a few games his frosh year.

Heaps started most of the season his frosh year.

As for surrounding talent: Feterik probably had the worst followed by Heaps.

That's right; in that sense of I have tilted the empirical design heavily in Heaps' favor. All the empirical work I have done (in the past) suggests that the one big jump in average efficiency for quarterback comes after his first year as a starter. Nothing else seems to matter very much, and there is usually not much improvement on average after the first year/second year jump.

[doctorcoug]

I actually, can do that but I have to make the empirical design different in that case. I would need to use all the years (say 1976-2011), and then impose more assumptions on the data. If I have time, I might do it. The comparison between Chow, Crowton, and Anae would be useful but the comparison involving Doman is basically statistically a joint comparison with Heaps.

Perfect multicolinearity! Sweet, I knew I learned something in Pope's class.

[pelagius]

Perfect multicolinearity! Sweet, I knew I learned something in Pope's class.

Well, we do have 1.5 quarters of Riley Nelson to break the the linear dependence, but it's pretty close. Although, technically we can identify off the change from last year and this year. Statistically, this would identify all the change as being "caused" by Doman.

[pelagius]

For Jay; here is the same thing using yards per completion.

[pellegrino]

Probably, should have went with "Did you know Heaps sucks this bad?"

no, no, don't change it. It's perfect as it is! I don't know enough about your field to know if this could even qualify as innovative research, but if this were on a conference program, I'd go listen to it.

[pelagius]

no, no, don't change it. It's perfect as it is! I don't know enough about your field to know if this could even qualify as innovative research, but if this were on a conference program, I'd go listen to it.

Oh, it is probably innovative in some sense (i.e., I am the only one that does it this way), but its not simply not important.

[pellegrino]

Oh, it is probably innovative in some sense (i.e., I am the only one that does it this way), but its not simply not important.

audience, pelagius, audience.

[pelagius]

audience, pelagius, audience.

oops.

[MarkGrace]

No I don't think I read your post wrong. This does sharpen the measure relative to the objections you raise. However, it certainly does not completely overcome them and relies on an assumption that you may find unpalatable.

Then maybe I just don't understand how this improves upon my objections. Your made two points to my comments.

1. You can adjust for opponent quality -- this is not what I was objecting to.
2. You can adjust for talent of supporting cast -- maybe this is what you are referring to, but I don't know that it is specficially what I'm getting at.

But it could very well be that I'm just reading this wrong. My impression of what I was saying is more in line with what BYU71 said: I have no idea what the hell everyone else on the field is doing.

I consider myself familiar with baseball stats, very familiar with basketball stats, and unfamiliar with football stats. When there was so much talk last year about PER, I look into this thing a bit and considered it, and that's why I said in my previous post it struck me as ERA for football. Starting from a simplistic point with a pitcher, we maybe evaluate him based on how many runs he gives up (actually, on second thought, PER is probably even closer to this than ERA). Then we say, wait a second, it's not really his fault the SS kicks a groundball, so let's break this down to earned run aveage. From there we say, ok, even beyond errors, not all pitchers are playing with 8 guys that can cover similar amounts of ground or make similar plays, so we start breaking that even further down and try and get as close as we can to isolating those things which primarily within control of the pitcher to evaluate him.

It doesn't work as well in basketball because there's more of a team component to what's going on. Maybe we look at assists like a QB and say, well, you can't get an assist unless someone on the other end converts. It's a two-party exchange here. But we can look at other things and say, hey, this guy can convert baskets without necessarily relying on a teammate. So we have FG%, and FT%, and 3pt%, and TS%, EFG%, and PPS, and all those other ways of measuring scoring efficiency. Of course what other people on the team are doing is something of a factor, but we can still whittle this down in some manner.

A quarterback can't do a single thing without relying on his teammates. So I look at PER and wonder why we place such value on it in terms of measuring QB's in a 1:1 comparison. Sizzle breaks out these Heaps vs. Beck stats, and Jay says "oh, big difference" and then cites their PER. I see that the result is a big difference, but I don't see anything about it that screams big difference in their play, because I don't know what's gone into the resulting performances in terms of what the QB could control. If a QB throws all five balls on target and all of them get dropped, then what? I don't know. If a RB or lineman whiffs on his assignment and the QB goes down instead of getting a chance to make a play, then what? I'm not really sure. If a receiver lets a perfect pass go through is hands and get picked? If receivers are dropping passes in the endzone (which is by far the biggest way to kill this thing)?

btw, I'm not using these as examples specific to Jake. I'm merely laying them out there as possibilities of things where the QB potentially had made the right play even though nothing is going to reflect in his PER. And there are all kinds of examples as well where a QB can be doing something lesser (say, throw a 1 yard screen that the WR breaks a bunch of tackles on and takes 50 yards) that would result in a better PER than a QB maybe did something greater that resulted in a lower PER (say, throws a 50 yard pass to a wide open receiver that he drops).

I don't know. Just thoughts as to why I'm just not grabbing at this thing the way others are. It's great and it's nice to look at and something to throw in the equation, but I'm not going to automatically assume something based on it (ie, big difference between Beck and Heaps).

[pelagius]

Then maybe I just don't understand how this improves upon my objections. Your made two points to my comments.

1. You can adjust for opponent quality -- this is not what I was objecting to.
2. You can adjust for talent of supporting cast -- maybe this is what you are referring to, but I don't know that it is specficially what I'm getting at.

But it could very well be that I'm just reading this wrong. My impression of what I was saying is more in line with what BYU71 said: I have no idea what the hell everyone else on the field is doing.

...

A quarterback can't do a single thing without relying on his teammates. S
...

btw, I'm not using these as example specific to Jake. I'm merely laying them out there as possibilities of things where the QB potentially had made the right play even though nothing is going to reflect in his PER. And there are all kinds of examples as well where a QB can be doing something lesser (say, throw a 1 yard screen that the WR breaks a bunch of tackles on and takes 50 yards) that would result in a better PER than a QB maybe did something greater that resulted in a lower PER (say, throws a 50 yard pass to a wide open receiver that he drops).

I'm not saying what you outline above isn't a real objection, but yes, I would say to some degree I am hoping my methodology is controlling for you central objection. That's why I suggested, it's not that I misread or misunderstood you, but rather that you don't think my identifying assumption overcomes your objection.

You use the following to illustrate your objection:

And there are all kinds of examples as well where a QB can be doing something lesser (say, throw a 1 yard screen that the WR breaks a bunch of tackles on and takes 50 yards) that would result in a better PER than a QB maybe did something greater that resulted in a lower PER (say, throws a 50 yard pass to a wide open receiver that he drops).

This is certainly a problem with passing efficiency particularly when making comparisons (although if I had to guess I think they are rarely first order objections when dealing with averages and controlling for opponent quality). I also agree that it can be a problem for cross-sectional comparisons or really any comparisons that involve significantly different offensive design, opponent quality, or supporting cast talent and execution level.

What I am assuming is that the non-quarterback talent level and non-quarterback execution level for this year is fairly similar to the average of BYU from 1993, 1997, 2004, and 2007. Some of those years probably had above average talent and execution level and some probably had below average. As long as this year isn't significantly worse than the average of those years over these dimensions than it's a decent way to do it. Specifically, we should end of up with that notion that the receivers had a similar probability to gain yards after a catch, drop a ball (this may be the most objectionable of them), and generally make a play when playing against teams of similar quality. This doesn't need to be perfectly true; just true enough that it is not a first order concern.

[jay santos]

Then maybe I just don't understand how this improves upon my objections. Your made two points to my comments.

1. You can adjust for opponent quality -- this is not what I was objecting to.
2. You can adjust for talent of supporting cast -- maybe this is what you are referring to, but I don't know that it is specficially what I'm getting at.

But it could very well be that I'm just reading this wrong. My impression of what I was saying is more in line with what BYU71 said: I have no idea what the hell everyone else on the field is doing.

I consider myself familiar with baseball stats, very familiar with basketball stats, and unfamiliar with football stats. When there was so much talk last year about PER, I look into this thing a bit and considered it, and that's why I said in my previous post it struck me as ERA for football. Starting from a simplistic point with a pitcher, we maybe evaluate him based on how many runs he gives up (actually, on second thought, PER is probably even closer to this than ERA). Then we say, wait a second, it's not really his fault the SS kicks a groundball, so let's break this down to earned run aveage. From there we say, ok, even beyond errors, not all pitchers are playing with 8 guys that can cover similar amounts of ground or make similar plays, so we start breaking that even further down and try and get as close as we can to isolating those things which primarily within control of the pitcher to evaluate him.

It doesn't work as well in basketball because there's more of a team component to what's going on. Maybe we look at assists like a QB and say, well, you can't get an assist unless someone on the other end converts. It's a two-party exchange here. But we can look at other things and say, hey, this guy can convert baskets without necessarily relying on a teammate. So we have FG%, and FT%, and 3pt%, and TS%, EFG%, and PPS, and all those other ways of measuring scoring efficiency. Of course what other people on the team are doing is something of a factor, but we can still whittle this down in some manner.

A quarterback can't do a single thing without relying on his teammates. So I look at PER and wonder why we place such value on it in terms of measuring QB's in a 1:1 comparison. Sizzle breaks out these Heaps vs. Beck stats, and Jay says "oh, big difference" and then cites their PER. I see that the result is a big difference, but I don't see anything about it that screams big difference in their play, because I don't know what's gone into the resulting performances in terms of what the QB could control. If a QB throws all five balls on target and all of them get dropped, then what? I don't know. If a RB or lineman whiffs on his assignment and the QB goes down instead of getting a chance to make a play, then what? I'm not really sure. If a receiver lets a perfect pass go through is hands and get picked? If receivers are dropping passes in the endzone (which is by far the biggest way to kill this thing)?

btw, I'm not using these as examples specific to Jake. I'm merely laying them out there as possibilities of things where the QB potentially had made the right play even though nothing is going to reflect in his PER. And there are all kinds of examples as well where a QB can be doing something lesser (say, throw a 1 yard screen that the WR breaks a bunch of tackles on and takes 50 yards) that would result in a better PER than a QB maybe did something greater that resulted in a lower PER (say, throws a 50 yard pass to a wide open receiver that he drops).

I don't know. Just thoughts as to why I'm just not grabbing at this thing the way others are. It's great and it's nice to look at and something to throw in the equation, but I'm not going to automatically assume something based on it (ie, big difference between Beck and Heaps).

If you're a stats guy, I can't see why you would possibly not like college pass efficiency as a stat to evaluate college QB's. It takes important elements: pass completion %, yardage (normalizing for attempts), and TD:INT ration and combines them all together in one stat.

Of all the stats in all the sports to pick on, this seems like the last culprit.

I still think I gave you the best answer to this.

Look at this list.

http://web1.ncaa.org/mfb/natlRank.js...&div=IA&dest=O

or this

http://web1.ncaa.org/mfb/natlRank.js...&div=IA&dest=O

Look at the list and who's at the top and the middle and the bottom. Look at the sprinkling of teams, some teams with good talent still with a QB in the middle or bottom, and some with lesser talent still with a QB at the top.