<$BlogRSDURL$>
Proceeds from the ads below will be donated to the Bob Wuesthoff scholarship fund.

Thursday, February 23, 2006

Overestimating The Fog

It's that dead time of year between the last of the big signings and March, when the regular spring training scrimmages begin, and so the writers amuse themselves by speculating who'll do what. It's totally understandable, of course; I did it last year, and like Tom Tippett, Nate Silver, Rich Lederer, and virtually the whole damn staff at Baseball Prospectus, I picked the A's to win the division; the only exceptions at BPro were Christina Kahrl, Jay Jaffe, Clay Davenport, and Joe Sheehan. Of these, three of them -- Kahrl, Jaffe, and Davenport -- correctly picked the order of finish in the AL West, while Joe Sheehan got the first two finishers in the right order but had the A's grabbing a Wild Card berth.

But it's the others in the BPro dogpile that have worked up the hackles of Matt Welch and others around the Halosphere. Take this Griddle interview with Rob Neyer, where Neyer once more dismisses the Halos, saying, "I don't find the AL West at all difficult; on paper the A's are easily the best team in that division, and they might be the best team in the major leagues." Or, how about ESPN's power rankings (good luck if you're running Firefox on that one, by the way), which rate the Angels' offseason a tepid 11th overall in the majors?

I bring this up in part because it's part and parcel of a theme that Matt Welch brought up in his Hardball Times Annual 2006 essay, "Getting With The Program". Matt argues that the sabermetricists have done themselves no favors by being so forlornly, consistently, pig-headedly wrong in their predictions, year in and year out, of the A's taking the division, league, and even World Series title.

In 2002, the year the Anaheim Angels would go on to win the World Series after racking up the best Pythagorean win-loss record in baseball, each and every one of the 11 Prospectus writers came to the exact same preseason conclusion: the team from Orange County would finish dead last.

Such are the many joys and delicious ironies of being both an avid Angels fan and consumer of sabermetric analysis. Year after year, Mike Scioscia's boys and their distasteful "productive outs" are picked to stumble; year after year, the widely scrutinized moves of Oakland's Billy Beane are picked to pay off with division titles, MVP Awards, even World Series rings.

Now, forget that the "productive out" correlates poorly to runs scored (something Larry Mahnken blasted in Hardball Times two years ago). It isn't even really something the Angels care about per se, as evidenced in Jeff Angus' Management By Baseball interview with Mike Scioscia last year, when Scioscia said that the two most important stats they care about are hitting with RISP and hitting with RISP and two out (RISP2) -- which is to say, they care about hitting in the clutch, and doing so in extremis.

The "Beaniacs", as some would call them, have for years said there's no such thing as clutch hitting as a statistically verifiable skill, or maybe more accurately, if there is, it requires a phalanx of statistical caveats so deep even an insurance company actuary couldn't find it. This got started by the research of Dick Cramer, and while there has been endless research on the topic, the distance between particular players -- Gary Sheffield, for instance, whom some say has this skill -- and the general mass of hitters is still quite great, and so to my knowledge, "hitting in the clutch" as a skill remains as solid as a mirage.

Or perhaps I should say, it remains as solid as the fog Bill James recently wrote about. In a thumb-in-the-nose at his most avid followers, a piece he wrote for the Baseball Research Journal entitled "Underestimating The Fog", James argues that Dick Cramer's discovery that clutch hitting is a chimera is as accurate as Columbus' claim to have discovered a new route to India:

Cramer was using random data as proof of nothingness and I did the same, many times, and many other people also have done the same. But I'm saying now that's not right; random data proves nothing and it cannot be used as proof of nothingness.

Why? Because whenever you do a study, if your study completely fails, you will get random data. Therefore, when you get random data, all you may conclude is that your study has failed. Cramer's study may have failed to identify clutch hitters because clutch hitters don't exist as he concluded or it may have failed to identify clutch hitters because the method doesn't work as I now believe. We don't know. All we can say is that the study has failed.

This is a very disturbing paragraph. He may be right about that, but I see no functional difference between the two. James here suggests that there may be clutch hitting, but we cannot yet prove it. Well, I reply, so what? We also cannot mathematically prove the existance of God, mermaids, and unicorns; does this now mean we get to (or have to) believe in all three? James, who is unquestionably a smart guy, here asks his audience to prove a negative, something that is logically impossible.

Worse, in pushing for a distinction between "the study has failed" and "there is no statistical proof of X as a skill" (where X may be "clutch hitting" or any of a dozen other similar issues within baseball), James has opened the door to all the snake oil peddlers like Buster Olney to tout their wares. If Nate Silver, Keith Woolner, Keith Davenport and the like over at Baseball Prospectus have, from time to time, been the victim of overzealous jacket blurbs (PECOTA, "deadly accurate"? What, it's a weapon of mass instruction?), at the same time at least they can say whether a player's future performance fit within the error bands the projection gave. It's a flawed system, to be sure, but player projection is notoriously difficult. But to surrender some knowledge for unverifiable speculation -- and make no mistake, in the absence of proof, that is what the notion of RISP- and RISP2-hitting-as-skills are -- amounts to creationism, the "intelligent design" of baseball.

Update 5/21/2014: Corrected the link for the Bill James essay, no longer available as a PDF.


Comments:
I wish I could understand this! It sounds profound. I should have studied harder in college!
 
1. How is Bill James - who is critiquing the methodology - trying to prove a negative ina critique of a study designed to prove that something does not exist?

2. Agree w/ you about the fallibility of the RISP(1 & 2) data - Hopefully Joe Maddon took those methods to TB and Dino Ebell will guide Scioscia toward the light...
 
James' example at the end of his essay likens Cramer to a sentry trying to disprove the existence of a foreign army in a fog. That's proving a negative.
 
James is actually thinking like a scientist does. Good scientists know that you can never prove anything outright. All you can do is run an experiment and if it works they way you expected, you find out a little bit about the universe (that that particularl aspect follows currently accepted rules of the universe). If the experiment fails, that does not mean that what you are doing is disproven, it just means that you can't say anything for sure. Heck, it very well could be there was a random error (human error, bad data, bad experiment, etc.), that that experiment doesn't conform to the accepted rules of the universe because it is evidence of different rules of the universe, or that some similar but different experiment needs to be runto find out more accurately if that experiment should work or not.

James seems to be saying that there is a lot of unknown unaccounted for in current baseball statistics (unquestionably true) and that just because some people have shown using certain methodologies that you get nothing new, that does not mean that using other methodologies not yet figured out nothing new will be learned.

As a matter of fact, this is something I have hated about the SABR ideas since I first heard about them: they are always so sure everything they do is perfect and if it doesn't conform to current ideas, it is wrong. Well, keeping such a closed mind about these things will never allow the field to move forward properly. Go ahead and look at clutch hitting data with skepticism, but don't believe it can never be proven.
 
Josh, I'm not saying X (clutch hitting, whatever) can't be proven, but what I am saying is that the onus is on the purveyors of the notion of X-as-a-skill to prove such exists, not on those who have been unable to find X in the extant data. That's the key problem I have with James trying to frame the debate this way: he's made it so that all manner of hoary myths now get the benefit of a doubt simply because of the error bands.
 
Creationism is a theory that explains how the very universe was created; it's a fundamental as it gets.

Neither does clutch hitting claim to explain the whole of baseball. Both are willful rejections of well-known and tested models.

The Angels didn't spend this offseason looking for someone to make productive outs; they knew they needed more offense, especially power but also OBP, from 1B, DH & CF. Even though they didn't acquire anybody, they managed to make improvements in this regard.

This is probably the most important point here: the Angels may say they care mostly about RISP and RISP2 hitting, but they're not acting much on it.
 
Well said, Matt.

Also, Creationism itself is completely compatible with all scientific thoeries currently. Biblical creationism (the stuff that Carl Everett believes in) is not. Not to get too far off topic, but this is an issue that many Evolutionists (those that believe in the religion of evolution) and Intelligent Designists don't seem to understand. Creationism does not have anything to do with evolution. Saying you believe one does not exclude you from being able to believe the other. One explains the origin of the universe (and life) and the other describes how the current species have come to exist from original life. Evolution cannot possibly explain the origin of life or the universe itself, while Creation explains ONLY those things.

Basically, Creationism is not a willful rejection of a tested model: it is an explanation for a model that cannot possibly be tested at this time. And if you want to say that it doesn't change your argument (ie clutch hitting still cannot be tested at this time), then either accept that it has an unknowable impact on the game (it is one of the intangibles that aren't yet fully accounted for) or try to come up with another method that can attempt to measure it. If the latter fails again, you try a new method. You cannot say UNEQUIVOCALLY that something is impossible. If it were easy to find equations to explain everything, they would have already found the Unified Equation that explains everything in the universe.
 
It must be nice to be so certain of your own ideas. I live in James' fog...
 
Creationism explicitly rejects the scientific method. It is not a theory, not as any scientist understands it. Instead, it is a declaration, one that seeks to end discovery by planting the flag "God" at the end of it. It makes no testable hypotheses. I recently came across the best commentary on intelligent design I have ever seen, useful for its conciseness:

... in order to be considered properly by scientists for research, a hypothesis
must make testable, repeatable predictions. As far as my research has
discovered, Intelligent Design doesn't make any predictions, simply declarations. If it doesn't make testable predictions, it's nothing more than an idea (aka a hypothesis) and definitely doesn't elevate to the level of "theory", a term with a very rigorous definition in the scientific
community. From this standpoint, Intelligent Design and the Flying Spaghetti Monster are on equal footing in the view of science and neither should be taught in a science course worthy of the name.

 
Matt -- I'm not saying guys like Sheffield don't exist, all's I'm saying is that in general, it doesn't exist as a skill for the vast majority of players. Sheffield's rareness is something that's been shown, but it's largely beside the point. The Angels have assembled team after team that have been pretty weak on offensive firepower; they won their 2002 title by slugging well over their regular season heads. Getting wrapped up in RISP voodoo isn't going to help the team's offense.
 
Rob, I think Josh is addressing a more general belief that Christians have, as opposed to "intelligent design" which is a pretty specific telling of how they think things went down. He's saying that there are Christians out there who believe that God created all this we see around us, but that this omnipotent God was actually smart enough to design it in a logical, testable way, and that the bible maybe isn't quite so literal.

Josh, to explain Rob's lack of differentiation, I grew up in Arkansas at a time when folks who believed in "Intelligent Design" called themselves "Creationists." The current theory is simply a rehash of the stuff they tried to get through the state legislature in the 70's and 80's. So around here, one term equals the other, just different decades. We figure folks who believe that God created everything, but maybe the Bible isn't literal, are just called "Christians."

As for clutch hitting, it's like porn. I can't define it, but I know it when I see it. ;-) (And it's fun, too.)
 
It's like cold fusion: maybe it exists, maybe it doesn't, but you sure as hell aren't going to charge customers for electricity based on it any time soon.
 
Don't confuse SABR with sabermetrics. While the latter may have had its origin in the former, SABR is about historical research, not statistical obsession.
 
Mmorgaine, thanks.

As far as Creationism being a testable hypothesis and scientific theory, I never said that. But just because at the moment there is no testable hypothesis does not mean that eventually there will not be methods to do so. What outer space is made up of was not a testable hypothesis for a long time, but that doesn't mean it wasn't a sceintific question that couldn't eventually be tested.

The question of Creationism is one similar to the clutch hitting one: just because you can't think of how it can eventually be tested, doesn't mean it is untestable or that it doesn't exist. All it means is that we do not know at this time and we cannot test it yet. (And if you are wondering how you could possibly test Creationism, well, you could invent time travel and go to the beginning of time and observe, but that's not yet a method available to us).

And as for the cold fusion analogy: just because we don't know if cold fusion can exist or if we will ever figure out how to make it work does not mean that it will forever be impossible and that we shouldn't keep trying to figure it out.
 
As far as Creationism being a testable hypothesis and scientific theory, I never said that. But just because at the moment there is no testable hypothesis does not mean that eventually there will not be methods to do so.

But how could there ever be such? What testable hypothesis do you arrange to divine an all-knowing creator? And then, how was that creator created? It's unscientific silliness.

And as for the cold fusion analogy: just because we don't know if cold fusion can exist or if we will ever figure out how to make it work does not mean that it will forever be impossible and that we shouldn't keep trying to figure it out.

But there's a huge difference between investigating cold fusion and planning on driving to Phoenix in a car with a cold fusion engine in it. That latter is the position the Angels have placed themselves in.
 
My point was exactly that. We don't even know how to begin testing the hypothesis of a Creator. But that doesn't mean that future generations won't be able to. We take many things for granted today because we are using knowledge that has built upon for 5000+ years. It is difficult to know how technology will differ in 150 years (just 150 years ago, there was no such thing as flying and driving cars...heck, go back around 200 years and there was no mechanism for powered travel other than biological organisms). Just because we don't know how to frame the questions or test the hypotheses don't mean that it can never be done. Just because you (or I) can't think of a why to test Creationism doesn't mean that no one will EVER think of a way to do it. A large part of science is creativity and skepticism. And skepticism includes skepticism of currently accepted facts.

And just because you haven't figured out how clutch hitting works doesn't mean that someone somewhere hasn't. You may think that that person would want to publish said results, but if freeing that information up will hurt you, then you won't do it. Just because you don't have the data in front of you saying that RISP matters does not mean taht someone else hasn't figured it out and just doesn't want to share it. With the cold fusion analogy: just because you don't know how to create cold fusion and you can't see it, it doesn't mean that someone somewhere isn't already driving to Phoenix with a cold fusion engine.
 
If the Angels really sought to build a team based upon RISP, they would have given Bengie Molina a long-term contract. Here's a guy who, throughout his career (not just a week or even a season), has consistently performed much better in the clutch than overall. Whether that is the product of any particular skill is the (at this point unanswerable) question.

I have my own belief about clutch hitting, which is that it involves some unique skills, but cannot be quantified. Basically, I believe that players react differently to "clutch" situations based upon their personality, experiences, and everything that makes us human. Some have the ability to focus better than others, and better than they might be able to in non-clutch situations.

Thus, it's not quite accurate to say that there are clutch hitters, but rather that some players might have a particular skill set that is more likely to produce success under specific conditions. E.g., a good-contact, fastball hitter who tends to hit the ball where the fielders aren't when there are runners on base.
 

Post a Comment

Note: Only a member of this blog may post a comment.



Newer›  ‹Older
This page is powered by Blogger. Isn't yours?
Google

WWW 6-4-2