Off-season: Explore the unlocked site! Projections reflect re-calculations of 2022, based on end-of-season knowledge.

By Subvertadown

Tagged under

Understanding StatisticsSome of you requested me to describe what is "overfit". So here's ...really probably... more than you really wanted to know! Anyway, I hope like 3 of you out in the world will get something from this discussion. Overfit it is an issue I treat seriously, to try to deliver the best possible recommendations. I strongly believe my efforts to address overfit make all the difference.

**What is overfit?** In my own words, an overfit is a feature in a model that causes wrong predictions-- despite the fact that the same feature well describes past trends.

**Are there different kinds of overfit?** Yes, I'm glad you asked. I'm not dealing with the classic example of polynomial overfit (when you try to fit a trend to y=a+bx+cx2+...+zx25 when it could have sufficed to use y=mx+b). The type of overfit I'm weeding out is of the "multivariable" kind. When there are more variables than just "x", then issues of overfit get tricky.

**Example?** Here's an easy example, and also the easiest kind to eliminate.

Suppose you're modeling kicker fantasy points, and you test the variable of "game O/U". It works! Kicker-score = 7.6 + 0.12*(O/U). p-value an great 10-14. Then suppose you also check if-- instead of using O/U-- the kicker's own-implied-team score works. Answer... Yes! It works even better! Kicker-score = 2.8 + 0.21 * (teamscore). p-value even lower, 10-15.

Now suppose you get a clever idea, realizing you have 2 variables that seem statistically significant: ** Why not use both?** Voila: You get Kicker-score = 4.2 + 0.15*(teamscore) + 0.05*(O/U), and this "model" has better correlation than using just 1 variable (which is always true).

**What does that example show?** Many people would expect that using both variables (both significant when used independently) could yield some small advantage. Often wrong. In reality, using the "extra" variable makes a worse model, very much killing maximum accuracy.

**Well, that's easy to do. Is there a harder example?** Yes. Here's what happens more frequently than you might think... I will add 1 extra variable to a regression, and yay! It appears to pass a normal test for significance, but... including the variable still decreases predictive accuracy. Very frustrating, but not uncommon.

**How do you get rid of these "imposters"?** You can *often* use cross-validation to catch such sneaky, misleading variables.

**What's cross-validation?** It's a huge element to my work. Example (using "years"): **(1)** Perform a regression using data that excludes 1 certain year, e.g. 2020. **(2)** Use the resulting equations to simulate the excluded year (2020) **(3)** Ask "Would steps 1&2 produce more accuracy if I include a certain variable, or exclude it? **(4)** Repeat this test for 2019, 2018, etc. This process helps weed out variables that do pass the significance test but do not actually have predictive value.

**So you just do something like that then... Isn't that enough?** No, it's not always enough. That's a shame (because I spend a lot of time on this!). The reasons could fill a whole chapter. But let's keep it brief-- It ** is** kind of interesting, but this is the part that sucks hours and hours from life.... In my experience, there are a few types of "imposter" variables.

**When does overfit come into Subvertadown models?** Each time I introduce (for screening) a wider array of inputs (as part of the search for "better" data), there's a potential need to weed out new overfit. Very few of the 100 new variables I last tested actually got added-- and only after passing tests for significance and cross-validation. Nevertheless, some of these were misleading. It feels a bit sad to cut out variables with p-value 0.001. But out they go.

**So how do I know when there's overfit?** In principle, most overfit could be removed during the model development process. But since that's not 100% foolproof, ** I refer to my in-season accuracy measurements** to try and detect if I need to address it. Once I establish the need, I can sniff it out with some analysis, or at least lean towards "underfit" to support minimum accuracy.

Tagged under

Quarterback , Understanding Statistics , Weekly StrategyTagged under

D/ST , Expectations , Quarterback , Understanding StatisticsTagged under

Exclusive , Expectations , Kicker , Quarterback , Understanding Statistics , Weekly StrategyTagged under

QuarterbackTagged under

QuarterbackTagged under

QuarterbackTagged under

Quarterback