I originally did not expect the time intensity involved with creating and maintaining models. At first, I thought some clever tricks could find patterns and quickly “crack the code”. Unfortunately, I learned there are huge time sinks.
For many followers, the Subvertadown project has been a story of continual improvement. The models require continuous maintenance, to consistently perform well. I recently read the Superforecasters book, which reaffirmed by belief: for the best forecasters, everything is in “perpetual beta”. There never comes a point of feeling it’s finished product! Every new season has seen a new modeling perspective: not just new variables, but new considerations around how to segment situational aspects.
And in-season, I have spent a lot of time analyzing the model results, so I could identify and explain what is causing an observed trend. (Why is team X rated lower than they were expected last week?) That has been automated and turned into a new feature on the site, which is another body of work to make improvements to features.
The Subvertadown project has always been about more than the models themselves: it has often been about trying to make sense of things— generating new ideas, a lot of trial-and-error, and then incorporating new learnings. It will always be a Work in Progress.
Tagged underExpectations , Current Season , Accuracy