diff --git a/doc/source/_static/.gitignore b/doc/source/_static/.gitignore new file mode 100644 index 0000000..dab1498 --- /dev/null +++ b/doc/source/_static/.gitignore @@ -0,0 +1 @@ +!*.png \ No newline at end of file diff --git a/doc/source/_static/validation_plot.png b/doc/source/_static/validation_plot.png new file mode 100644 index 0000000..174d2a9 Binary files /dev/null and b/doc/source/_static/validation_plot.png differ diff --git a/doc/source/index.rst b/doc/source/index.rst index 99502c5..d99eb22 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -78,59 +78,17 @@ Once data is loaded, using the model to predict WP is easy: Why NFLWin? -------------- -NFLWin is far from the first effort to compute Win Probabilities for -NFL plays. Brian Burke at Advanced NFL Analytics was one of the first -to popularize WP in recent years, `writing about the theory behind it -`_ -as well as providing `real-time WP charts for games -`_. Others -have picked up on this technique: Pro Football Reference (PFR) has their -`own model -`_ as well -as an `interactive WP calculator -`_, -and the technique is offered by `multiple -`_ analytics `startups -`_. It's one of the most -important tools in the NFL analytics toolchest. - -So why create NFLWin? Well, to put it bluntly, while there are many -other analysts using WP, they're not publishing their methodologies -and algorithms or quantifying the quality of their results. This -information is critical in order to allow others both to use WP -themselves but also `to validate the correctness of the models `_. Brian Burke has never discussed any of the details of -his WP model (and now that `he's at ESPN `_, that situation is unlikely to -improve any time soon), and analytics startups are (unsurprisingly) -treating their models as trade secrets. PFR goes -into more detail about their model, but it relies on an Estimated -Points model that is `not explained in sufficient detail to reproduce -it -`_. - -Possibly the best description of a WP model comes from Dennis Lock and -Dan Nettleton, -`who wrote an academic paper outline his approach -`_. Lock -and Nettleton's -paper provides information regarding the data source used to train the -model, the type of model used, the software used to build the model, -and some statistics indicating the quality of the model. It even -includes a qualitative comparison with Brian Burke's WP estimates. This is far -and away the most complete, transparent accounting of the guts of a WP -model and is laudable. However, as often happens in academia, none of -the code used to build and test their WP model is available for others -to use; while in principle it would be possible for anyone to recreate -their model to build on or validate their work, this would require -building their entire pipeline from scratch based off of prose -descriptions. +Put simply, there are no other options: while WP models have been +widely used in NFL analytics for years, the analytics community has +almost totally dropped the ball in making these models available for the +general public or even explaining their algorithms at all. +For a (much) longer explanation, see the `PhD Football blog +`_. Resources --------- - -Default Model metrics - +------------ .. toctree:: :maxdepth: 2 diff --git a/doc/source/model.rst b/doc/source/model.rst index 5ddcf6a..63f2ba2 100644 --- a/doc/source/model.rst +++ b/doc/source/model.rst @@ -1,2 +1,183 @@ Creating a New WP Model ============================== +While NFLWin ships with a fairly robust default model, there is always +room for improvement. Maybe there's a new dataset you want to use to +train the model, a new feature you want to add, or a new machine +learning model you want to evaluate. + +Good news! NFLWin makes it easy to train a new model, whether you just +want to refresh the data or to do an entire refit from scratch. We'll +start with the simplest case: + +Default Model, New Data +----------------------- +Refreshing the data with NFLWin is a snap. If you want to change the +data used by the default model but keep the source as nfldb, all you +have to do is override the default keyword arguments when calling the +:meth:`~nflwin.model.WPModel.train_model` and :meth:`~nflwin.model.WPModel.validate_model` +methods. For instance, if for some insane reason you wanted to train on the 2009 and 2010 regular +seasons and validate on the 2011 and 2012 playoffs, you would do the following: + +.. code-block:: python + + >>> from nflwin.model import WPModel + >>> new_data_model = WPModel() + >>> new_data_model.train_model(training_seasons=[2009, 2010], training_season_types=["Regular"]) + >>> new_data_model.validate_model(validation_seasons=[2011, 2012], validation_season_types=["Postseason"]) + 0.14963235412213988 + +If you want to supply your own data, that's easy too - simply set the +`source_data` kwarg of :meth:`~nflwin.model.WPModel.train_model` and +:meth:`~nflwin.model.WPModel.validate_model` to be a Pandas DataFrame of your training and validation data (respectively): + +.. code-block:: python + + >>> from nflwin.model import WPModel + >>> new_data_model = WPModel() + >>> training_data.head() + gsis_id drive_id play_id offense_team yardline down yards_to_go \ + 0 2012090500 1 35 DAL -15.0 0 0 + 1 2012090500 1 57 NYG -34.0 1 10 + 2 2012090500 1 79 NYG -34.0 2 10 + 3 2012090500 1 103 NYG -29.0 3 5 + 4 2012090500 1 125 NYG -29.0 4 5 + + home_team away_team offense_won quarter seconds_elapsed curr_home_score \ + 0 NYG DAL True Q1 0.0 0 + 1 NYG DAL False Q1 4.0 0 + 2 NYG DAL False Q1 11.0 0 + 3 NYG DAL False Q1 55.0 0 + 4 NYG DAL False Q1 62.0 0 + + curr_away_score + 0 0 + 1 0 + 2 0 + 3 0 + 4 0 + >>> new_data_model.train_model(source_data=training_data) + >>> validation_data.head() + gsis_id drive_id play_id offense_team yardline down yards_to_go \ + 0 2014090400 1 36 SEA -15.0 0 0 + 1 2014090400 1 58 GB -37.0 1 10 + 2 2014090400 1 79 GB -31.0 2 4 + 3 2014090400 1 111 GB -26.0 1 10 + 4 2014090400 1 132 GB -11.0 1 10 + + home_team away_team offense_won quarter seconds_elapsed curr_home_score \ + 0 SEA GB True Q1 0.0 0 + 1 SEA GB False Q1 4.0 0 + 2 SEA GB False Q1 30.0 0 + 3 SEA GB False Q1 49.0 0 + 4 SEA GB False Q1 88.0 0 + + curr_away_score + 0 0 + 1 0 + 2 0 + 3 0 + 4 0 + >>> new_data_model.validate_model(source_data=validation_data) + 6.8222808634589248e-35 + +Building a New Model +-------------------- +If you want to construct a totally new model, that's possible +too. Just instantiate +:class:`~nflwin.model.WPModel`, then replace the +:attr:`~nflwin.model.WPModel.model` attribute with either a +scikit-learn `classifier +`_ +or `Pipeline +`_. From +that point :meth:`~nflwin.model.WPModel.train_model` and +:meth:`~nflwin.model.WPModel.validate_model` should work as normal. + +.. note:: + If you create your own model, the + :attr:`~nflwin.model.WPModel.column_descriptions` attribute will no longer be + accurate unless you update it manually. + +.. note:: + If your model uses a data structure other than a Pandas DataFrame, + you will not be able to use the ``source_data="nfldb"`` default + kwarg of :meth:`~nflwin.model.WPModel.train_model` and + :meth:`~nflwin.model.WPModel.validate_model`. If you want to use nfldb + data, query it through :func:`nflwin.utilities.get_nfldb_play_data` + first and convert it from a DataFrame to the format required by your model. + +Using NFLWin's Preprocessors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +While you can completely roll your own WP model from scratch, NFLWin +comes with several classes designed to aid in preprocessing your +data. These can be found in the appropriately named +:mod:`~nflwin.preprocessing` module. Each of these preprocessors inherits +from scikit-learn's BaseEstimator class, and therefore is fully +compatible with scikit-learn Pipelines. Available preprocessors +include: + +* :class:`~nflwin.preprocessing.ComputeElapsedTime`: Convert the time + elapsed in a quarter into the total seconds elapsed in the game. +* :class:`~nflwin.preprocessing.ComputeIfOffenseIsHome`: Create an + indicator variable for whether or not the offense is the home team. +* :class:`~nflwin.preprocessing.CreateScoreDifferential`: Create a + column indicating the difference between the offense and defense + point totals (offense-defense). Uses home team and away team plus + an indicator giving if the offense is the home team to compute. +* :class:`~nflwin.preprocessing.MapToInt`: Map a column of values to + integers. Useful for string columns (e.g. a quarter column with "Q1", "Q2", etc). +* :class:`~nflwin.preprocessing.CheckColumnNames`: Ensure that only the desired data gets passed to + the model in the right order. Useful to guarantee that the + underlying numpy arrays in a Pandas DataFrame used for model + validation are in the same order as they were when the model was + trained. + +To see examples of these preprocessors in use to build a model, look +at :meth:`nflwin.model.WPModel.create_default_pipeline`. + +Model I/O +--------- +To save a model to disk, use the +:meth:`nflwin.model.WPModel.save_model` method. + +.. note:: + If you do not provide + a filename, the default model will be overwritten and in order to + recover it you will need to reinstall NFLWin (which will then + overwrite any non-default models you have saved). + +To load a model from disk, use the +:meth:`nflwin.model.WPModel.load_model` class method. By default this +will load the standard model that comes bundled with pip installs of +NFLWin. Simply specify the ``filename`` kwarg to load a non-standard +model. + +.. note:: + By default, models are saved to and loaded from the path given by + :attr:`nflwin.model.WPModel.model_directory`, which by default is + located inside your NFLWin install. + +Estimating Quality of Fit +------------------------- +When you care about measuring the probability of a classification +model rather than getting a yes/no prediction it is challenging to +estimate its quality. This is an area I'm actively looking to improve +upon, but for now NFLWin does the following. + +First, it takes the probabilities given by the model for each play in the +validation set, then produces a `kernel density estimate +`_ (KDE) of all +the plays as well as just the ones that were predicted +correctly. The ratio of these two KDEs is the actual WP measured +from the test data set at a given *predicted* WP. While all of this is +measured in :meth:`~nflwin.model.WPModel.validate_model`, you can plot +it for yourself by calling the +:meth:`~nflwin.model.WPModel.plot_validation` method: + +.. image:: _static/validation_plot.png + +From there NFLWin estimates if the deviations between predicted and actual WP are +statistically significant - that's what is actually returned by :meth:`~nflwin.model.WPModel.validate_model`. This is decidedly not awesome - even if the deviations aren't +significant you can't prove that your model is correct; rather all you can +say that it is "not inconsistent" with the validation data. If anyone +has an idea for how to do this better I would welcome it enthusiastically.