Detecting the Effects of Goalie Fatigue

This is a much-abridged version of a research paper of mine, currently in submission.

Do goalies fatigue? More specifically, does their performance worsen as their workload accumulates?

By examining the play-by-play data of the NHL's real-time event tracker, I can find the outcome and details of any individual shot attempt, including the number of shots or shot attempts that a given goalie has withstood up to this point. I aggregated the results from these shots to estimate the save percentage of starting goalies against their 1st, 2nd, ... , up to 50th shot of the game.

For example, of the 7822 game-sides with that had a 10th shot on net against either team's starting goalie, 694 scored goals, so the estimated save percentage is 1 – (694/7822), or 0.9113. These simple estimates are shown in the solid black line of the first figure. I also apply a smoothed fit, weighted by sample size, which is shown in the red dashed line.

It appears that save percentage starts high, reaches a minimum between the 20^th and 25^th shots faced before improving slightly. Using only the number of previous shots as a guide, each of these later shots is 20-30% more likely to score a goal than the first shot (9% chance instead of a 6.5-7% chance) . A goalie's endurance limitations is but one of many possible explanations for this phenomenon.

The same pattern appears when considering previous shot attempts faced, rather than just shots. The next figure shows the same analysis, but including missed and blocked shots in the 'previous shots' count.

In the first 60 minutes, a starting goalie faces an average of about 28 or 29 shots, so anything beyond the 30^th shot may indicate poor defensive skaters, or lots of low quality shots, or simply lots of penalties against that goalie's team.

The next thing I looked at was the shot quality of the 1st, 2nd, ... , 50^th shot each team makes. Although there are many factors that go into the quality of a shot, king among these is distance to the net. This next figure shows the relationship between the save percentage and distance from the net to the shot. The solid black line represents the simple estimator of save percentage of shots from a given distance, binned into distance groups of five feet. The dashed red line represents a smoothed, weighted fit. The chance of a shot becoming a goal diminishes linearly until about 60 feet from the net, after which distance matters much less*.

From this model, I find the predicted save percentage for each shot, based on its distance. Then I took the mean of the predicted save percentages for each shot number to get an estimate of the save percentage for each save number based solely on shot distance. If shot distance, a surrogate for shot quality, can explain the differences in save percentage by shot number, I would expect the estimate based on distance, the red line in this next figure, to follow the same pattern as the estimate based on shot number, the black line. Although the expected save chance is biased upwards, likely due to the pattern of the missingness, the disjoint between distance-based shot quality and the number of previous shots is clear. The flat red line implies that shot quality stays the same across shot number, at least when averaged across many games.

Is this phenomenon universal to all starting NHL goalies, or are some able to hang onto their peak skill longer?

We also stratified by global goalie skill and fit the curve to each tier. For the games observed, I found the number of saves and goals allowed for each starting goalie. I removed those goalies that faced fewer than 400 shots during the observed time, leaving 123 goalies. These 123 goalies were split into 3 tiers of equal size, according to their save percentages during the observed games. As before, I used a simple estimator and weighted smoothed fit of save percentage against the 1st, 2nd, ... , 50th shot. This time, analysis is done separately for each tier in order to estimate save percentage of the best third of goalies, the middle third, and worst third. 

 

The results are shown in this last figure. The simple estimators of any given shot are a mess of overlaps, so they're faded to grey. Instead, consider the red curves for each tier. Aside from the differences after the 40^th shot, which could easily be the result of instability due to data sparsity, all three regression-based predictions follow the same fundamental pattern. Better goalies appear to reach their worst performance earlier, and the difference between their initial and worst performances appears smaller. This suggests that what separates elite goalies from the rest is not just their high level of performance, but the consistency of that high level through a game.

*About 5% of the shot locations are missing from the data, and that a disproportionate number of missing locations were from shots that were likely near to the goal, such as wraparounds (the type of shot is still recorded, even when the location is missing).

This post strongly inspired by Rob Vollman's Hockey Abstract, 2014 edition. Specifically question 1 in 'Goaltending Q&A', 'how are goalies affected by workload'.

Statistical Thesaurus update, and generating cross-references in R

The post is about an expansion on the statistics thesaurus that was started in this post. The entries in  The Oxford Dictionary of Statistical Terms, 6th edition, edited by Yadolah Dodge was an inspiration to update the thesaurus, and was used as a 'to do list' of terms to include. Compared to the Oxford Dictionary, this thesaurus is much less technical, and more cross-referenced. Perhaps with time, it could be a workable companion volume.

A Long term vision for a Master's Degree in Data Science Program

I was recently asked to write a statement of vision for a potential one-year coursework master's program. A colleague was kind enough to look at it for me; he said was 'a bit too bombastic'.

This statement has already been submitted to its intended destination.





----------

Successful data scientists need to be able communicate not just verbally and in writing, but also
visually by way of graphs, dashboards, and animations. They need to have working knowledge of
modern database languages like SQL and big data architectures like Hadoop. They need to be able to
determine when to use modern statistical, machine learning, and optimization methods like the LASSO, neural networks, and random forests.

But successful students won't just have a practitioner's knowledge of these tools, because these tools
will be replaced eventually. They also need the depth in their backgrounds to evaluate and adapt to
additional systems and methods as they become available.

Therein lays the challenge: the demands upon a data scientist are broad, whereas a Master's degree is
typically a structured, focused, deep exploration of a single field. There simply isn't enough time to
cover all that's necessary to develop a prospective student starting with a bachelors degree in
Mathematics or Computer Science into a consummate data scientist in ten months. 

Some topics will need to be sacrificed for the sake of brevity, but what? Different students will bring
diverse skills and affinities into such a degree program, and they will be good judges of what they
should focus on. However, a degree is essentially a set of requirements, which is another way of saying it's a set of guarantees. The better defined those requirements are, the clearer the guarantee of skill that the bearer of such a degree brings to future employers.

In the face of program that will inevitably be stretched thin across many competencies, there are two
competing needs: the need for students to develop the subset of these competencies that maximize their personal return, and the need for standardization across the program to make its value and quality obvious to all stakeholders. To reconcile these two needs, I envision a specialization system. Graduates from the ideal Master in Data Science program will also graduate with one of four specialties: visual analytics, databases, methodology, or algorithms.

Under this specialization program, all MDS candidates will be required to take a core of data scraping, imputation, R or SAS programming with an SQL component, modern regression such as GLMs, and scientific writing. This totals 15 graduate credits. The remaining 9 credits form a specialty.

Database experts would be most akin to software engineers. The courses for this specialty would
include one focused on the extract-transform-load paradigm, one focused on handling big data tools
like Hadoop. Graduates from this specialty would be expected to be able to implement automated tasks
for gathering, cleaning, and summarizing information from the web or some other digital sensor.

Methologists would take applied statistical courses like design of experiments, sampling, dimension
reduction, time series, and spatial statistics. What separates this specialty from a Master's degree in
statistics is the lack of emphasis on proofs. Students in these courses need not understand why a
method works, only how to assess through diagnostics and checklists that it is working and when it is
appropriate.

Visual analytics specialists would take courses focused on user interfaces and communication,
including graphing and data cartography, dashboards, and additional writing work such as survey
design. Graduates from this specialty would be expected to demonstrate familiarity with popular
database interface like Jaspersoft and Tableau.

Algorithm experts would focus their additional coursework on new ways to find meaning from the
data deluge. Their corpus would include machine learning, optimization methods like quadrature and
simulated annealing, text processing concepts such as regular expressions and edit distance, image
processing, clustering, compression and information theory.

A graduate with skills in any one of these four specialties fits nicely under what we know as a data
science today.  This vision is a grand one, and far too large for a new master's program to take on, but
it's the endgame i have in mind for this program.

Reading Assignment - Designing Survey Questions

In one of the second year service courses I taught this semester, some people were unable to do the participation assignment for non-academic reasons. This means I needed an alternative assignment, which gave me a chance to field test the following reading assignment.

This is based on Chapter 8 of the book Successful Surveys - Research Methods and Practice by George Gray and Neil Guppy. The chapter is "Designing Questions of the book Successful Surveys."

Analyzing Jeopardy in R – The College Championship effect.

How much easier or harder is the Jeopardy! College Championship than regular Jeopardy? In the College Championship Jeopardy!. In this tournament, 1 undergraduate student from each of 15 U.S. post-secondary schools compete in a tournament of elimination rounds. The intended audience of the categories are different than it is for regular Jeopardy shows. Some of the clues referred to new and popular video games, and neologisms like 'woke'. 

To me, the College Championship questions were qualitatively easier, but for the sake of tracking, I want to measure that. The following method will also work if you find the College Championship questions harder than those from the regular show.