Template for a technical report, with example rubric

A completed technical report might look like this:

1. Executive Summary

2. Introduction / Problem statement

3. Methods

4. Results

5. Conclusion / Discussion

 Executive Summary: The TL;DR

This is the LAST thing that you should write. This would be the tl;dr of the technical report. “tl;dr” stands for “too long; didn’t read”. More formally this is called the executive summary, which means ‘if this report was given to a major decision maker, whom has tons of things they need to know already, what would you like them to know from the report that can be reduced to 100 words or fewer.’

Here you should write the research question as shortly as you can, one main result, and the name of the main method used. Nothing from the discussion / conclusion section is needed here.

Introduction:

The introduction typically follows a close formula.

1. Describe the research problem or state the research questions that were posed. If you can, tell why this research problem is important. The explanation of importance doesn’t have to be too specific to the research problem. If you are working with data about a medical problem, mention that many people suffer this medical problem; in a research paper, this is a good opportunity to cite a well-known related paper that has found the scope of the problem for you.

If you don’t know why a problem is important and a quick literature search won’t tell you, leave the problem’s importance to a co-author whose expertise is more suited to this part. It’s much better to admit you don’t know something than to say something wrong.

2. Describe each section in the paper or report in very short detail. (e.g. “in the methods section, we describe the data cleaning and the regression tree method that we used. In the results section, we describe the goal scoring rate of different hockey players. In the discussion section, we follow up with a comparison of this method to an older, more traditional one.”)

Methods

The methods: What did you do to get these results?

If this were a field science, you would list the days and describe the conditions under which you went out into the field and gathered information (e.g. ‘we collected our samples on sunny days in the North Okanagan valley between June 10^th and September 20^th, 2015’). In a data science, you would instead describe the dataset that you used, its format and size, and key variables and features (e.g. ‘We gathered the data from NHL.com’s event-tracking database using the nhlscrapr package along with our own patch, The data we collected included each goal, shot, hit, penalty, and faceoff recorded in each regular season game from October 2012 to April 2017’)

This is where the bulk of your writing should be. About 50% of your report will be the methods section. You don’t need to explain the entire data cleaning process, but you should mention where the data came from, and the tools / software that were used. It’s also good practice to mention when the data was taken (especially in the case of news reports which may be updated, altered, or archived such that scraping may produce different results later).

If there were any judgement calls in your data cleaning process, such as…

- what was done about extreme and influential cases,

- how problematic variables were used,

- how tuning parameters for complex methods were selected, and

- how missing values were either filled in or explained away,

…these should be included as well.

In short, you don’t have to give everything away, but an expert with the same software and data access should be able to recreate what you did.

A methods section serves two purposes: first to give legitimacy to your results. If you show results without explaining how you got them, a reader might assume that the results were invented or made up. With a methods section, the reader should be able to see a logical path between the data and the results.

After the data preparation is explained, describe the model you selected or the process you used to select the model. If you just did linear regression, say that. If you used a random forest, or the LASSO, or stepwise regression, say that instead.

Normally, you only need to include the final method that you decided upon. However, there is a good chance that the method you used wasn’t the only method that you tried. In a research paper, you wouldn’t necessarily mention these ‘dead ends’ because paper length is limited by the journal. In a technical report (or a thesis) these other approaches are useful to help you justify your choice and that alternatives were considered. You can explain why these rejected methods didn’t work or what about the results they produced was bad. Don’t overdo these dead-end explanations. The reader is much more interested in what you did and what worked instead of what didn’t work, typically.

Example: “After an exploratory analysis, we tried to classify events using random forests, dimension reduction, and neural nets. We decided to further pursue neural nets because they produce models with much lower out-of-bag errors than other approaches.”

Results

It’s easiest to write the results first, even though they don’t appear first. Any tables of figures you want to show, make these as soon as the analysis work is done. Talk about your results a little. Explain the importance of any tables and figures; why are they there?

Mention the general trend (e.g. ‘there is a negative, non-linear trend between playing time per game and shots against goal’), and any notable observations (‘however, the New Jersey Devils break this trend’)

You don’t need to write much here. The charts should explain themselves.

Discussion  / Conclusion:

In a technical report, this is where you take the results and give them meaning in the context of the research questions that were in the introduction. You can also quickly summarize what you did.

In a journal paper or a thesis, this section might also include future research questions that could be answered with more data or by a different analysis. A technical report should be more self-contained, and allusions to the further work are not required.

In every case, no new information about the project should in introduced in the conclusion. If you have an interesting finding, it should be in the results. If that interesting finding doesn’t fit with the rest of the results, a new subsection for it can always be made, but keep it out of the discussion section.

Remember, when giving context to the results, don’t reach beyond your expertise. If the data is genetic, and you are not a geneticist or biologist, do not make conclusions about the importance of a gene. Often statistical publications are co-authored with subject experts; let those experts write about their topics and stick to the data analysis.

Example Rubric: Total out of 100

Length	3-6 full pages	10
	7 pages	8
	8 pages	6
	More than 8	0
	2.5 pages	8
	2 pages	4
	Less than 2 full	0
		Possible 10	/10
Grammar	Start with 10, any OBVIOUS grammar or spelling mistakes, reduce this by 2. Minimum 0/10
		Possible 10	/10
Executive Summary	Name of main method included	3
	Primary finding described	7
		Possible 10	/10
Introduction	Describes the research problem	6
	Makes a case for the its importance	4
		Possible 10	/10
Methods	Describes the data used	5
	Describes the data preparation and/or data cleaning	5
	Describes any decisions / judgement	5
	Describes method used	10
	Justifies choice of method	5
		Possible 30	/30
Results	Summarizing data through text, table, or a figure	10
	Description of a general trend	5
		Possible 15	/15
Conclusion	Ties results back to research question	10
	Does NOT include any new information that would be better in the results or methods sections.	5
		Possible 15	/15
	TOTAL		/100

AlphaZero, Stockfish, and flexibility regarding chess variants.

Recently there was a high-profile set of matches between reigning champion chess AI stockfish and a newcomer called AlphaZero. AlphaZero was created with the same deep learning System that created AlphaGo, an AI that beat the world's best at the game Go. In the 50 matches that AlphaZero played as white, it won 24 of them and drew on the other 26. In the 50 games that it played as black it won three of them and drew on the other 47.

This advantage towards the white player may seem startling, however it's not out of line with other matches between artificial intelligence programs at the world-class level, nor is it out of line between matches between world-class human players. Stockfish, which evaluates positions in a chess game in terms of pawns of advantage starts the game with an advantage towards white of 0.1 pawns. AlphaZero, on the other hand, has no idea how many pawns of advantage white has, because it looks at the game holistically which is a radically different method of analysis compared to other modern AIs.

Part of the reason I bring up issues of artificial intelligence is to look at how well the these various systems will carry over to different chess variants rather than just the orthrodox version of the game.

Let's start with Stockfish: Stockfish is a system very much like Deep Blue and many of the other ones that came between it and Stockfish. The difference being that Stockfish is open source, meaning anyone can examine the code and edit it. This and many of the artificial intelligence programs that came before it run on a minimax principle, meaning that they try to choose the move on the assumption that their opponent will choose the best counter move in response to it, thus they try to pick the move which has the worst best counter-solution. (They try to minimize their opponent's maximum move quality.

To simplify, consider this abstract game. You have two options: Option A allows your opponent to score 5 points. Option B allows your opponent to choose between a move that scores 6 points, and a move that scores 2 points. Assuming that your opponent will choose their best move, your best choice is to select Option A, because it limits their score to 5. The fact that Option B provides the possibility for your opponent to score only 2 points is irrelevant. This is the minimax principle.

Most of what a traditional chess AI does when selecting a move is to evaluate a particular position is worth in terms of some abstract score, such as 'number of pawns'. The value of pieces is straightfoward: a pawn is worth approximately 1, a bishop or knight is worth about 3, and a queen is worth roughly 9. However the position of these pieces also matters. Having a piece in the middle or able to reach the middle at any point is worth a premium. A 'passed pawn', or one which has no opposing pawn directly ahead of it, is worth more than it would otherwise be, because of its greater potential to be promoted. The alpha-beta algorithm (not related to AlphaZero) contains set of parameters which decide how much each board piece is worth on each square. Different machine learning methods such as neural networks can be used to determine what these parameters should be.

For variants of chess that are very close to the original game such as Chess 960 (a.k.a Fischer Random Chess) or Really Bad Chess, which both feature 8 by 8 grids, 16 pieces per side, and only the orthodox six pieces, an AI using the alpha-beta algorithm should be able to play such games with few if any complications.

These AIs work even after pieces have been removed from the game so variants that use fewer pieces don't produce any difficulties either. In practice, variants with different board sizes are different arrangements such as Martin Gardener's mini chess or Romanchenko Chess (shown in the figure, source: Jocly) work well too as long as the value of any squares beyond the board are hard-coded to zero. This also means in practice that an alpha-beta algorithm can produce a viable chess AI on a board that is not a perfect square or rectangle. However it can increase the computational load the non-viable squares are considered, as they are in Jocly's implementation of alpha-beta on Romanchenko Chess.

Some systems, including Deep Blue, take advantage of chess literature, specifically for the orthodox game also take advantage of openings and their reputations methods for winning particular and games such as when you have a rook and a bishop against an opponent who just has a rook. However, after the opening and before the end game it's pretty much alpha-beta all the way. [1]

Variants that included new pieces such as fairy chess, or non-linear board movement such as Smess, the Ninny's Chess, can also be supported by AI programs that uses the alpha-beta algorithm. However these programs will need additional manual training to be able to evaluate the value of different pieces and space.

AlphaZero works on an entirely different principle; it does not assume that its opponent is the best possible opponent, one which will make the best possible counter move. Instead, AlphaZero evaluates a candidate position by simulating games of weighted random moves starting from the position to be evaluated. The evaluation is simply the proportion those random-move games that win* from AlphaZero's side. It evaluates the position this way for each move that it could make, and simply chooses the move that results in the best win proportion.

In these simulation games that AlphaZero uses, the weighting of the moves is based on moves that are likely to lead to a win based on games that AlphaZero played against itself. For example, AlphaZero may assign more weight towards a move that takes a piece over one that doesn't. It may also assign greater weight towards moves that give it control of the centre of the board. But these weight assignments would not be the result of any human supervision.

Similarly, AlphaZero has no concept of chess theory such as openings or their refutations, and it doesn't have a book of endgames to rely upon. AlphaZero was trained simply by giving the system the rules of chess, and letting it play many games against different versions of itself. It's reasonable to assume from here that AlphaZero would be able to handle many chess variants without any additional modifications other than informing it of the new rules. Furthermore a very similar training system could be given to nearly any chess variant to produce an AI program that could play that particular game.

* More exactly, the evaluation is (Proportion of Wins) + 1/2*(Proportion of Ties)

[1] Beyond Deep Blue: Chess in the Stratosphere, Monty Newborn

Reflection on teaching a 400-level course on Big Data for statisticians.

This was the first course I have taught that I would consider a 'main track' course, in that the students were to learn more about what they were already competent in at the start. Most of the courses I have previously taught were 'service courses', in that they were designed and delivered by the Statistics Department in service to other departments that wanted their own students to have a stronger quantitative and experimental background (e.g. Stat 201, 203, 302, and 305). The exception, Stat 342, was designed for statistics majors, but is built as an introduction to SAS programming. Since most other courses in the program are taught using R or Python, teaching SAS feels like teaching a service course as well, in that I am teaching something away from the students' main competency, and enrollment is mainly driven by requirement rather than interest.

In my usual courses, I am frequently grilled from anxious students about what exactly is going to be on exams. Frequent complaints I receive in student responses are about how I spend too much time on 'for interest' things are not explicitly being tested on the exams. I've also found that I needed to adhere to rigid structure in course planning and in grading policy. Moving an assignment's due date, teaching something out of the stated syllabus order, changing the scope or schedule of a midterm, or even dropping an assignment and moving the grade weight have all caused a cascade of problems in previous classes.

Stat 440, Learning from Big Data, was a major shift.

I don't know which I prefer, and I don't know which is easier in the long term, but it is absolutely a different skill set. The bulk of the effort changed from managing people to managing content. I did not struggle to keep the classroom full, but I did struggle to meaningfully fill the classroom's time. I had planned to cover the principles of modern methods (cross validation, model selection, LASSO, dimension reduction, regression trees, neural nets), some data cleaning (missing data, imputation, image processing), some technology (SQL, parallelization, Hadoop), some text analysis (regular expressions, edit distance, XML processing), but I still had a couple of weeks that I had to fill at the end because of the lightning speed that I was able to burn through these topics without protest.

In 'big data', the limits of my own knowledge became a major factor. Most of what I covered in class wouldn't have been considered undergrad material ten years ago when I was a senior (imputation, LASSO, neural nets); some of it didn't exist (Hadoop). There are plenty of textbook and online resources for learning regression or ANOVA, but the information for many of the topics of this course were cobbled together from blog posts, technical reports, and research papers. A lot of resources were either extremely narrow in scope or vague to the point of uselessness. I needed materials that were high-level enough for someone not already a specialist to understand, and technical enough that someone well-versed in data science would get something of value from it, and I didn't find enough.

The flip side of this was that motivation was easy. Two of the three case studies assigned had an active competition component to them. The first such study was a US-based based challenge to use police data from three US cities. In this one, presentation was a major basis that the police departments would judge the results. As such, I had requests for help with plotting and geographic methods that were completely new to me. A similar thing happened with the 'iceberg' case study, based on this Kaggle competition. I taught the basics of neural nets, and at least three groups asked me about generalizations and modifications to neural nets that I didn't know about. (The other case study was a 'warm-up' in which I adapted material from a case study competition held by the Statistical Society of Canada. The students were not in active competition). At least 20% of the class has more statistical talent than I do.

In order to adapt to this challenge and advantage, I changed about mid-semester from my usual delivery method of a PDF slideshow to one of commentary while running through computer code. This worked well for material that would be directly useful for the three case study projects, such as all the image processing work I showed for the case study on determining the difference between icebergs and ships. It wasn't as good for material that would be studied for the final exam. I went through some sample programs on web scraping, and the feedback wasn't as positive for that, and the answers I got on the final exam for the web scraping question were too specific to the examples I had given.

A side challenge was the ethical dilemma of limiting my advice to students looking to improve their projects. I had to avoid using insights that other students had shared with me because of the competitive nature of the class. Normally if someone had difficulty with a homework problem, I could use their learning and share it with others, but this time, that wasn't automatically the case.

There was also a substantial size difference, I had 20-30 students, which is by far the smallest class I've ever lectured to. Previously, Stat 342 was my 'small' class, which had enrollment between 50-80, because it was compared to service classes of 100-300 students. This allowed me to actually communicate with students an a much more one-on-one level. Furthermore, since most of the work was done in small team settings, I got to know what each group of students was working on for their projects.

I worry that what I delivered wasn't exactly big data, and was really more of a mixed bag of data science. However, there was a lot of feedback from the students that they found it valuable, and value-added was what the goal all along.