Replication report: Multilevel-Linear-Models using SAS and SPSS for repeated measures designs

The following is a report on the reproduction of the statistical work in the paper “Differences of Type I error rates for ANOVA and Multilevel-Linear-Models using SAS and SPSS for repeated measures designs" by Nicolas Haverkamp and André Beauducel at the University of Bonn.

Replication Report - Informative Priors and Bayesian Updating

The following is a report on the reproduction of the statistical work in the paper "The use of informative priors and Bayesian updating: implications for behavioral research" by Charlotte O. Brand et al.

The original paper was accepted for publication by Meta-Psychology, https://open.lnu.se/index.php/metapsychology, a journal focused on methodology and reproductions of existing work. This report continues my attempt to establishing a standard for replication reports according to the Psych Data Standards found here https://github.com/psych-ds/psych-DS

Replication Report - Signal Detection Analysis

The following is a report on the reproduction of the statistical work in the paper “Insights into Criteria for Statistical Significance from Signal Detection Analysis” by Jessica K. Witt at Colorado State University.

The original paper was accepted for publication by the Journal of Meta-Psychology, a journal focused on methodology and reproductions of existing work. This report on is my first attempt at establishing a standard template for future such reports according to the Psych Data Standards found here https://github.com/psych-ds/psych-DS

Template for a technical report, with example rubric

A completed technical report might look like this:

1. Executive Summary

2. Introduction / Problem statement

3. Methods

4. Results

5. Conclusion / Discussion

 Executive Summary: The TL;DR

This is the LAST thing that you should write. This would be the tl;dr of the technical report. “tl;dr” stands for “too long; didn’t read”. More formally this is called the executive summary, which means ‘if this report was given to a major decision maker, whom has tons of things they need to know already, what would you like them to know from the report that can be reduced to 100 words or fewer.’

Here you should write the research question as shortly as you can, one main result, and the name of the main method used. Nothing from the discussion / conclusion section is needed here.

Introduction:

The introduction typically follows a close formula.

1. Describe the research problem or state the research questions that were posed. If you can, tell why this research problem is important. The explanation of importance doesn’t have to be too specific to the research problem. If you are working with data about a medical problem, mention that many people suffer this medical problem; in a research paper, this is a good opportunity to cite a well-known related paper that has found the scope of the problem for you.

If you don’t know why a problem is important and a quick literature search won’t tell you, leave the problem’s importance to a co-author whose expertise is more suited to this part. It’s much better to admit you don’t know something than to say something wrong.

2. Describe each section in the paper or report in very short detail. (e.g. “in the methods section, we describe the data cleaning and the regression tree method that we used. In the results section, we describe the goal scoring rate of different hockey players. In the discussion section, we follow up with a comparison of this method to an older, more traditional one.”)

Methods

The methods: What did you do to get these results?

If this were a field science, you would list the days and describe the conditions under which you went out into the field and gathered information (e.g. ‘we collected our samples on sunny days in the North Okanagan valley between June 10^th and September 20^th, 2015’). In a data science, you would instead describe the dataset that you used, its format and size, and key variables and features (e.g. ‘We gathered the data from NHL.com’s event-tracking database using the nhlscrapr package along with our own patch, The data we collected included each goal, shot, hit, penalty, and faceoff recorded in each regular season game from October 2012 to April 2017’)

This is where the bulk of your writing should be. About 50% of your report will be the methods section. You don’t need to explain the entire data cleaning process, but you should mention where the data came from, and the tools / software that were used. It’s also good practice to mention when the data was taken (especially in the case of news reports which may be updated, altered, or archived such that scraping may produce different results later).

If there were any judgement calls in your data cleaning process, such as…

- what was done about extreme and influential cases,

- how problematic variables were used,

- how tuning parameters for complex methods were selected, and

- how missing values were either filled in or explained away,

…these should be included as well.

In short, you don’t have to give everything away, but an expert with the same software and data access should be able to recreate what you did.

A methods section serves two purposes: first to give legitimacy to your results. If you show results without explaining how you got them, a reader might assume that the results were invented or made up. With a methods section, the reader should be able to see a logical path between the data and the results.

After the data preparation is explained, describe the model you selected or the process you used to select the model. If you just did linear regression, say that. If you used a random forest, or the LASSO, or stepwise regression, say that instead.

Normally, you only need to include the final method that you decided upon. However, there is a good chance that the method you used wasn’t the only method that you tried. In a research paper, you wouldn’t necessarily mention these ‘dead ends’ because paper length is limited by the journal. In a technical report (or a thesis) these other approaches are useful to help you justify your choice and that alternatives were considered. You can explain why these rejected methods didn’t work or what about the results they produced was bad. Don’t overdo these dead-end explanations. The reader is much more interested in what you did and what worked instead of what didn’t work, typically.

Example: “After an exploratory analysis, we tried to classify events using random forests, dimension reduction, and neural nets. We decided to further pursue neural nets because they produce models with much lower out-of-bag errors than other approaches.”

Results

It’s easiest to write the results first, even though they don’t appear first. Any tables of figures you want to show, make these as soon as the analysis work is done. Talk about your results a little. Explain the importance of any tables and figures; why are they there?

Mention the general trend (e.g. ‘there is a negative, non-linear trend between playing time per game and shots against goal’), and any notable observations (‘however, the New Jersey Devils break this trend’)

You don’t need to write much here. The charts should explain themselves.

Discussion  / Conclusion:

In a technical report, this is where you take the results and give them meaning in the context of the research questions that were in the introduction. You can also quickly summarize what you did.

In a journal paper or a thesis, this section might also include future research questions that could be answered with more data or by a different analysis. A technical report should be more self-contained, and allusions to the further work are not required.

In every case, no new information about the project should in introduced in the conclusion. If you have an interesting finding, it should be in the results. If that interesting finding doesn’t fit with the rest of the results, a new subsection for it can always be made, but keep it out of the discussion section.

Remember, when giving context to the results, don’t reach beyond your expertise. If the data is genetic, and you are not a geneticist or biologist, do not make conclusions about the importance of a gene. Often statistical publications are co-authored with subject experts; let those experts write about their topics and stick to the data analysis.

Example Rubric: Total out of 100

Length	3-6 full pages	10
	7 pages	8
	8 pages	6
	More than 8	0
	2.5 pages	8
	2 pages	4
	Less than 2 full	0
		Possible 10	/10
Grammar	Start with 10, any OBVIOUS grammar or spelling mistakes, reduce this by 2. Minimum 0/10
		Possible 10	/10
Executive Summary	Name of main method included	3
	Primary finding described	7
		Possible 10	/10
Introduction	Describes the research problem	6
	Makes a case for the its importance	4
		Possible 10	/10
Methods	Describes the data used	5
	Describes the data preparation and/or data cleaning	5
	Describes any decisions / judgement	5
	Describes method used	10
	Justifies choice of method	5
		Possible 30	/30
Results	Summarizing data through text, table, or a figure	10
	Description of a general trend	5
		Possible 15	/15
Conclusion	Ties results back to research question	10
	Does NOT include any new information that would be better in the results or methods sections.	5
		Possible 15	/15
	TOTAL		/100

Advice on Selecting the Right Journal

When I try to get something published in a journal, it's for the prestige and implied proof of quality.
Otherwise, if I wanted to write something and get attention for the idea quickly, I could write a blog post like this.

As such, I aim for a balance between the perceived importance of the journal and the chance of acceptance.

(Updated March 2019 with a section on predatory journals)

Creating Homework Material for Statistical Writing Classes and Workshops.

This summer, I'll be offering a workshop on statistical writing to the graduate students in the SFU Stats department. In the year that follows, I hope to teach SFU's Stat 300 course, an undergraduate writing course that is mandatory for all statistics majors. I want this course to be offered at more universities, but since the typical undergrad degree in stats is already overloaded, this would need some additional motivation, so here's my anecdotal hook:

When graduate schools ask for a letter about your motivations for applying, or about your general background, there's a few things that are being examined. First, different graduate supervisors have different specialties, and they may be looking at your letter for signs of a good match. Second, the letter is a means of seeing firsthand your personal ability to communicate in English. They ask for something personal rather than a set topic to discourage plagiarism. They are not interested in your level of motivation; everyone says they are highly motivated.

I've recently been writing reference letters and filling out graduate school reference forms for students that have been in my previous 300-level classes. All of these schools are asking about the ability to write. Many statistics undergrads are applying to fields outside of pure statistics, for example economics, medical science, and business. All of them are asking their references about skills in English, often spoken English and always written. Likewise, the Statistical Society of Canada asks referees about spoken English as a part of their A.Stat accreditation.

In order to develop these skills in a program that usually focuses on mathematical theory and programming ability, I've sketched out some projects and exercises that could fit into a workshop or course.

The instructor (me) would ask colleagues from other fields that use quantitative data, such as sociology, anthropology, biology, business, ecological restoration, and history for reports from previous or current projects.

The students would read the reports and identify what the sampling method was, what analysis was done, and to check the assumptions on those methods to the data. They would interpret the results in their own words (not in the words of the discussion or conclusion section), and compose a couple of questions about the research that weren't answered. These questions don't have to pertain to the domain; they should pertain to the methodology. These would be questions like: were there any outliers that affected your results? Did you consider multiple testing? Did you consider regression/anova?

For example, for this research report, Water Quality of Stoney Creek and its Effects on Salmon Spawning, by SFU undergrads by Oak, Tony; Thai, Michelle; Orgil, Indra; Ngo, Kevin; Lu, Jerry , available at http://summit.sfu.ca/item/12770

A writing assignment could be:

• What were the parameters to be estimated?
• What biologically important thresholds or critical values are mentioned for the parameters being estimated?
• What sampling method was used? How many sampling units are there?
• Were there any null hypotheses being tested? If so, were they rejected? Should they have been?
• Describe the qualitative differences between the four sites. Use the map and Figs 4,5,6, and 7.
• In your own words, summarize the quantitative data described in Figures 2 and 3.
• Compose two suggestions or questions about the analysis that may not have been considered.
• Describe one way you could build upon (e.g. expand, follow up) this study, the type of information you would collect from this new work, and what additional conclusions you could make if the data matched your expectations.

(Note: Please don't take these questions as a criticism of this research report. It was selected to show how material is available within one's on campus. Even a quick skim of the report will show it's truly excellent work for undergrads.)

Another exercise I'm thinking of doing is to have students copy edit some scientific writing. This could be a publication or a blog post. I would do this twice; once with a well written work, and once in something of questionable quality like the unedited papers that come out of discount publishers. If I used something from ArXiv, with the authors' permission, I could use later, more complete versions of the same work as a 'key'. However using ArXiv is untenable for graded work because the 'key' is publicly available.


Also, there are the reading comprehension exercises.The ones I had posted earlier (here and here) were meant to be done in less than 2 hours, and had questions focused on specific statistical topics. Another possibility is to write more such assignments that require less statistical expertise, but more in-depth composition.