A Long term vision for a Master's Degree in Data Science Program

I was recently asked to write a statement of vision for a potential one-year coursework master's program. A colleague was kind enough to look at it for me; he said was 'a bit too bombastic'.

This statement has already been submitted to its intended destination.





----------

Successful data scientists need to be able communicate not just verbally and in writing, but also
visually by way of graphs, dashboards, and animations. They need to have working knowledge of
modern database languages like SQL and big data architectures like Hadoop. They need to be able to
determine when to use modern statistical, machine learning, and optimization methods like the LASSO, neural networks, and random forests.

But successful students won't just have a practitioner's knowledge of these tools, because these tools
will be replaced eventually. They also need the depth in their backgrounds to evaluate and adapt to
additional systems and methods as they become available.

Therein lays the challenge: the demands upon a data scientist are broad, whereas a Master's degree is
typically a structured, focused, deep exploration of a single field. There simply isn't enough time to
cover all that's necessary to develop a prospective student starting with a bachelors degree in
Mathematics or Computer Science into a consummate data scientist in ten months. 

Some topics will need to be sacrificed for the sake of brevity, but what? Different students will bring
diverse skills and affinities into such a degree program, and they will be good judges of what they
should focus on. However, a degree is essentially a set of requirements, which is another way of saying it's a set of guarantees. The better defined those requirements are, the clearer the guarantee of skill that the bearer of such a degree brings to future employers.

In the face of program that will inevitably be stretched thin across many competencies, there are two
competing needs: the need for students to develop the subset of these competencies that maximize their personal return, and the need for standardization across the program to make its value and quality obvious to all stakeholders. To reconcile these two needs, I envision a specialization system. Graduates from the ideal Master in Data Science program will also graduate with one of four specialties: visual analytics, databases, methodology, or algorithms.

Under this specialization program, all MDS candidates will be required to take a core of data scraping, imputation, R or SAS programming with an SQL component, modern regression such as GLMs, and scientific writing. This totals 15 graduate credits. The remaining 9 credits form a specialty.

Database experts would be most akin to software engineers. The courses for this specialty would
include one focused on the extract-transform-load paradigm, one focused on handling big data tools
like Hadoop. Graduates from this specialty would be expected to be able to implement automated tasks
for gathering, cleaning, and summarizing information from the web or some other digital sensor.

Methologists would take applied statistical courses like design of experiments, sampling, dimension
reduction, time series, and spatial statistics. What separates this specialty from a Master's degree in
statistics is the lack of emphasis on proofs. Students in these courses need not understand why a
method works, only how to assess through diagnostics and checklists that it is working and when it is
appropriate.

Visual analytics specialists would take courses focused on user interfaces and communication,
including graphing and data cartography, dashboards, and additional writing work such as survey
design. Graduates from this specialty would be expected to demonstrate familiarity with popular
database interface like Jaspersoft and Tableau.

Algorithm experts would focus their additional coursework on new ways to find meaning from the
data deluge. Their corpus would include machine learning, optimization methods like quadrature and
simulated annealing, text processing concepts such as regular expressions and edit distance, image
processing, clustering, compression and information theory.

A graduate with skills in any one of these four specialties fits nicely under what we know as a data
science today.  This vision is a grand one, and far too large for a new master's program to take on, but
it's the endgame i have in mind for this program.

Now you're thinking with gates!

What do Nintendo and Bitcoin enthusiasts have in common? They weren't content with solving their problems through software advancements alone. The statistical computing field shouldn't be either.

-------------------

The Super Nintendo Entertainment System is a cartridge-based system, meaning that its games were stored on circuit boards encased in plastic cartridges. Unlike disc-based media of most later generations of game consoles, the contents of cartridges were not restricted to read-only data. The most common addition to game cartridges was a small cache of re-writable memory used to store progress data in the cartridge.

Originally, active objects in games, called sprites, could only be displayed as one of a set of pre-drawn frames.  That's why sprite animations are usually simple loops of a few frames, and why characters are rarely seen changing size as they move towards or away from the player's point of view.

However, later games also included special-purpose microchips that expanded the graphical capabilities of the Super Nintendo console itself. One of these chips allowed the SNES to change the way sprites look as the game was happening, which made sprites look much more alive. This chip also allowed for rudimentary three-dimensional rendering.

Any software workaround to get these effects using only the hardware given in the Super Nintendo would have taking much longer and produced much worse results, if any at all. The video on the Super Nintendo (SNES) by video game trivia group Did You Know Gaming covers these effects and the chips in more detail, and shows some great demonstrations.

------------------

Bitcoin, is a cryptocurrency. Part of what gives it value is the premise that it is computationally hard to create or 'mine' for new ones. In fact, there is a self-adjustment mechanism that increases the mining difficulty in proportion to the total computing power of all miners.

I've appended this historical chart of the log of the total computer power (and the log difficulty), over time with the two hardware advancements that defined the trend in bitcoin mining power.

 

 The first event represents the first time mining using a more specialized graphical processing unit (GPU) rather than a more general central processing unit (CPU) was made publicly possible. Since many miners had compatible graphics cards already, we see a tenfold jump in power almost instantly. 

The second event represents the first time mining using a single-purpose processor, called an ASIC*   was introduced to the market. This time, another rapid increase in processing power is sparked, but without the initial leap.

An ASIC is orders of magnitude faster at the simple, repetitive task of mining bitcoins than a GPU is, and a GPU mines orders of magnitude faster than a comparably priced CPU. In both cases, the new hardware quickly rendered any previous mining methods obsolete.

* Application Specific Integrated Circuit.

---------------

When developing new methods to solve computational problems, a software approach usually works best. The results of purely software-based are often portable as packaged programs,  and the dissemination of improvements can be as quick and as cheap as a software update. The feedback loop of testing and improvement is very quick, and there are many languages such as the R, SAS, and Julia that can make software-based solutions a routine task.

Making hardware to solve a problem may sound insane by comparison - why would anyone willingly give up all of those advantages? This is where Field Programmable Gate Arrays come in. An FPGA is essentially a circuit board that can be programmed down to the gate level. That is, a user can write a program in terms of the fundamental particles of computation, OR, NOT, XOR, AND, and NAND gates. 

The FPGA takes a set of gate instructions and physically wires itself into the programmed configuration. When set, the FPGA is essentially an ASIC, an processor that can only do one task but potentially much faster than a general purpose computer. However, if needed, an FPGA can be re-programmed, so the advantage of a quick trial-and-error turnaround is there. Also, the program can be disseminated like any other software. The most popular FPGAs cost between $200 and $500 USD.

The bitcoin ASIC started off as an FPGA. Once the FPGA program was made, it took about a year for the first ASICs to be sold. This is encouraging for anyone looking towards FPGAs for the next great leap in statistical computing, as it means the endeavor has commercial viability. Just think how much faster some common methods could become even if only large matrix inversion was made faster.

It's time to start thinking with gates.

Using optim() to get 'quick and dirty' solutions, a case study on network graphs

The optim() function in base R, is a generalized optimizer for continuous functions. It looks for the minimum of some objective function that you define across some continuous parameter space that you define.

In this post, I show how to use optim() to find a (inelegant, but workable) solution to do something very complex, plot a network of nodes based on the shortest path between them, with relatively little programming effort.

Prelude to a FUSS

I apologize in advance if this one is incoherent, as most of it has come in fever dreams over the last couple days.

I want to make a FUSS. That is, a Formula-Unspecified System Solver. 

R Packette - Weighted Binomial Sums

This R code file Weighted_BinSum.r is a... packette? Proto-package? Mini-package?

It's by no means a full R package because it doesn't have the proper documentation, but it's the start of what could be one.

Fractional data types

There's likely a good solution to this already, but it eludes me:  Why are there no fraction data types at the same basic level at int, char, and float?

A lot of statistical operations, like matrix inversions, are computationally very expensive. Part of the reason they take so long to perform for large matrices is because of all the division of numbers that's going on. Compared to addition and multiplication, division of floating point numbers is very hard for computers. It has to be done in a form of binary long division, which involves a subtraction and multiplication for each bit.

However, division of fractions is simply cross-multiplication, which is cheap.

Consider a double length float in a 32-bit system. It has 64 bits, 52 which store the actual number ( the mantissa), 11 which store how large the number is, and 1 for the plus or minus sign.

Modelled on this, a 64-bit fraction data type could have two 28-bit mantissae, 11 bits for the magnitude, and 1 for the sign.

Note that fractions are not unique representations of numbers. 18/12 is 6/4 is 3/2. We can exploit that to use only a single sign and magnitude for the fraction as a whole.

Conversion from fraction to double is a simple division, so we can't avoid division entirely, but we could do intermediate steps as fractions.

Conversion from double to fraction is harder. It can be done in a arbitrary manner, of setting the denominator to 1, but in the above system, 28 bits of precision are lost, and the whole denominator is wasted storing '1'.

Is there a way to convert a double to fraction to minimize loss? Can it be determined in a way that's cheap enough to make conversion worthwhile? My uninformed hunch is converting x to x^1.5 / x^0.5 would work, but square roots are still a bit costly, and I have no idea if it's particularly good at preserving precision.

Any ideas?