Is applied statistics, and more particularly model building, an art, with each new case having to be treated from scratch (although even artistic endeavors require techniques which can be systematized and learned), completely on its own merits, or does theory have a contribution to make to this process? — Erich Lehmann (1990)
Research Program: Sharp Statistics
SHARP STATISTICS is the product of a decade-long research effort, where SHARP denotes the fourth fundamental operation of statistics—after ESTIMATE (Fisher, 1922), TEST (Neyman and Pearson, 1936), and PREDICT (Vapnik, 1995)—that establishes the statistical rules for recursively sharpening models based on new data. This can be algorithmically expressed as:
Modelk ← SHARP (Modelk-1, datak), k=1,2,… (1)
where Model0 is the user-selected initial “seed” model for the problem, constructed either using subject matter knowledge, purely empirical means (machine learning techniques), or a combination of the two. SHARP describes a statistical mechanism for iteratively adding empirical knowledge to the latest understanding of reality in an automated and scalable manner.
Radical shift in perspective and objective. The SHARP model-building formulation (1), liberates statistical modeling from the unproductive concept of “fixed but unknown true distribution” and advocates continuous model revision for advancing our knowledge. This radical shift in our understanding and perspective on model building is necessary for statistics to live up to its reputation as a “science of knowledge.”
What makes Sharp Statistics
special?
Time-tested general theories in science usually exhibit two vital traits:
- Unifying power [Beauty]: Does this approach capable of organizing the vast collection of statistical methods into a nearly structured body of knowledge by forging a harmonious union between them?
- Predictive power [Utility]: Does this approach offer insights into the design of new statistical methodologies that have yet to be developed?
The remarkable thing about the SHARP framework is that it possesses both of these qualities. This is achieved by designing the SHARP engine in a special way, which is rooted in the unique principles of abductive inference and quantile mechanics.
What makes Sharp Statistics
unique?
The table below contrasts Sharp statistical learning with the traditional statistical learning paradigm on several dimensions.
| Character | Statistics | SHARP Statistics |
|---|---|---|
| Focus of learning | Specific learning model | Model’s learning process |
| Purpose of learning | Model estimation | Model sharpening |
| End goal of learning | Description and prediction | Revision and explanation |
| Theoretical basis | Theory of approximation | Theory of change |
| Inferential basis | Inductive reasoning | Abductive reasoning |
| Computing domain | Distribution (raw data) domain | Quantile-transform domain |
| Algorithmic rational | Optimization mindset | Revisionary mindset |
| Visionary pioneers | Pearson, Fisher, Neyman | Peirce, Tukey, Parzen |