Key aspects that make something "science"

  1. A theory must be falsifiable [source].  This requires the "two legs of science", which are theory and experimentation \cite{Vardi_2010}
  2. Generalisability: how well does the theory generalise to new data?  Also, a theory should be able to generate new predictions \cite{Vardi_2010}.  "If a theory makes no predictions, it ceases to be physics" -Ben Freivogel, "The Crisis of the Multiverse".
  3. Reproducibility. \cite{Peng_2011} and \cite{Leek_2015}. This includes methods reproducibility, results reproducibility, and inferential reproducibility [Stephanie Wykstra, "What do we mean by reproducibility?". 

Things that can go wrong

  1. Is the data reliable? To what extent can the process of data collection be trusted?: There could be statistical assumptions and random error or systematic errors. Are there assumptions made with regards to selection bias, statistical distribution of the data / statistical distribution of the errors, etc. How sensitive is the output data to the inputs? How large are the error bars?  Is this the original dataset, or have transformations on the data already been applied (and have these data transformations been checked)?  Is a control group necessary?
  2. Is the process of arriving at an explanation / hypothesis reliable?: what steps have been taken to rule out the following - data dredging / p-hacking \cite{Nuzzo_2014}, data contamination (also called data leakage), cherry-picking results, overfitting, confirmation bias, etc?  Correlation does not imply causation. Under what set of assumptions is the theory valid?

Limitations of Science or the Scientific Process

The Approach to Science & the Scientific Process

  1. "All models are wrong but some are useful" -George Box. Scientific models will not always be able to capture natural fully, some simplifications are always involved. The issue is therefore about whether those simplifications are reasonable (see also the spherical cow metaphor).
  2. Self correction: at what speed is science expected to "self-correct" \cite{Ioannidis_2012}? See also: The Economist "Trouble at the lab" and Sarah Estes "The Myth of Self-Correcting Science".
  3. Publication bias - do "positive results" have a much higher probability of getting published, compared to "negative results"?

Philosophy of Science Aspects

  1. Inductive reasoning vs deductive reasoning: In the process of deduction, you can prove a hypothesis (or model) as incorrect, but you do not prove that the model is correct; you only increase the "confidence" that you are correct [source: Peter Ellerton "What exactly is the scientific method and why do so many people get it wrong?"]. Proving requires induction, but there is the issue that it requires personal judgment (there may be multiple ways to generalise the data that was observed) [source: Jonathan Keith, "Why should we place our faith in science?" - The Conversation] - the problem of induction is well known. In abductive reasoning the issue that there are multiple ways to generalise the data is addressed by choosing the "simplest and most likely explanation".
  2. Gödel's Incompleteness Theorems and Alan Turing's reformulation of it.
  3. Undecidable problems, such as the halting problem.