1.1 The bayesian paradigm

Statistical analysis is fundamentally an inversion process. The objective is to the "causes"--parameters of the probabilistic data generationmodel--from the "effects"--observations. This can be seen in our interpretation of the likelihood function.

Given a parameter $$ , observations are generated according to $$p(, x)$$ The likelihood function has the same form as the conditional density function above $$l(|x)\equiv p(, x)$$ except now $x$ is given (we take measurements) and $$ is the variable. The likelihood function essentially inverts the role of observation(effect) and parameter (cause).

Unfortunately, the likelihood function does not provide a formal framework for the desired inversion.

One problem is that the parameter $$ is supposed to be a fixed and deterministic quantity while the observation $x$ is the realization of a random process. So their role aren't really interchangeable in thissetting.

Moreover, while it is tempting to interpret the likelihood $l(|x)$ as a density function for $$ , this is not always possible; for example, often $$\int l(|x)\,d \to$$∞

Another problematic issue is the mathematical formalization of statements like: "Based on the measurements $x$ , I am 95% confident that $$ falls in a certain range."

All of these "deficiencies" can be circumvented by a change in how we view the parameter $$ .

If we view $$ as the realization of a random variable with density $p()$ , then Bayes Rule (Bayes, 1763) shows that $$p(x, )=\frac{p(, x)p()}{\int p(, x)p()\,d }$$ Thus, from this perspective we obtain a well-defined inversion: Given $x$ , the parameter $$ is distributing according to $p(x, )$ .

From here, confidence measures such as $p(x, \in H_0)$ are perfectly legitimate quantities to ask for.

Bayesian statistical model

A statistical model compose of a data generation model, $p(, x)$ , and a prior distribution on the parameters, $p()$ .

The prior distriubtion (or prior for short) models the uncertainty in the parameter. More specifically, $p()$ models our knowledge--or lack thereof--prior to collecting data.

Notice that $$p(x, )=(\frac{p(, x)p()}{p(x)}, p(, x)p())$$ since the data $x$ are known , $p(x)$ is just a constant. Hence, $p(x, )$ is proportional to the likelihood function multiplied by the prior.

Bayesian analysis has some significant advantages over classical statistical analysis:

• properly inverts the relationship between causes and effects
• permits meaningful assessments in confidence regions
• enables the incorporation of prior knowledge into the analysis (which could come from previous experiments, forexample)
• leads to more accurate estimators (provided the prior knowledge is accurate)
• obeys the Likelihood and Sufficiency principles