October 7, 2010

Statistical Inference (Casella and Berger)

Set Theory:

Union - combination of two sets

Complement - everything that's not in A

Empty - no elements

Definitions:

experiment - any activity generating observable results

outcome - result of experiment (IMPORTANT TO KEEP STRAIGHT! don't confuse events and outcomes)

trial - single performance of experiment

sample space - set of all possible outcomes

countable/uncountable: - countable = one-to-one correspondence (e.g. 1/n) - uncountable = no one-to-one correspondence can be made -- infinite loop: you can do an infinite loop, but still count it -- flipping a coin: countable; temperature: uncountable

event - any subset of the sample space

Example:

experiment - roll a dice outcome - 1, or 2, or 3, or 4, or 5, or 6 trial - one roll of the dice COUNTABLE sample space - {1,2,3,4,5,6} event - may be {1}, or {1,2,3}, etc...

Operators:

Union, empty set, complement, intersection

Commutative:

Associative:

Distributive:

DeMorgan's Law:

Call a set abnormal if it can be put into itself (otherwise it's normal)

Example: the set of all squares is not itself square, so it is not a member of the set of squares The complimentary set, containing all non-squares, is itself not a square, so is normal

Consider the set of all normal sets Is it normal or abnormal? If it were normal, it would be contained in itself, and would therefore be abnormal If it were abnormal, it would not be contained in itself, and would therefore be normal

You can resolve this using more rigorous set theory...

More Definitions:

Disjoint ("set" term) / mutually exclusive ("probability" term) - if the intersection of two sets is the null set, they are mutually exclusive

Partition - take a group of sets; if the union of these sets is the sample sapce, and they are mutually exclusive, this is a partition

Distinction between probability theory that has a physical meaning (and is therefore "contaminated" by intuition) and a more abstract probability theory that doesn't have a corresponding physical meaning

Axiomatic probability theory (Komolgorov)

A probability is a function that follows 3 axioms:

Sample space $$ S $$

(The domain) $\sigma$ -algebra $\mathfrak{B}$ (means the set is fully consistent)

Function P -> probability over the domain $\mathfrak{B}$

1. $P(A) \geq 0$ for all $A \in \mathfrak{B}$

2. $$ P(S) = 1 $$

3. If $A \in \mathfrak{B}$ and $B \in \mathfrak{B}$ are disjoint, then $P(A \bigcup B) = P(A) + P(B)$

In other words, $P( \bigcup_{i=1}^{\infty} = \sum_{l=1}^{\infty} P(A_{i})$

This is a mathematician's viewpoint: a clean definition, as long as we follow these rules, the function is a probability.

What is the probability of the null set?

Create a partition: $S = {S, \emptyset}$

The probability of the sample space is $$ P(S) = 1 $$

So $P(\emptyset) = 1 - P(S) = 0$

$$ P(A) = 1 $$

$$ P(A^c) = 1-P(A) $$

If $A \subset B$ then $P(A) \leq P(B)$

The size of the set is directly related to the probability...

Another way to do this is using measure theory (another route, besides rigorous set theory, that leads to probability theory)

[wikipedia:Measure theory]

[wikipedia:Sigma-algebra]

Bonferroni's inequality: $P(A \bigcap B) \geq P(A) + P(B) - 1$

Reading Assignment Discussion

Classical Definition of Probability (Laplace, 1812)

"If a random experiment can result in $$ N $$ mutually exclusive and equally likely outcomes and if $$ N_A $$ of these outcomes result in the occurrence of the event $$ A $$ , the probability of $$ A $$ is defined by $P(A) = P{A} = {N_A \over N}$ ."

Example: rolling a dice

Event A might be how many times we roll a 1... or how many times we roll a 1 or a 2...

What if one side of the dice is weighted to favor 5? This definition doesn't work... No mathematical proof to show that the outcomes were mutually exclusive and equally likely.

Frequency

This is the limit, as the number of experimental trials performed (trials must be performed under "identical" conditions) goes to infinity, of the number of outcomes of the event of interest $$ N_A $$ :

$P(A) = lim_{n \rightarrow \infty} {N_A \over N}$

Limitations:

can never actually perform an infinite number of trials
what does "identical" mean? e.g. if you're performing a turbulence experiment, how can you initialize everthing the exact same way?
(Tony): what's your infinity?
- (Sean): whatever it is, it's not finite

Probability can be seen as lack of information (e.g. fluid mechanics is deterministic, so we could describe it if we know the state perfectly - we use probability to fill in/make up for the lack of knowledge)

Quantum theory: in two-split experiment, the very state of nature is random and needs to be defined using probability

Probability of an outcome in the future: judging confidence based on prior experience... statement of confidence

Bayesian vs. fequentist

Side discussion:

wikipedia:Noether's theorem

October 11, 2010

Derivation of Bonferoni's Inequality

Show:

$P(A \bigcap B) \leq P(A) + P(B) - 1$

Proof:

$P(A \bigcup B) = P(A) + P(B) - P(A \bigcap B) P(A \bigcap B) = P(A) + P(B) - P(A \bigcup B)$

But in general,

$P(A \bigcup B) \leq 1$

Plugging this back in,

$P(A \bigcap B) \leq P(A) + P(B) - 1$

On Wikipedia: more general case; more than two sets

wikipedia:Boole's inequality

Conditional Probability and Bayes' Rule

if $A,B \in S$ , and $P(B) \neq 0$ ,

$P(A|B) = \frac{P(A \bigcap B)}{P(B)}$

And as a result, we get Bayes' Rule:

$P(A|B) = P(B|A) P(A) \frac{ P(A) }{ P(B) }$

Derivation:

$\frac{ P(A) }{ P(B) } \times P(B|A) = \frac{ P(B \bigcap A) }{ P(A) } = \frac{ P(A \bigcap B) }{ P(A) } \times \frac{ P(A) }{ P(B) } = P(A|B)$

Example application of conditional probability:

Run a combustion simulation... "Given that the temperature is in range X, what is the concentration range?"

Statistical Independence

Definition: $P(A \bigcap B) = P(A) * P(B)$

Consequences:

$$ P(A|B) = P(A) $$

Random Variable

wikipedia:Random variable

A random variable is a mapping from a sample space to real numbers.

Example: Dice roll

Mapping rolls to a set $$ {1,2,3,4,5,6} $$

Example: Morse code

Looking at statistics of morse code...

Mapping dots and dashes to $$ {0,1} $$ (or alternatively $$ {1,2} $$ )

IMPORTANT: Sample space is different from the random variable

Induced Probability Function

Let $$ P(B) $$ for event B $B \in \mathfrak{B}$ on $$ S $$ , outcome $S_{i}$

(e.g. there's a sample space, and in the sample space there are events...

Before, we were talking about probability of events. Now, we're talking about probability of a random variable)

Random variable $$ X(s) $$ , random variable realization $\chi$ , $A \subset \chi$

$P_{X} (X \in A) = P{ S_{i} \in S : X(S_{i}) \in A }$

$S_{i}$ are the outcomes that are in B

Cumulative distribution function

Definition:

$F_{X}(x) = P_{X} (X \leq x) \forall x$

Example: Dice roll

Probability of rolling a given number is constant (1/6)

Cumulative distribution function is a line, because for x=1, cumulative probability is 1/6; for x=2, cumulative probability is 2/6; and so on.

This definition is more general than the integral definition, because in general you need a cumulative distribution function that is differentiable

Sometimes there will be situations where a probability distribution function can't be defined, and only a cumulative distribution function can be defined

$F_{X} \geq 0$

Further information:

$\displaystyle{ \lim_{x \rightarrow -\infty} } = 0$

$\displaystyle{ \lim_{x \rightarrow +\infty} } = 1$

Examples

Given 5% of men are colorblind, and 0.25% of women are colorblind: if a person is chosen at random and they are colorblind, what is the probability of their gender?

$P(CB|M) = 5% \ P(CB|W) = 0.25% \ P(M|CB) = ?$

$P(M|CB) = \frac{ P(M\bigcapCB)}{P(CB)} = \frac{ P(CB|M) P(M) }{ P(CB) } = \frac{ (0.05) (0.50) }{ P(M) P(CB|M) + P(CB|W)P(W) } = 0.952$

ProbabilityDiscussion

From charlesreid1

Contents