\[ \newcommand{\nRV}[2]{{#1}_1, {#1}_2, \ldots, {#1}_{#2}} \newcommand{\pnRV}[3]{{#1}_1^{#3}, {#1}_2^{#3}, \ldots, {#1}_{#2}^{#3}} \newcommand{\onRV}[2]{{#1}_{(1)} \le {#1}_{(2)} \le \ldots \le {#1}_{(#2)}} \newcommand{\RR}{\mathbb{R}} \newcommand{\Prob}[1]{\mathbb{P}\left({#1}\right)} \newcommand{\PP}{\mathcal{P}} \newcommand{\iidd}{\overset{\mathsf{iid}}{\sim}} \newcommand{\X}{\times} \newcommand{\EE}[1]{\mathbb{E}\left[{#1}\right]} \newcommand{\Var}[1]{\mathsf{Var}\left({#1}\right)} \newcommand{\Ber}[1]{\mathsf{Ber}\left({#1}\right)} \newcommand{\Geom}[1]{\mathsf{Geom}\left({#1}\right)} \newcommand{\Bin}[1]{\mathsf{Bin}\left({#1}\right)} \newcommand{\Poi}[1]{\mathsf{Pois}\left({#1}\right)} \newcommand{\Exp}[1]{\mathsf{Exp}\left({#1}\right)} \newcommand{\SD}[1]{\mathsf{SD}\left({#1}\right)} \newcommand{\sgn}[1]{\mathsf{sgn}} \newcommand{\dd}[1]{\operatorname{d}\!{#1}} \]
3.1 Estimator
Form this section onwards you will learn specific methods to estimte parameters of the distribution from the sample.
Example 3.1 (Coin Toss) Suppose a coin has probability \(p\) of heads. To gather information about \(p\), we toss the coin \(100\) times and observe results as \(X_1, X_2, \dots, X_{100} \iidd Ber(p)\). We get that \[ \sum_{i=1}^{100} X_i = 67 \] Now the question comes Can we use this information to estimate \(p\)?
We will discuss 4 methods for the estimation of \(p\). (1) Method of moments and (2) Maximum liklihood estimate \(-\) both these methods will produce a number as estimate for \(p\). Later you will see (3) Interval Estimation with which you will get a range which will contain the true value of \(p\) with some accuracy (probability). In the subsequent chapter, you get to know about (4) Hypothesis testing.
Assumptions
In this chapter, we will assume the following:
- \(X_1, X_2, \dots, X_n\) are \(\textsf{iid}\) form \(X\) and for both Discrete and Continuous random variable \(X\) we will denote the both pmf and pdf with \(f(\circ)\)
- The pdf or pmf \(f(\circ)\) will depend on unknown parameters \((p_1, \ldots, p_d) \in \R^d, d \geq 1\)
Notation
- Instead of using \(f(\circ | p_1, \ldots, p_d)\) (pmf or pdf depending on the parameters \({\bf p} = (p_1, \ldots, p_d)\)) we will use, \(f(\circ | {\bf p})\)
- We will denote the parameter space with \(\PP \subseteq \R^d\). The parameter \({\bf p} = (p_1, \ldots, p_d) \in \PP\)
Definition 3.1 (Estimator) Let \(X_1, X_2, \dots, X_n \iidd f(\circ | p)\). Suppose we are interested in estimating \(\Theta(p)\) where, \(\Theta : \PP \to \R\). Let, \(g: \R^n \to \R\). Then \(g(X_1, X_2, \dots, X_n)\) is defined as a point estimator of \(\Theta(p)\). It’s value from a particular realistion of \(X_1, X_2, \dots, X_n\) is called an estimate
So, Estimator is a function of random sample which is used to estimate parameters of the underlying distribution.
Example 3.2 (Sample mean) Let \(X\) is a random variable with \(\EE{X} = \mu\) and \(\Var{X} = \sigma^2\) Then, we have \(X \sim f(\circ | \mu, p_1, \ldots, p_d)\) with \(d \geq 1\). Let, \(\Theta(\mu, p_1, \ldots, p_d) = \mu\). Taking sample \(X_1, X_2, \dots, X_n \iidd X\), we define the point estimator for \(\mu\) \(g: \R^n \to \R\) with \(g(X_1, X_2, \dots, X_n) = \frac{1}{n}\sum_{i=1}^n x_i\).
We have seen before, \(\EE{g(X_1, X_2, \dots, X_n)} = \mu\) and \(\Var{X_1, X_2, \dots, X_n} = \frac{\sigma^2}{n}\)
Exercises
Exercise 3.1 Consider the following steps:
- We generate n samples, say \(X_1, X_2, \dots, X_n\) from \(N(100,16^2)\). Compute the sample variance \(S^2\)
- Repeat the previous step \(1000\) times to generate \(S^2_i, 1 \leq i \leq 1000\) and plot histogram of \(\frac{(n-1)S^2_i}{16^2}\)
- Plot the density of true distribution of \(\frac{(n-1)S^2_1}{16^2}\)
Write a R-code to do the above for \(n = 8, 16\), and \(50\).
Exercise 3.2 Consider the following steps:
- We generate n samples, say \(X_1, X_2, \dots, X_n\) from \(N(0,1)\) and also generate another \(n\) samples, say \(Y_1, Y_2, \dots, Y_{n}\) from \(\chi^2(3)\) values.
- Plot histogram of \(\frac{X_i}{\sqrt{\frac{Y_i}{3}}}\)
- Plot the density of true distribution of \(\frac{X_1}{\sqrt{\frac{Y_1}{3}}}\).
Write a R-code to do the above for \(n = 50\), and \(100\).
The Student’s \(t-\)distribution was first discovered by a man named W.S. Gosset when working for an Irish brewery. He published it under the pseudonym Student14.