The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality, educational
resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: So for the last three lectures we're going to
talk about classical statistics, the way statistics
can be done if you don't want to assume a prior distribution
on the unknown parameters.
Today we're going to focus, mostly, on the estimation side
and leave hypothesis testing for the next two lectures.
So where there is one generic method that one can use to
carry out parameter estimation, that's the maximum
likelihood method.
We're going to define what it is.
Then we will look at the most common estimation problem
there is, which is to estimate the mean of a given
distribution.
And we're going to talk about confidence intervals, which
refers to providing an interval around your
estimates, which has some properties of the kind that
the parameter is highly likely to be inside that interval,
but we will be careful about how to interpret that
particular statement.
Ok.
So the big framework first.
The picture is almost the same as the one that we had in the
case of Bayesian statistics.
We have some unknown parameter.
And we have a measuring device.
There is some noise, some randomness.
And we get an observation, X, whose distribution depends on
the value of the parameter.
However, the big change from the Bayesian setting is that
here, this parameter is just a number.
It's not modeled as a random variable.
It does not have a probability distribution.
There's nothing random about it.
It's a constant.
It just happens that we don't know what that constant is.
And in particular, this probability distribution here,
the distribution of X, depends on Theta.
But this is not a conditional distribution in the usual
sense of the word.
Conditional distributions were defined when we had two random
variables and we condition one random variable on the other.
And we used the bar to separate the X from the Theta.
To make the point that this is not a conditioned
distribution, we use a different notation.
We put a semicolon here.
And what this is meant to say is that X has a distribution.
That distribution has a certain parameter.
And we don't know what that parameter is.
So for example, this might be a normal distribution, with
variance 1 but a mean Theta.
We don't know what Theta is.
And we want to estimate it.
Now once we have this setting, then your job is to design
this box, the estimator.
The estimator is some data processing box that takes the
measurements and produces an estimate
of the unknown parameter.
Now the notation that's used here is as if X and Theta were
one-dimensional quantities.
But actually, everything we say remains valid if you
interpret X and Theta as vectors of parameters.
So for example, you may obtain several
measurements, X1 up to 2Xn.
And there may be several unknown parameters in the
background.
Once more, we do not have, and we do not want to assume, a
prior distribution on Theta.
It's a constant.
And if you want to think mathematically about this
situation, it's as if you have many different
probabilistic models.
So a normal with this mean or a normal with that mean or a
normal with that mean, these are alternative candidate
probabilistic models.
And we want to try to make a decision about which one is
the correct model.
In some cases, we have to choose just between a small
number of models.
For example, you have a coin with an unknown bias.
The bias is either 1/2 or 3/4.
You're going to flip the coin a few times.
And you try to decide whether the true bias is this one or
is that one.
So in this case, we have two specific, alternative
probabilistic models from which we want to distinguish.
But sometimes things are a little more complicated.
For example, you have a coin.
And you have one hypothesis that my coin is unbiased.
And the other hypothesis is that my coin is biased.
And you do your experiments.
And you want to come up with a decision that decides whether
this is true or this one is true.
In this case, we're not dealing with just two
alternative probabilistic models.
This one is a specific model for the coin.
But this one actually corresponds to lots of
possible, alternative coin models.
So this includes the model where Theta is 0.6, the model
where Theta is 0.7, Theta is 0.8, and so on.
So we're trying to discriminate between one model
and lots of alternative models.
How does one go about this?
Well, there's some systematic ways that one can approach
problems of this kind.
And we will start talking about these next time.
So today, we're going to focus on estimation problems.
In estimation problems, theta is a quantity, which is a real
number, a continuous parameter.
We're to design this box, so what we get out of this box is
an estimate.
Now notice that this estimate here is a random variable.
Even though theta is deterministic, this is random,
because it's a function of the data that we observe.
The data are random.
We're applying a function to the data to
construct our estimate.
So, since it's a function of random variables, it's a
random variable itself.
The distribution of Theta hat depends on the distribution of
X. The distribution of X is affected by Theta.
So in the end, the distribution of your estimate
Theta hat will also be affected by whatever Theta
happens to be.
Our general objective, when designing estimators, is that
we want to get, in the end, an error, an estimation error,
which is not too large.
But we'll have to make that specific.
Again, what exactly do we mean by that?
So how do we go about this problem?
One general approach is to pick a Theta, under which the
data that we observe, that this is the X's, our most
likely to have occurred.
So I observe X. For any given Theta, I can calculate this
quantity, which tells me, under this particular Theta,
the X that you observed had this probability of occurring.
Under that Theta, the X that you observe had that
probability of occurring.
You just choose that Theta, which makes the data that you
observed most likely.
It's interesting to compare this maximum likelihood
estimate with the estimates that you would have, if you
were in a Bayesian setting, and you were using maximum
approach theory probability estimation.
In the Bayesian setting, what we do is, given the data, we
use the prior distribution on Theta.
And we calculate the posterior distribution of Theta given X.
Notice that this is sort of the opposite from
what we have here.
This is the probability of X for a particular value of
Theta, whereas this is the probability of Theta for a
particular X. So it's the opposite type of conditioning.
In the Bayesian setting, Theta is a random variable.
So we can talk about the probability
distribution of Theta.
So how do these two compare, except for this syntactic
difference that the order X's and Theta's are reversed?
Let's write down, in full detail, what this posterior
distribution of Theta is.
By the Bayes rule, this conditional distribution is
obtained from the prior, and the model of the measurement
process that we have.
And we get to this expression.
So in Bayesian estimation, we want to find the most likely
value of Theta.
And we need to maximize this quantity over
all possible Theta's.
First thing to notice is that the denominator is a constant.
It does not involve Theta.
So when you maximize this quantity, you don't care about
the denominator.
You just want to maximize the numerator.
Now, here, things start to look a little more similar.
And they would be exactly of the same kind, if that term
here was absent, it the prior was absent.
The two are going to become the same if that prior was
just a constant.
So if that prior is a constant, then maximum
likelihood estimation takes exactly the same form as
Bayesian maximum posterior probability estimation.
So you can give this particular interpretation of
maximum likelihood estimation.
Maximum likelihood estimation is essentially what you have
done, if you were in a Bayesian world, and you had
assumed a prior on the Theta's that's uniform, all the
Theta's being equally likely.
Okay.
So let's look at a simple example.
Suppose that the Xi's are independent, identically
distributed random variables, with a
certain parameter Theta.
So the distribution of each one of the Xi's is this
particular term.
So Theta is one-dimensional.
It's a one-dimensional parameter.
But we have several data.
We write down the formula for the probability of a
particular X vector, given a particular value of Theta.
But again, when I use the word, given, here it's not in
the conditioning sense.
It's the value of the density for a
particular choice of Theta.
Here, I wrote down, I defined maximum likelihood estimation
in terms of PMFs.
That's what you would do if the X's were
discrete random variables.
Here, the X's are continuous random variables, so instead
of I'm using the PDF instead of the PMF.
So this a definition, here, generalizes to the case of
continuous random variables.
And you use F's instead of X's, our usual recipe.
So the maximum likelihood estimate is defined.
Now, since the Xi's are independent, the joint density
of all the X's together is the product of
the individual densities.
So you look at this quantity.
This is the density or sort of probability of observing a
particular sequence of X's.
And we ask the question, what's the value of Theta that
makes the X's that we observe most likely?
So we want to carry out this maximization.
Now this maximization is just a calculational problem.
We're going to do this maximization by taking the
logarithm of this expression.
Maximizing an expression is the same as
maximizing the logarithm.
So the logarithm of this expression, the logarithm of a
product is the sum of the logarithms.
You get contributions from this Theta term.
There's n of these, so we get an n log Theta.
And then we have the sum of the logarithms of these terms.
It gives us minus Theta.
And then the sum of the X's.
So we need to maximize this expression
with respect to Theta.
The way to do this maximization is you take the
derivative, with respect to Theta.
And you get n over Theta equals to the sum of the X's.
And then you solve for Theta.
And you find that the maximum likelihood
estimate is this quantity.
Which sort of makes sense, because this is the reciprocal
of the sample mean of X's.
Theta, in an exponential distribution, we know that
it's 1 over (the mean of the exponential distribution).
So it looks like a reasonable estimate.
So in any case, this is the estimates that the maximum
likelihood estimation procedure tells us that we
should report.
This formula here, of course, tells you what to do if you
have already observed specific numbers.
If you have observed specific numbers, then you observe this
particular number as your estimate of Theta.
If you want to describe your estimation procedure more
abstractly, what you have constructed is an estimator,
which is a box that's takes in the random variables, capital
X1 up to Capital Xn, and produces out your estimate,
which is also a random variable.
Because it's a function of these random variables and is
denoted by an upper case Theta, to indicate that this
is now a random variable.
So this is an equality about numbers.
This is a description of the general procedure, which is an
equality between two random variables.
And this gives you the more abstract view of what we're
doing here.
All right.
So what can we tell about our estimate?
Is it good or is it bad?
So we should look at this particular random variable and
talk about the statistical properties that it has.
What we would like is this random variable to be close to
the true value of Theta, with high probability, no matter
what Theta is, since we don't know what Theta is.
Let's make a little more specific the
properties that we want.
So we cook up the estimator somehow.
So this estimator corresponds, again, to a box that takes
data in, the capital X's, and produces an
estimate Theta hat.
This estimate is random.
Sometimes it will be above the true value of Theta.
Sometimes it will be below.
Ideally, we would like it to not have a systematic error,
on the positive side or the negative side.
So a reasonable wish to have, for a good estimator, is that,
on the average, it gives you the correct value.
Now here, let's be a little more specific about what that
expectation is.
This is an expectation, with respect to the probability
distribution of Theta hat.
The probability distribution of Theta hat is affected by
the probability distribution of the X's.
Because Theta hat is a function of the X's.
And the probability distribution of the X's is
affected by the true value of Theta.
So depending on which one is the true value of Theta, this
is going to be a different expectation.
So if you were to write this expectation out in more
detail, it would look something like this.
You need to write down the probability
distribution of Theta hat.
And this is going to be some function.
But this function depends on the true Theta, is affected by
the true Theta.
And then you integrate this with respect to Theta hat.
What's the point here?
Again, Theta hat is a function of the X's.
So the density of Theta hat is affected by the
density of the X's.
The density of the X's is affected by the
true value of Theta.
So the distribution of Theta hat is affected by
the value of Theta.
Another way to put it is, as I've mentioned a few minutes
ago, in this business, it's as if we are considering
different possible probabilistic models, one
probabilistic model for each choice of Theta.
And we're trying to guess which one of these
probabilistic models is the true one.
One way of emphasizing the fact that this expression
depends on the true Theta is to put a little subscript
here, expectation, under the particular value of the
parameter Theta.
So depending on what value the true parameter Theta takes,
this expectation will have a different value.
And what we would like is that no matter what the true value
is, that our estimate will not have a bias on the positive or
the negative sides.
So this is a property that's desirable.
Is it always going to be true?
Not necessarily, it depends on what estimator we construct.
Is it true for our exponential example?
Unfortunately not, the estimate that we have in the
exponential example turns out to be biased.
And one extreme way of seeing this is to consider the case
where our sample size is 1.
We're trying to estimate Theta.
And the estimator from the previous slide, in that case,
is just 1/X1.
Now X1 has a fair amount of density in the vicinity of 0,
which means that 1/X1 has significant probability of
being very large.
And if you do the calculation, this ultimately makes the
expected value of 1/X1 to be infinite.
Now infinity is definitely not the correct value.
So our estimate is biased upwards.
And it's actually biased a lot upwards.
So that's how things are.
Maximum likelihood estimates, in general, will be biased.
But under some conditions, they will turn out to be
asymptotically unbiased.
That is, as you get more and more data, as your X vector is
longer and longer, with independent data, the estimate
that you're going to have, the expected value of your
estimator is going to get closer and
closer to the true value.
So you do have some nice asymptotic properties, but
we're not going to prove anything like this.
Speaking of asymptotic properties, in general, what
we would like to have is that, as you collect more and more
data, you get the correct answer, in some sense.
And the sense that we're going to use here is the limiting
sense of convergence in probability, since this is the
only notion of convergence of random variables that we have
in our hands.
This is similar to what we had in the pollster
problem, for example.
If we had a bigger and bigger sample size, we could be more
and more confident that the estimate that we obtained is
close to the unknown true parameter of the distribution
that we have.
So this is a desirable property.
If you have an infinitely large amount of data, you
should be able to estimate an unknown parameter
more or less exactly.
So this is it desirable property of estimators.
It turns out that maximum likelihood estimation, given
independent data, does have this property, under mild
conditions.
So maximum likelihood estimation, in this respect,
is a good approach.
So let's see, do we have this consistency property in our
exponential example?
In our exponential example, we used this quantity to estimate
the unknown parameter Theta.
What properties does this quantity have
as n goes to infinity?
Well this quantity is the reciprocal of that quantity up
here, which is the sample mean.
We know from the weak law of large numbers, that the sample
mean converges to the expectation.
So this property here comes from the weak
law of large numbers.
In probability, this quantity converges to the expected
value, which, for exponential distributions, is 1/Theta.
Now, if something converges to something, then the reciprocal
of that should converge to the reciprocal of that.
That's a property that's certainly correct for numbers.
But you're not talking about convergence of numbers.
We're talking about convergence in probability,
which is a more complicated notion.
Fortunately, it turns out that the same thing is true, when
we deal with convergence in probability.
One can show, although we will not bother doing this, that
indeed, the reciprocal of this, which is our estimate,
converges in probability to the reciprocal of that.
And that reciprocal is the true parameter Theta.
So for this particular exponential example, we do
have the desirable property, that as the number of data
becomes larger and larger, the estimate that we have
constructed will get closer and closer to the true
parameter value.
And this is true no matter what Theta is.
No matter what the true parameter Theta is, we're
going to get close to it as we collect more data.
Okay.
So these are two rough qualitative properties that
would be nice to have.
If you want to get a little more quantitative, you can
start looking at the mean squared error that your
estimator gives.
Now, once more, the comment I was making up there applies.
Namely, that this expectation here is an expectation with
respect to the probability distribution of Theta hat that
corresponds to a particular value of little theta.
So fix a little theta.
Write down this expression.
Look at the probability distribution of Theta hat,
under that little theta.
And do this calculation.
You're going to get some quantity that depends on the
little theta.
And so all quantities in this equality here should be
interpreted as quantities under that particular value of
little theta.
So if you wanted to make this more explicit, you could start
throwing little subscripts everywhere in those
expressions.
And let's see what those expressions tell us.
The expected value squared of a random variable, we know
that it's always equal to the variance of this random
variable, plus the expectation of the
random variable squared.
So the expectation value of that random variable, squared.
This equality here is just our familiar formula, that the
expected value of X squared is the variance of X plus the
expected value of X squared.
So we apply this formula to X equal to
Theta hat minus Theta.
Now, remember that, in this classical setting, theta is
just a constant.
We have fixed Theta.
We want to calculate the variance of this quantity,
under that particular Theta.
When you add or subtract a constant to a random variable,
the variance doesn't change.
This is the same as the variance of our estimator.
And what we've got here is the bias of our estimate.
It tells us, on the average, whether we
fall above or below.
And we're taking the bias to be b squared.
If we have an unbiased estimator, the bias
term will be 0.
So ideally we want Theta hat to be very close to Theta.
And since Theta is a constant, if that happens, the variance
of Theta hat would be very small.
So Theta is a constant.
If Theta hat has a distribution that's
concentrated just around own little theta, then Theta hat
would have a small variance.
So this is one desire that have.
We're going to have a small variance.
But we also want to have a small bias at the same time.
So the general form of the mean squared error has two
contributions.
One is the variance of our estimator.
The other is the bias.
And one usually wants to design an estimator that
simultaneously keeps both of these terms small.
So here's an estimation method that would do very well with
respect to this term, but badly with
respect to that term.
So suppose that my distribution is, let's say,
normal with an unknown mean Theta and variance 1.
And I use as my estimator something very dumb.
I always produce an estimate that says my estimate is 100.
So I'm just ignoring the data and report 100.
What does this do?
The variance of my estimator is 0.
There's no randomness in the estimate that I report.
But the bias is going to be pretty bad.
The bias is going to be Theta hat, which is 100 minus the
true value of Theta.
And for some Theta's, my bias is going to be horrible.
If my true Theta happens to be 0, my bias
squared is a huge term.
And I get a large error.
So what's the moral of this example?
There are ways of making that variance very small, but, in
those cases, you pay a price in the bias.
So you want to do something a little more delicate, where
you try to keep both terms small at the same time.
So these types of considerations become
important when you start to try to design sophisticated
estimators for more complicated problems.
But we will not do this in this class.
This belongs to further classes on
statistics and inference.
For this class, for parameter estimation, we will basically
stick to two very simple methods.
One is the maximum likelihood method we've just discussed.
And the other method is what you would do if you were still
in high school and didn't know any probability.
You get data.
And these data come from some distribution
with an unknown mean.
And you want to estimate that the unknown mean.
What would you do?
You would just take those data and average them out.
So let's make this a little more specific.
We have X's that come from a given distribution.
We know the general form of the distribution, perhaps.
We do know, perhaps, the variance of that distribution,
or, perhaps, we don't know it.
But we do not know the mean.
And we want to estimate the mean of that distribution.
Now, we can write this situation.
We can represent it in a different form.
The Xi's are equal to Theta.
This is the mean.
Plus a 0 mean random variable, that you
can think of as noise.
So this corresponds to the usual situation you would have
in a lab, where you go and try to
measure an unknown quantity.
You get lots of measurements.
But each time that you measure them, your measurements have
some extra noise in there.
And you want to kind of get rid of that noise.
The way to try to get rid of the measurement noise is to
collect lots of data and average them out.
This is the sample mean.
And this is a very, very reasonable way of trying to
estimate the unknown mean of the X's.
So this is the sample mean.
It's a reasonable, plausible, in general, pretty good
estimator of the unknown mean of a certain distribution.
We can apply this estimator without really knowing a lot
about the distribution of the X's.
Actually, we don't need to know anything about the
distribution.
We can still apply it, because the variance, for example,
does not show up here.
We don't need to know the variance to
calculate that quantity.
Does this estimator have good properties?
Yes, it does.
What's the expected value of the sample mean?
If the expectation of this, it's the expectation of this
sum divided by n.
The expected value for each one of the X's is Theta.
So the expected value of the sample mean
is just Theta itself.
So our estimator is unbiased.
No matter what Theta is, our estimator does not have a
systematic error in either direction.
Furthermore, the weak law of large numbers tells us that
this quantity converges to the true parameter in probability.
So it's a consistent estimator.
This is good.
And if you want to calculate the mean squared error
corresponding to this estimator.
Remember how we defined the mean squared error?
It's this quantity.
Then it's a calculation that we have done a fair number of
times by now.
The mean squared error is the variance of the distribution
of the X's divided by n.
So as we get more and more data, the mean squared error
goes down to 0.
In some examples, it turns out that the sample mean is also
the same as the maximum likelihood estimate.
For example, if the X's are coming from a normal
distribution, you can write down the likelihood, do the
maximization with respect to Theta, you'll find that the
maximum likelihood estimate is the same as the sample mean.
In other cases, the sample mean will be different from
the maximum likelihood.
And then you have a choice about which one of the
two you would use.
Probably, in most reasonable situations, you would just use
the sample mean, because it's simple, easy to compute, and
has nice properties.
All right.
So you go to your boss.
And you report and say, OK, I did all my
experiments in the lab.
And the average value that I got is a certain number, 2.37.
So is that the informative to your boss?
Well your boss would like to know how much they can trust
this number, 2.37.
Well, I know that the true value is not going to be
exactly that.
But how close should it be?
So give me a range of what you think are
possible values of Theta.
So the situation is like this.
So suppose that we observe X's that are coming from a certain
distribution.
And we're trying to estimate the mean.
We get our data.
Maybe our data looks something like this.
You calculate the mean.
You find the sample mean.
So let's suppose that the sample mean is a number, for
some reason take to be 2.37.
But you want to convey something to your boss about
how spread out these data were.
So the boss asks you to give him or her some kind of
interval on which Theta, the true parameter, might lie.
So the boss asked you for an interval.
So what you do is you end up reporting an interval.
And you somehow use the data that you have seen to
construct this interval.
And you report to your boss also the
endpoints of this interval.
Let's give names to these endpoints,
Theta_n- and Theta_n+.
The ends here just play the role of keeping track of how
many data we're using.
So what you report to your boss is this interval as well.
Are these Theta's here, the endpoints of the interval,
lowercase or uppercase?
What should they be?
Well you construct these intervals after
you see your data.
You take the data into account to construct your interval.
So these definitely should depend on the data.
And therefore they are random variables.
Same thing with your estimator, in general, it's
going to be a random variable.
Although, when you go and report numbers to your boss,
you give the specific realizations of the random
variables, given the data that you got.
So instead of having just a single box
that produces estimates.
So our previous picture was that you have your estimator
that takes X's and produces Theta hats.
Now our box will also be producing Theta hats minus and
Theta hats plus.
It's going to produce an interval as well.
The X's are random, therefore these quantities are random.
Once you go and do the experiment and obtain your
data, then your data will be some
lowercase x, specific numbers.
And then your estimates and estimator
become also lower case.
What would we like this interval to do?
We would like it to be highly likely to contain the true
value of the parameter.
So we might impose some specs of the following kind.
I pick a number, alpha.
Usually that alpha, think of it as a
probability of a large error.
Typical value of alpha might be 0.05, in which case this
number here is point 0.95.
And you're given specs that say something like this.
I would like, with probability at least 0.95, this to happen,
which says that the true parameter lies inside the
confidence interval.
Now let's try to interpret this statement.
Suppose that you did the experiment, and that you ended
up reporting to your boss a confidence interval
from 1.97 to 2.56.
That's what you report to your boss.
And suppose that the confidence
interval has this property.
Can you go to your boss and say, with probability 95%, the
true value of Theta is between these two numbers?
Is that a meaningful statement?
So the statement is, the tentative statement is, with
probability 95%, the true value of Theta is
between 1.97 and 2.56.
Well, what is random in that statement?
There's nothing random.
The true value of theta is a constant.
1.97 is a number.
2.56 is a number.
So it doesn't make any sense to talk about the probability
that theta is in this interval.
Either theta happens to be in that interval, or it
happens to not be.
But there are no probabilities associated with this.
Because theta is not random.
Syntactically, you can see this.
Because theta here is a lower case.
So what kind of probabilities are we talking about here?
Where's the randomness?
Well the random thing is the interval.
It's not theta.
So the statement that is being made here is that the
interval, that's being constructed by our procedure,
should have the property that, with probability 95%, it's
going to fall on top of the true value of theta.
So the right way of interpreting what the 95%
confidence interval is, is something like the following.
We have the true value of theta that we don't know.
I get data.
Based on the data, I construct a confidence interval.
I get my confidence interval.
I got lucky.
And the true value of theta is in here.
Next day, I do the same experiment, take my data,
construct a confidence interval.
And I get this confidence interval, lucky once more.
Next day I get data.
I use my data to come up with an estimate of theta and the
confidence interval.
That day, I was unlucky.
And I got a confidence interval out there.
What the requirement here is, is that 95% of the days, where
we use this certain procedure for constructing confidence
intervals, 95% of those days, we will be lucky.
And we will capture the correct value of theta by your
confidence interval.
So it's a statement about the distribution of these random
confidence intervals, how likely are they to fall on top
of the true theta, as opposed to how likely
they are to fall outside.
So it's a statement about probabilities associated with
a confidence interval.
They're not probabilities about theta, because theta,
itself, is not random.
So this is what the confidence interval is, in general, and
how we interpret it.
How do we construct a 95% confidence interval?
Let's go through this exercise, in
a particular example.
The calculations are exactly the same as the ones that you
did when we talked about laws of large numbers and the
central limit theorem.
So there's nothing new calculationally but it's,
perhaps, new in terms of the language that we use and the
interpretation.
So we got our sample mean from some distribution.
And we would like to calculate a 95% confidence interval.
We know from the normal tables, that the standard
normal has 2.5% on the tail, that's after 1.96.
Yes, by this time, the number 1.96
should be pretty familiar.
So if this probability here is 2.5%, this
number here is 1.96.
Now look at this random variable here.
This is the sample mean.
Difference, from the true mean, normalized by the usual
normalizing factor.
By the central limit theorem, this is approximately normal.
So it has probability 0.95 of being less than 1.96.
Now take this event here and rewrite it.
This the event, well, that Theta hat minus theta is
bigger than this number and smaller than that number.
This event here is equivalent to that event here.
And so this suggests a way of constructing our 95% percent
confidence interval.
I'm going to report the interval, which gives this as
the lower end of the confidence interval, and gives
this as the upper end of the confidence interval
In other words, at the end of the experiment, we report the
sample mean, which is our estimate.
And we report also, an interval
around the sample mean.
And this is our 95% confidence interval.
The confidence interval becomes
smaller, when n is larger.
In some sense, we're more certain that we're doing a
good estimation job, so we can have a small interval and
still be quite confident that our interval captures the true
value of the parameter.
Also, if our data have very little noise, when you have
more accurate measurements, you're more confident that
your estimate is pretty good.
And that results in a smaller confidence interval, smaller
length of the confidence interval.
And still you have 95% probability of capturing the
true value of theta.
So we did this exercise by taking 95% confidence
intervals and the corresponding value from the
normal tables, which is 1.96.
Of course, you can do it more generally, if you set your
alpha to be some other number.
Again, you look at the normal tables.
And you find the value here, so that the tail has
probability alpha over 2.
And instead of using these 1.96, you use whatever number
you get from the normal tables.
And this tells you how to construct
a confidence interval.
Well, to be exact, this is not necessarily a
95% confidence interval.
It's approximately a 95% confidence interval.
Why is this?
Because we've done an approximation.
We have used the central limit theorem.
So it might turn out to be a 95.5% confidence interval
instead of 95%, because our calculations are
not entirely accurate.
But for reasonable values of n, using the central limit
theorem is a good approximation.
And that's what people almost always do.
So just take the value from the normal tables.
Okay, except for one catch.
I used the data.
I obtained my estimate.
And I want to go to my boss and report this theta minus
and theta hat, which is the confidence interval.
What's the difficulty?
I know what n is.
But I don't know what sigma is, in general.
So if I don't know sigma, what am I going to do?
Here, there's a few options for what you can do.
And the first option is familiar from what we did when
we talked about the pollster problem.
We don't know what sigma is, but maybe we have an upper
bound on sigma.
For example, if the Xi's Bernoulli random variables, we
have seen that the standard deviation is at most 1/2.
So use the most conservative value for sigma.
Using the most conservative value means that you take
bigger confidence intervals than necessary.
So that's one option.
Another option is to try to estimate sigma from the data.
How do you do this estimation?
In special cases, for special types of distributions, you
can think of heuristic ways of doing this estimation.
For example, in the case of Bernoulli random variables, we
know that the true value of sigma, the standard deviation
of a Bernoulli random variable, is the square root
of theta1 minus theta, where theta is
the mean of the Bernoulli.
Try to use this formula.
But theta is the thing we're trying to estimate in the
first place.
We don't know it.
What do we do?
Well, we have an estimate for theta, the estimate, produced
by our estimation procedure, the sample mean.
So I obtain my data.
I get my data.
I produce the estimate theta hat.
It's an estimate of the mean.
Use that estimate in this formula to come up with an
estimate of my standard deviation.
And then use that standard deviation, in the construction
of the confidence interval, pretending
that this is correct.
Well the number of your data is large, then we know, from
the law of large numbers, that theta hat is a pretty good
estimate of theta.
So sigma hat is going to be a pretty good estimate of sigma.
So we're not making large errors by using this approach.
So in this scenario here, things were simple, because we
had an analytical formula.
Sigma was determined by theta.
So we could come up with a quick and
dirty estimate of sigma.
In general, if you do not have any nice formulas of this
kind, what could you do?
Well, you still need to come up with an
estimate of sigma somehow.
What is a generic method for
estimating a standard deviation?
Equivalently, what could be a generic method for estimating
a variance?
Well the variance is an expected value
of some random variable.
The variance is the mean of the random variable inside of
those brackets.
How does one estimate the mean of some random variable?
You obtain lots of measurements of that random
variable and average them out.
So this would be a reasonable way of estimating the variance
of a distribution.
And again, the weak law of large numbers tells us that
this average converges to the expected value of this, which
is just the variance of the distribution.
So we got a nice and consistent way
of estimating variances.
But now, we seem to be getting in a vicious circle here,
because to estimate the variance, we
need to know the mean.
And the mean is something we're trying to estimate in
the first place.
Okay.
But we do have an estimate from the mean.
So a reasonable approximation, once more, is to plug-in,
here, since we don't know the mean, the
estimate of the mean.
And so you get that expression, but with a theta
hat instead of theta itself.
And this is another reasonable way of
estimating the variance.
It does have the same consistency properties.
Why?
When n is large, this is going to behave the same as that,
because theta hat converges to theta.
And when n is large, this is approximately the same as
sigma squared.
So for a large n, this quantity also converges to
sigma squared.
And we have a consistent estimate of
the variance as well.
And we can take that consistent estimate and use it
back in the construction of confidence interval.
One little detail, here, we're dividing by n.
Here, we're dividing by n-1.
Why do we do this?
Well, it turns out that's what you need to do for these
estimates to be an unbiased estimate of the variance.
One has to do a little bit of a calculation, and one finds
that that's the factor that you need to have here in order
to be unbiased.
Of course, if you get 100 data points, whether you divide by
100 or divided by 99, it's going to make only a tiny
difference in your estimate of your variance.
So it's going to make only a tiny difference in your
estimate of the standard deviation.
It's not a big deal.
And it doesn't really matter.
But if you want to show off about your deeper knowledge of
statistics, you throw in the 1 over n-1 factor in there.
So now one basically needs to put together this story here,
how you estimate the variance.
You first estimate the sample mean.
And then you do some extra work to come up with a
reasonable estimate of the variance and
the standard deviation.
And then you use your estimate, of the standard
deviation, to come up with a confidence interval, which has
these two endpoints.
In doing this procedure, there's basically a number of
approximations that are involved.
There are two types of approximations.
One approximation is that we're pretending that the
sample mean has a normal distribution.
That's something we're justified to do, by the
central limit theorem.
But it's not exact.
It's an approximation.
And the second approximation that comes in is that, instead
of using the correct standard deviation, in general, you
will have to use some approximation of
the standard deviation.
Okay so you will be getting a little bit of practice with
these concepts in recitation and tutorial.
And we will move on to new topics next week.
But the material that's going to be covered in the final
exam is only up to this point.
So next week is just general education.
Hopefully useful, but it's not in the exam.