The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: So we're going to start now with a new chapter.
We're going to talk about Markov processes.
The good news is that this is a subject that is a lot more
intuitive and simple in many ways than, let's say, the
Poisson processes.
So hopefully this will be enjoyable.
So Markov processes is, a general
class of random processes.
In some sense, it's more elaborate than the Bernoulli
and Poisson processes, because now we're going to have
dependencies between difference times, instead of
having memoryless processes.
So the basic idea is the following.
In physics, for example, you write down equations for how a
system evolves that has the general form.
The new state of a system one second later is some function
of old state.
So Newton's equations and all that in physics allow you to
write equations of this kind.
And so if that a particle is moving at a certain velocity
and it's at some location, you can predict when it's going to
be a little later.
Markov processes have the same flavor, except that there's
also some randomness thrown inside the equation.
So that's what Markov process essentially is.
It describes the evolution of the system, or some variables,
but in the presence of some noise so that the motion
itself is a bit random.
So this is a pretty general framework.
So pretty much any useful or interesting random process
that you can think about, you can always described it as a
Markov process if you define properly the
notion of the state.
So what we're going to do is we're going to introduce the
class of Markov processes by, example, by talking about the
checkout counter in a supermarket.
Then we're going to abstract from our example so that we
get a more general definition.
And then we're going to do a few things, such as how to
predict what's going to happen n time steps later, if we
start at the particular state.
And then talk a little bit about some structural
properties of Markov processes or Markov chains.
So here's our example.
You go to the checkout counter at the supermarket, and you
stand there and watch the customers who come.
So customers come, they get in queue, and customers get
served one at a time.
So the discussion is going to be in terms of supermarket
checkout counters, but the same story applies to any
service system.
You may have a server, jobs arrive to that server, they
get put into the queue, and the server processes those
jobs one at a time.
Now to make a probabilistic model, we need to make some
assumption about the customer arrivals and the customer
departures.
And we want to keep things as simple as
possible to get started.
So let's assume that customers arrive according to a
Bernoulli process with some parameter b.
So essentially, that's the same as the assumption that
the time between consecutive customer arrivals is a
geometric random variable with parameter b.
Another way of thinking about the arrival process--
that's not how it happens, but it's helpful, mathematically,
is to think of someone who's flipping a coin with bias
equal to b.
And whenever the coin lands heads,
then a customer arrives.
So it's as if there's a coin flip being done by nature that
decides the arrivals of the customers.
So we know that coin flipping to determine the customer
arrivals is the same as having geometric inter-arrival times.
We know that from our study of the Bernoulli process.
OK.
And now how about the customer service times.
We're going to assume that--
OK.
If there is no customer in queue, no one being served,
then of course, no one is going to
depart from the queue.
But if there a customer in queue, then that customer
starts being served, and is going to be served for a
random amount of time.
And we make the assumption that the time it takes for the
clerk to serve the customer has a geometric distribution
with some known parameter q.
So the time it takes to serve a customer is random, because
it's random how many items they got in their cart, and
how many coupons they have to unload and so on.
So it's random.
In the real world, it has some probability distribution.
Let's not care exactly about what it would be in the real
world, but as a modeling approximation or just to get
started, let's pretend that customer service time are well
described by a geometric distribution,
with a parameter q.
An equivalent way of thinking about the customer service,
mathematically, would be, again, in
terms of coin flipping.
That is, the clerk has a coin with a bias, and at each time
slot the clerk flips the coin.
With probability q, service is over.
With probability 1-q, you continue the service process.
An assumption that we're going to make is that the coin flips
that happen here to determine the arrivals, they're all
independent of each other.
The coin flips that determine the end of service are also
independent from each other.
But also the coin flips involved here are independent
from the coin flips that happened there.
So how arrivals happen is independent with what happens
at the service process.
OK.
So suppose now you want to answer a
question such as the following.
The time is 7:00 PM.
What's the probability that the customer will be departing
at this particular time?
Well, you say, it depends.
If the queue is empty at that time, then you're certain that
you're not going to have a customer departure.
But if the queue is not empty, then there is probability q
that a departure will happen at that time.
So the answer to a question like this has something to do
with the state of the system at that time.
It depends what the queue is.
And if I ask you, will the queue be empty at 7:10?
Well, the answer to that question depends on whether at
7 o'clock whether the queue was huge or not.
So knowing something about the state of the queue right now
gives me relevant information about what may
happen in the future.
So what is the state of the system?
Therefore we're brought to start using this term.
So the state basically corresponds to
anything that's relevant.
Anything that's happening right now that's kind of
relevant to what may happen in the future.
Knowing the size of the queue right now, is useful
information for me to make predictions about what may
happen 2 minutes later from now.
So in this particular example, a reasonable choice for the
state is to just count how many customers
we have in the queue.
And let's assume that our supermarket building is not
too big, so it can only hold 10 people.
So we're going to limit the states.
Instead of going from 0 to infinity, we're going to
truncate our model at ten.
So we have 11 possible states, corresponding to 0 customers
in queue, 1 customer in queue, 2 customers, and so on, all
the way up to 10.
So these are the different possible states of the system,
assuming that the store cannot handle more than 10 customers.
So this is the first step, to write down the set of possible
states for our system.
Then the next thing to do is to start describing the
possible transitions between the states.
At any given time step, what are the
things that can happen?
We can have a customer arrival, which
moves the state 1 higher.
We can have a customer departure, which moves the
state 1 lower.
There's a possibility that nothing happens, in which case
the state stays the same.
And there's also the possibility of having
simultaneously an arrival and a departure, in which case the
state again stays the same.
So let's write some representative probabilities.
If we have 2 customers, the probability that during this
step we go down, this is the probability that we have a
service completion, but to no customer arrival.
So this is the probability associated with this
transition.
The other possibility is that there's a customer arrival,
which happens with probability p, and we do not have a
customer departure, and so the probability of that particular
transition is this number.
And then finally, the probability that we stay in
the same state, this can happen in 2 possible ways.
One way is that we have an arrival and a departure
simultaneously.
And the other possibility is that we have no arrival and no
departure, so that the state stays the same.
So these transition probabilities would be the
same starting from any other states, state 3, or
state 9, and so on.
Transition probabilities become a little different at
the borders, at the boundaries of this diagram, because if
you're in a state 0, then you cannot have any customer
departures.
There's no one to be served, but there is a probability p
that the customer arrives, in which case the number of
customers in the system goes to 1.
Then probability 1-p, nothing happens.
Similarly with departures, if the system is full, there's no
room for another arrival.
But we may have a departure that happens with probability
q, and nothing happens with probability 1-q.
So this is the full transition diagram annotated with
transition probabilities.
And this is a complete description of a discrete
time, finite state Markov chain.
So this is a complete probabilistic model.
Once you have all of these pieces of information, you can
start calculating things, and trying to predict what's going
to happen in the future.
Now let us abstract from this example and come up with a
more general definition.
So we have this concept of the state which describes the
current situation in the system that we're looking at.
The current state is random, so we're going to think of it
as a random variable Xn is the state, and transitions after
the system started operating.
So the system starts operating at some initial state X0, and
after n transitions, it moves to state Xn.
Now we have a set of possible states.
State 1 state 2, state 3, and in general,
state i and state j.
To keep things simple, we assume that the set of
possible states is a finite set.
As you can imagine, we can have systems in which the
state space is going to be infinite.
It could be discrete, or continuous.
But all that is more difficult and more complicated.
It makes sense to start from the simplest possible setting
where we just deal with the finite state space.
And time is discrete, so we can think of this state in the
beginning, after 1 transition, 2 transitions, and so on.
So we're in discrete time and we have finite in many states.
So the system starts somewhere, and at every time
step, the state is, let's say, here.
A whistle blows, and the state jumps to a random next state.
So it may move here, or it may move there, or it may move
here, or it might stay in the place.
So one possible transition is the transition before you
jump, and just land in the same place
where you started from.
Now we want to describe the statistics of these
transitions.
If I am at that state, how likely is it to that, next
time, I'm going to find myself at that state?
Well, we describe the statistics of this transition
by writing down a transition probability, the transition
probability of going from state 3 to state 1.
So this transition probability is to be thought of as a
conditional probability.
Given that right now I am at state i what is the
probability that next time I find myself at state j?
So given that right now I am at state 3, P31 is the
probability that the next time I'm going to find
myself at state 1.
Similarly here, we would have a probability P3i, which is
the probability that given that right now I'm at state 3,
next time I'm going to find myself at state i.
Now one can write such conditional probabilities down
in principle, but we need to make--
so you might think of this as a definition here, but we need
to make one additional big assumption, and this is the
assumption that to make a process
to be a Markov process.
This is the so-called Markov property, and
here's what it says.
Let me describe it first in words here.
Every time that I find myself at state 3, the probability
that next time I'm going to find myself at state 1 is this
particular number, no matter how I got there.
That is, this transition probability is not affected by
the past of the process.
It doesn't care about what path I used to find
myself at state 3.
Mathematically, it means the following.
You have this transition probability that from state i
jump to state j.
Suppose that I gave you some additional information, that I
told you everything else that happened in the past of the
process, everything that happened, how did you
get to state i?
The assumption we're making is that this information about
the past has no bearing in making predictions about the
future, as long as you know where you are right now.
So if I tell you, right now, you are at state i, and by the
way, you got there by following a particular path,
you can ignore the extra information of the particular
path that you followed.
You only take into account where you are right now.
So every time you find yourself at that state, no
matter how you got there, you will find yourself next time
at state 1 with probability P31.
So the past has no bearing into the future, as long as
you know where you are sitting right now.
For this property to happen, you need to choose your state
carefully in the right way.
In that sense, the states needs to include any
information that's relevant about the
future of the system.
Anything that's not in the state is not going to play a
role, but the state needs to have all the information
that's relevant in determining what kind of transitions are
going to happen next.
So to take an example, before you go to Markov process, just
from the deterministic world, if you have a ball that's
flying up in the air, and you want to make predictions about
the future.
If I tell you that the state of the ball is the position of
the ball at the particular time, is that enough for you
to make predictions where the ball is going to go next?
No.
You need to know both the position and the velocity.
If you know position and velocity, you can make
predictions about the future.
So the state of a ball that's flying is position together
with velocity.
If you were to just take position, that would not be
enough information, because if I tell you current position,
and then I tell you past position, you could use the
information from the past position to complete the
trajectory and to make the prediction.
So information from the past is useful if you don't know
the velocity.
But if both position and velocity, you don't care how
you got there, or what time you started.
From position and velocity, you can make predictions about
the future.
So there's a certain art, or a certain element of thinking, a
non-mechanical aspect into problems of this kind, to
figure out which is the right state variable.
When you define the state of your system, you need to
define it in such a way that includes all information that
has been accumulated that has some relevance for the future.
So the general process for coming up with a Markov model
is to first make this big decision of what your state
variable is going to be.
Then you write down if it may be a picture of
the different states.
Then you identify the possible transitions.
So sometimes the diagram that you're going to have will not
include all the possible arcs.
You would only show those arcs that correspond to transitions
that are possible.
For example, in the supermarket example, we did
not have a transition from state 2 to state 5, because
that cannot happen.
You can only have 1 arrival at any time.
So in the diagram, we only showed the possible
transitions.
And for each of the possible transitions, then you work
with the description of the model to figure out the
correct transition probability.
So you got the diagram by writing down transition
probabilities.
OK, so suppose you got your Markov model.
What will you do with it?
Well, what do we need models for?
We need models in order to make predictions, to make
probabilistic predictions.
So for example, I tell you that the process started in
that state.
You let it run for some time.
Where do you think it's going to be 10 time steps from now?
That's a question that you might want to answer.
Since the process is random, there's no way for you to tell
me exactly where it's going to be.
But maybe you can give me probabilities.
You can tell me, with so much probability, the
state would be there.
With so much probability, the state would be
there, and so on.
So our first exercise is to calculate those probabilities
about what may happen to the process a number of steps in
the future.
It's handy to have some notation in here.
So somebody tells us that this process starts at the
particular state i.
We let the process run for n transitions.
It may land at some state j, but that state j at which it's
going to land is going to be random.
So we want to give probabilities.
Tell me, with what probability the state, n times steps
later, is going to be that particular state j?
The shorthand notation is to use this symbol here for the
n-step transition probabilities that you find
yourself at state j given that you started at state i.
So the way these two indices are ordered, the way to think
about them is that from i, you go to j.
So the probability that from i you go to j if you have n
steps in front of you.
Some of these transition probabilities are, of course
easy to write.
For example, in 0 transitions, you're going to be exactly
where you started.
So this probability is going to be equal to 1 if i is equal
to j, And 0 if i is different than j.
That's an easy one to write down.
If you have only 1 transition, what's the probability that 1
step later you find yourself in state j given that you
started at state i?
What is this?
These are just the ordinary 1-step transition
probabilities that we are given in the description of
the problem.
So by definition, the 1-step transition probabilities are
of this form.
This equality is correct just because of the way that we
defined those two quantities.
Now we want to say something about the n-step transition
probabilities when n is a bigger number.
OK.
So here, we're going to use the total probability theorem.
So we're going to condition in two different scenarios, and
break up the calculation of this quantity, by considering
the different ways that this event can happen.
So what is the event of interest?
The event of interest is the following.
At time 0 we start i.
We are interested in landing at time n at the
particular state j.
Now this event can happen in several different ways, in
lots of different ways.
But let us group them into subgroups.
One group, or one sort of scenario, is the following.
During the first n-1 time steps, things happen, and
somehow you end up at state 1.
And then from state 1, in the next time step you make a
transition to state j.
This particular arc here actually corresponds to lots
and lots of different possible scenarios, or different spots,
or different transitions.
In n-1 time steps, there's lots of possible ways by which
you could end up at state 1.
Different paths through the state space.
But all of them together collectively have a
probability, which is the (n-1)-step transition
probability, that from state i, you end up at state 1
And then there's other possible scenarios.
Perhaps in the first n-1 time steps, you follow the
trajectory that took you at state m.
And then from state m, you did this transition, and you ended
up at state j.
So this diagram breaks up the set of all possible
trajectories from i to j into different collections, where
each collection has to do with which one happens to be the
state just before the last time step, just before time n.
And we're going to condition on the state at time n-1.
So the total probability of ending up at state j is the
sum of the probabilities of the different scenarios --
the different ways that you can get to state j.
If we look at that type of scenario, what's the
probability of that scenario happening?
With probability Ri1(n-1), I find myself at
state 1 at time n-1.
This is just by the definition of these multi-step transition
probabilities.
This is the number of transitions.
The probability that from state i, I end up at state 1.
And then given that I found myself at state 1, with
probability P1j, that's the transition probability, next
time I'm going to find myself at state j.
So the product of these two is the total probability of my
getting from state i to state j through state
1 at the time before.
Now where exactly did we use the Markov assumption here?
No matter which particular path we used to get from i to
state 1, the probability that next I'm going to make this
transition is that same number, P1j.
So that number does not depend on the particular path that I
followed in order to get there.
If we didn't have the Markov assumption, we should have
considered all possible individual trajectories here,
and then we would need to use the transition probability
that corresponds to that particular trajectory.
But because of the Markov assumption, the only thing
that matters is that right now we are at state 1.
It does not matter how we got there.
So now once you see this scenario, then this scenario,
and that scenario, and you add the probabilities of these
different scenarios, you end up with this formula here,
which is a recursion.
It tells us that once you have computed the (n-1)-step
transition probabilities, then you can compute also the
n-step transition probabilities.
This is a recursion that you execute or you run for all i's
and j's simultaneously.
That is fixed.
And for a particular n, you calculate this quantity for
all possible i's, j's, k's.
You have all of those quantities, and then you use
this equation to find those numbers again for all the
possible i's and j's.
Now this is formula which is always true, and there's a big
idea behind the formula.
And now there's variations of this formula, depending on
whether you're interested in something
that's slightly different.
So for example, if you were to have a random initial state,
somebody gives you the probability distribution of
the initial state, so you're told that with probability
such and such, you're going to start at state 1.
With that probability, you're going to start at
state 2, and so on.
And you want to find the probability at the time n you
find yourself at state j.
Well again, total probability theorem, you condition on the
initial state.
With this probability you find yourself at that particular
initial state, and given that this is your initial state,
this is the probability that n time steps later you find
yourself at state j.
Now building again on the same idea, you can run every
recursion of this kind by conditioning
at different times.
So here's a variation.
You start at state i.
After 1 time step, you find yourself at state 1, with
probability pi1, and you find yourself at state m with
probability Pim.
And once that happens, then you're going to follow some
trajectories.
And there is a possibility that you're going to end up at
state j after n-1 time steps.
This scenario can happen in many possible ways.
There's lots of possible paths from state 1 to state j.
There's many paths from state 1 to state j.
What is the collective probability of all these
transitions?
This is the event that, starting from state 1, I end
up at state j in n-1 time steps.
So this one has here probability R1j of n-1.
And similarly down here.
And then by using the same way of thinking as before, we get
the formula that Rij(n) is the sum over all k's of Pik, and
then the Rkj(n-1).
So this formula looks almost the same as this one, but it's
actually different.
The indices and the way things work out are a bit different,
but the basic idea is the same.
Here we use the total probability theory by
conditioning on the state just 1 step before the end of our
time horizon.
Here we use total probability theorem by conditioning on the
state right after the first transition.
So this generally idea has different variations.
They're all valid, and depending on the context that
you're dealing with, you might want to work with one of these
or another.
So let's illustrate these calculations
in terms of an example.
So in this example, we just have 2 states, and somebody
gives us transition probabilities to be those
particular numbers.
Let's write down the equations.
So the probability that starting from state 1, I find
myself at state 1 n time steps later.
This can happen in 2 ways.
At time n-1, I might find myself at state 2.
And then from state 2, I make a transition back to state 1,
which happens with probability--
why'd I put 2 there --
anyway, 0.2.
And another way is that from state 1, I go to state 1 in
n-1 steps, and then from state 1 I stay where I am, which
happens with probability 0.5.
So this is for R11(n).
Now R12(n), we can write a similar
recursion for this one.
On the other hand, seems these are probabilities.
The state at time n is going to be either
state 1 or state 2.
So these 2 numbers need to add to 1, so we can just write
this as 1 - R11(n).
And this is an enough of a recursion to propagate R11 and
R12 as time goes on.
So after n-1 transitions, either I find myself in state
2, and then there's a point to transition that I go to 1, or
I find myself in state 1, which with that probability,
and from there, I have probability 0.5 of staying
where I am.
Now let's start calculating.
As we discussed before, if I start at state 1, after 0
transitions I'm certain to be at state , and I'm certain not
to be at state 1.
If I start from state 1, I'm certain to not to be at state
at that time, and I'm certain that I am right
now, it's state 1.
After I make transition, starting from state 1, there's
probability 0.5 that I stay at state 1.
And there's probability 0.5 that I stay at state 2.
If I were to start from state 2, the probability that I go
to 1 in 1 time step is this transition that has
probability 0.2, and the other 0.8.
OK.
So the calculation now becomes more interesting, if we want
to calculate the next term.
How likely is that at time 2, I find myself at state 1?
In order to be here at state 1, this can happen in 2 ways.
Either the first transition left me there, and the second
transition is the same.
So these correspond to this 0.5, that the first transition
took me there, and the next transition was
also of the same kind.
That's one possibility.
But there's another scenario.
In order to be at state 1 at time 2 -- this can
also happen this way.
So that's the event that, after 1
transition, I got there.
And the next transition happened to be this one.
So this corresponds to 0.5 times 0.2.
It corresponds to taking the 1-step transition probability
of getting there, times the probability that from state 2
I move to state 1, which in this case, is 0.2.
So basically we take this number, multiplied with 0.2,
and then add those 2 numbers.
And after you add them, you get 0.35.
And similarly here, you're going to get 0.65.
And now to continue with the recursion, we keep doing the
same thing.
We take this number times 0.5 plus this number times 0.2.
Add them up, you get the next entry.
Keep doing that, keep doing that, and eventually you will
notice that the numbers start settling into a
limiting value at 2/7.
And let's verify this.
If this number is 2/7, what is the next number going to be?
The next number is going to be 2/7 -- (not 2.7) --
it's going to be 2/7.
That's the probability that I find myself at that state,
times 0.5--
that's the next transition that takes me to state 1 --
plus 5/7--
that would be the remaining probability that I find myself
in state 2 --
times 1/5.
And so that gives me, again, 2/7.
So this calculation basically illustrates, if this number
has become 2/7, then the next number is
also going to be 2/7.
And of course this number here is going to have to be 5/7.
And this one would have to be again, the same, 5/7.
So the probability that I find myself at state 1, after a
long time has elapsed, settles into some steady state value.
So that's an interesting phenomenon.
We just make this observation.
Now we can also do the calculation about the
probability, starting from state 2.
And here, you do the calculations --
I'm not going to do them.
But after you do them, you find this probability also
settles to 2/7 and this one also settles to 5/7.
So these numbers here are the same as those numbers.
What's the difference between these?
This is the probability that I find myself at state 1 given
that I started at 1.
This is the probability that I find myself at state 1 given
that I started at state 2.
These probabilities are the same, no matter where I
started from.
So this numerical example sort of illustrates the idea that
after the chain has run for a long time, what the state of
the chain is, does not care about the initial
state of the chain.
So if you start here, you know that you're going to stay here
for some time, a few transitions, because this
probability is kind of small.
So the initial state does that's tell you something.
But in the very long run, transitions of this kind are
going to happen.
Transitions of that kind are going to happen.
There's a lot of randomness that comes in, and that
randomness washes out any information that could come
from the initial state of the system.
We describe this situation by saying that the Markov chain
eventually enters a steady state.
Where a steady state, what does it mean it?
Does it mean the state itself becomes steady and
stops at one place?
No, the state of the chain keeps jumping forever.
The state of the chain will keep making transitions, will
keep going back and forth between 1 and 2.
So the state itself, the Xn, does not become
steady in any sense.
What becomes steady are the probabilities
that describe Xn.
That is, after a long time elapses, the probability that
you find yourself at state 1 becomes a constant 2/7, and
the probability that you find yourself in
state 2 becomes a constant.
So jumps will keep happening, but at any given time, if you
ask what's the probability that right now I am at state
1, the answer is going to be 2/7.
Incidentally, do the numbers sort of makes sense?
Why is this number bigger than that number?
Well, this state is a little more sticky than that state.
Once you enter here, it's kind of harder to get out.
So when you enter here, you spend a lot of time here.
This one is easier to get out, because the probability is
0.5, so when you enter there, you tend to get out faster.
So you keep moving from one to the other, but you tend to
spend more time on that state, and this is reflected in this
probability being bigger than that one.
So no matter where you start, there's 5/7 probability of
being here, 2/7 probability being there.
So there were some really nice things that
happened in this example.
The question is, whether things are always as nice for
general Markov chains.
The two nice things that happened where the following--
as we keep doing this calculation, this number
settles to something.
The limit exists.
The other thing that happens is that this number is the
same as that number, which means that the initial state
does not matter.
Is this always the case?
Is it always the case that as n goes to infinity, the
transition probabilities converge to something?
And if they do converge to something, is it the case that
the limit is not affected by the initial state i at which
the chain started?
So mathematically speaking, the question we are raising is
whether Rij(n) converges to something.
And whether that something to which it converges to has only
to do with j.
It's the probability that you find yourself at state j, and
that probability doesn't care about the initial state.
So it's the question of whether the initial state gets
forgotten in the long run.
So the answer is that usually, or for nice chains, both of
these things will be true.
You get the limit which does not depend
on the initial state.
But if your chain has some peculiar or unique structure,
this might not happen.
So let's think first about the issue of convergence.
So convergence, as n goes to infinity at a steady value,
really means the following.
If I tell you a lot of time has passed, then you tell me,
OK, the state of the probabilities are equal to
that value without having to consult your clock.
If you don't have convergence, it means that Rij can keep
going up and down, without settling to something.
So in order for you to tell me the value of Rij, you need to
consult your clock to check if, right now,
it's up or is it down.
So there's some kind of periodic behavior that you
might get when you do not get convergence, and this example
here illustrates it.
So what's happened in this example?
Starting from state 2, next time you go here, or there,
with probability half.
And then next time, no matter where you are, you move back
to state 2.
So this chain has some randomness, but the randomness
is kind of limited type.
You go out, you come in.
So there's a periodic pattern that gets repeated.
It means that if you start at state 2 after an even number
of steps, you are certain to be back at state 2.
So this probability here is 1.
On the other hand, if the number of transitions is odd,
there's no way that you can be at your initial state.
If you start here, at even times you would be here, at
odd times you would be there or there.
So this probability is 0.
As n goes to infinity, these probabilities, the n-step
transition probability does not converge to anything.
It keeps alternating between 0 and 1.
So convergence fails.
This is the main mechanism by which convergence can fail if
your chain has a periodic structure.
And we're going to discuss next time that, if periodicity
absent, then we don't have an issue with convergence.
The second question if we have convergence, whether the
initial state matters or not.
In the previous chain, where you could keep going back and
forth between states 1 and 2 numerically, one finds that
the initial state does not matter.
But you can think of situations where the initial
state does matter.
Look at this chain here.
If you start at state 1, you stay at state 1 forever.
There's no way to escape.
So this means that R11(n) is 1 for all n.
If you start at state 3, you will be moving between stage 3
and 4, but there's no way to go in that direction, so
there's no way that you go to state 1.
And for that reason, R31 is 0 for all n.
OK So this is a case where the initial state matters.
R11 goes to a limit, as n goes to infinity,
because it's constant.
It's always 1 so the limit is 1.
R31 also has a limit.
It's 0 for all times.
So these are the long term probabilities of finding
yourself at state 1.
But those long-term probabilities are affected by
where you started.
If you start here, you're sure that's, in the long term,
you'll be here.
If you start here, you're sure that, in the long term, you
will not be there.
So the initial state does matter here.
And this is a situation where certain states are not
accessible from certain other states, so it has something to
do with the graph structure of our Markov chain.
Finally let's answer this question here, at
least for large n's.
What do you think is going to happen in the long term if you
start at state 2?
If you start at state 2, you may stay at state 2 for a
random amount of time, but eventually this transition
will happen, or that transition would happen.
Because of the symmetry, you are as likely to escape from
state 2 in this direction, or in that direction, so there's
probability 1/2 that, when the transition happens, the
transition happens in that direction.
So for large N, you're certain that the
transition does happen.
And given that the transition has happened, it has
probability 1/2 that it has gone that particular way.
So clearly here, you see that the probability of finding
yourself in a particular state is very much affected by where
you started from.
So what we want to do next is to abstract from these two
examples and describe the general structural properties
that have to do with periodicity, and that have to
do with what happened here with certain states, not being
accessible from the others.
We're going to leave periodicity for next time.
But let's talk about the second kind of
phenomenon that we have.
So here, what we're going to do is to classify the states
in a transition diagram into two types,
recurrent and transient.
So a state is said to be recurrent if the
following is true.
If you start from the state i, you can go to some places, but
no matter where you go, there is a way of coming back.
So what's an example for the recurrent state?
This one.
Starting from here, you can go elsewhere.
You can go to state 7.
You can go to state 6.
That's all where you can go to.
But no matter where you go, there is a path that can take
you back there.
So no matter where you go, there is a chance, and there
is a way for returning where you started.
Those states we call recurrent.
And by this, 8 is recurrent.
All of these are recurrent.
So this is recurrent, this is recurrent.
And this state 5 is also recurrent.
You cannot go anywhere from 5 except to 5 itself.
Wherever you can go, you can go back to where you start.
So this is recurrent.
If it is not the recurrent, we say that it is transient.
So what does transient mean?
You need to take this definition, and reverse it.
Transient means that, starting from i, there is a place to
which you could go, and from which you cannot return.
If it's recurrent, anywhere you go, you
can always come back.
Transient means there are places where you can go from
which you cannot come back.
So state 1 is recurrent - because starting from here,
there's a possibility that you get there, and then
there's no way back.
State 4 is recurrent, starting from 4, there's somewhere you
can go and--
sorry, transient, correct.
State 4 is transient starting from here, there are places
where you could go, and from which you cannot come back.
And in this particular diagram, all these 4 states
are transients.
Now if the state is transient, it means that there is a way
to go somewhere where you're going to get stuck and not to
be able to come.
As long as your state keeps circulating around here,
eventually one of these transitions is going to
happen, and once that happens, then there's no way that you
can come back.
So that transient state will be visited only a finite
number of times.
You will not be able to return to it.
And in the long run, you're certain that you're going to
get out of the transient states, and get to some class
of recurrent states, and get stuck forever.
So, let's see, in this diagram, if I start here,
could I stay in this lump of states forever?
Well as long as I'm staying in this type of states, I would
keep visiting states 1 and 2 Each time that I visit state
2, there's going to be positive
probability that I escape.
So in the long run, if I were to stay here, I would visit
state 2 an infinite number of times, and I would get
infinite chances to escape.
But if you have infinite chances to escape, eventually
you will escape.
So you are certain that with probability 1, starting from
here, you're going to move either to those states, or to
those states.
So starting from transient states, you only stay at the
transient states for random but finite amount of time.
And after that happens, you end up in a class
of recurrent states.
And when I say class, what they mean is that, in this
picture, I divide the recurrent states into 2
classes, or categories.
What's special about them?
These states are recurrent.
But there's no communication between the 2.
If you start here, you're stuck here.
If you start here, you are stuck there.
And this is a case where the initial state does matter,
because if you start here, you get stuck here.
You start here, you get stuck there.
So depending on the initial state, that's going to affect
the long term behavior of your chain.
So the guess you can make at this point is that, for the
initial state to not matter, we should not have multiple
recurrent classes.
We should have only 1.
But we're going to get back to this point next time.