1
00:00:01,069 --> 00:00:03,189

the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  okay so today we're gonna do some really  cool stuff having to do with  non-deterministic parallel programming  this is where the course starts to get  hard because non determinism is really  nasty we'll talk about a little bit it's  really nasty parallel computing as you  know is pretty easy right you know it's  just working span easy stuff right it  makes sense can measure these things can  learn some skills around them and so  forth but non determinism is is is nasty  really nasty so first let's talk about  what we mean by determinism so we say  that a program is deterministic on a  given input if every memory location is  updated with the sequence the same  sequence of values in every execution so  the program always behaves the same and  you may end up if it's a parallel  program having different memory  locations updated in different orders I  may do a and then B versus updating B  and then a but if I look at a single  memory location a say I'm always  updating a with the same sequence of  values okay there are lots of  definitions of determinism this is not  the only one okay there's some where  people say well it's only matters if the  output is always the same okay and there  are others where you say not only does  it have to be the same but every right  to a location has to be in the same  order globally okay that turns out to be  actually pretty hard because if you have  parallel computing you're not going to  get them all updated the same unless you  only have one  you know processor executing  instructions okay and so we'll talk  about this we'll talk a little bit more  about this kind of thing so so why  what's the big advantage of  deterministic programs why should we  care whether it's deterministic or  non-deterministic sure  it's repeatable okay so what  [Music]  [Music]  why is that  because sometimes that's what you want  okay that doesn't say okay so if I mean  there's a lot of things I might  sometimes want how why is that important  to want that yeah yeah makes it easier  to debug that's probably the number one  reason debugging okay if it does the  same thing every time right then if you  have a bug you can run it again you  expect to see the bug again right so  every time you run through hey I get the  same bug okay but if it's  non-deterministic I get a bug and now I  go to look for it and the bug is nowhere  to be found  makes debugging a lot harder there are  other reasons for wanting repeatability  so your answer is actually a broader  correct answer but but the the big  advantage is in the specific application  of repeatability to debugging okay so  here's the golden rule of parallel  programming okay  never write non-deterministic parallel  programs okay they can exhibit anomalous  behaviors and it's hard to debug them  okay so never ever write  non-deterministic programs unfortunately  this is one of these things that is kind  of hard in practice to do so why might  you want to write a non-deterministic  program even though even when famous  masters in the area of performance  engineering okay with with highly  credentialed you know numerous awards  and so forth tell you you shouldn't  write non-deterministic programs why  might you want to do it anyway  yeah yeah you might get better  performance that's a that's one of the  big ones that's one of the big ones and  sometimes you can't  okay the nature of the problem is maybe  that it's that it's non determinacy you  may have asynchronous inputs coming in  and so forth so so this is the Golden  Rule we also have a silver rule silver  rule says never write non-deterministic  parallel programs but if you must always  devise a test strategy to manage the non  determinism okay so this gets into you  better have some way of handling how  you're gonna tell what's going on if you  have a bug okay so what are some of the  typical test strategies that you could  use that would manage the non  determinism okay so imagine you've got a  parallel program and it's it's got races  in it and so forth  you know and its operating  non-deterministically what and that's  okay if it's everything's going right  how would you you find a bug in the  program how are you what are you going  to do  what kinds of ideas do you have yeah  [Music]  yeah you could turn off the  non-determinism okay you put a switch in  there that says well I know the source  of this non-deterministic behavior let  me do that let me give you an example of  that for security reasons these days  when you allocate memory its allocated  to different locations on different runs  of the program its allocated in random  places they want to randomize the  addresses when you call malloc okay that  means that you can end up with different  behaviors okay from one to run and that  can compromise your your performance but  there turns out that there is a compiler  switch and if you run it in debug mode  it'll always deliver the the results of  malloc in in deterministic locations  okay where the locations the of the  things your mouth looking are you know  are repeatable okay so that's that's  good okay because there's support they  said yes we have to randomize okay for  security reasons so that people can't  deterministically exploit buffer  overflow errors for example but you know  I don't want to have to do that every  time okay so you know I I don't want to  randomize you know every time I run I  won't have the option of making it so  that that randomization is turned off  okay so that's a good one  what's another one that that can be done  you're full of good ideas this is like  let's try somebody else or not okay  that's good I like that I like that what  are some other ideas what else can you  do to handle non determinism okay you  got a program and it's yeah yeah yeah  yeah if you have random numbers you use  the same seed in some sense that's a  kind of of same thing if you're turning  off non determinism okay but that's a  great one you know there are other  places for example if you read you know  if you do get time of day you know for  something in your program for something  you could have an option where it'll put  in a particular fixed value there so you  can make sure that it doesn't you know  even a serial program isn't  non-deterministic okay so that's good  but I also consider that to be that's  another great example of turning off non  determinism okay what other things can  you do yeah  [Music]  ah yep you can do record replay for some  things is that what you're saying is  that what you mean or am i okay so  record replay says you run it through  you can run it through with random  numbers and then but it's recording  those things so that when you when you  run it again you can instead of using  the random numbers that you you know new  random numbers it uses the ones that you  used to use okay so that's a record  replay thing is that what you're saying  are you saying something else yeah okay  good  okay so this so that's using using some  tools there's actually a lot of  strategies let me just move on in answer  so another thing you can do is  encapsulate the non-determinism so  that's actually done in the silk runtime  system already the runtime system is  using a random scheduling strategy but  you don't see that it's random in the  execution of your code if you don't if  you have no race conditions in your code  its encapsulated okay so that so that  the you know in the platform so the  platform is going to guarantee you  deterministic results even though  underneath the covers it's doing  non-deterministic things okay you can  also substitute a deterministic  alternative okay sometimes there's  there's a way of computing something  that is non-deterministic but in debug  mode let me not use the  non-deterministic one okay and you can  also use analysis tools which can tell  you things about your program and which  you can can control things so there's a  lot of things so whenever you have a  non-deterministic program you want to  find some way of controlling and often  the non determinism is over in this  corner but your bug is over in this  corner so if you can turn this thing off  in some way or encapsulate it or you  know otherwise control the non  determinism over there now you have a  better chance of catching the stuff over  here  okay that's going to be particularly  important in project four when we get to  it  because that's going to be actually  going to be doing non-deterministic  programming for a game playing program  and one of the one of the things is that  the processors are in this case keeping  the game positions together and so the  if one processor stores something into a  into what's called the transposition  table which is a essentially a big hash  table of positions at scene another one  can see that value and change its  behavior and so one of the things you  want to be able do is turn off  transposition table okay so that so that  you don't take advantage that  performance advantage but now you can  debug the search code or you can debug  the evaluation code and so forth you can  also do things like you know unit  testing so you know whether or not a  particular piece is correct that might  have you know so that so that you can  test this thing separately from the rest  of your system which may have non  determinism and way this is a major  thing so so you never write them but if  you have to you always want to have some  test strategy okay and so for the people  who are not watching this video and who  are you know not in class today they are  going to be sorely hampered by not  knowing this lesson when they go into  the fourth project okay so so what we're  gonna do is now we're going to talk  about how to do non-deterministic  programming okay so this is you know  there's always there's always some part  of your code you know which has a skull  and crossbones  like you know you have this abstraction  it's beautiful and you can design etc  and then somewhere there's this really  ugly thing that nobody should know and  you put the skull and crossbones on that  and only experts go in well anyway  that's the that's the barrier we're  crossing here okay and we're going to  start out by talking about something  that you've probably seen in  your other classes mutual exclusion and  atomicity so I'm going to use the  example of the hash table so here's a  typical hash table it's got collisions  resolved by chaining so you have a bunch  of linked lists you hash to a particular  slot in the table and then you chase  down the linked list to find the value  and so for example if I'm going to  insert X which has a key value of 81 you  know what I do is figure out which slot  I go to by by hashing the key and then  in this case I made it be the last one  so the animations could be easier but if  it were in the middle and so so now what  do I do is I make the the pointer of X  go to the you know first element of that  list and then I make the slot value now  point to X and that effectively in with  a constant number of operations inserts  X into the hash table and in particular  into the linked list in the slot that  it's supposed to be okay this is all  familiar right okay so now what happens  when you have multiple parallel  instructions that are accessing the same  the same locations okay so so here we  have two threads one inserting X and one  inserting Y and X goes it does its thing  it hashes to there and it then sets the  next pointer to be the to add itself  into the list and then there's this  other thing going on in parallel which  effectively says oh I'm going to hash  oh we're going to the same slot it  doesn't know that somebody's already  there and so then it decides it's going  to put itself in as the first element  of the list and then it sets the value  of y that sets the value of the slot to  point to Y and then along comes X  finishing off what it's doing and it  points the value to X and you can see  that we have a race bug here a really  nasty one because we've just destroyed  the integrity of our of our system we  now have in particular y is sort of  floating not in the list when it's  supposed to be in the list right so the  standard solution to this is to make  some of these instructions be atomic ok  so and what that means is the rest of  the system can never view them as being  partially executed okay so they're  either all have been executed or none of  them have been executed at any point in  time as far as the as far as the rest of  the system is concerned okay and the  part of code that is within the atomic  regions called the critical section and  it can typically a critical section of  code is someplace that should not be  being executed by two things at the same  time so the standard solution to  atomicity is to use what's called a  mutex lock or a mutual exclusion lock  okay and it's basically an object with a  lock and unlock member functions and an  attempt by a thread to lock an already  locked mutex causes the thread to block  okay that is wait until the mutex is  unlocked okay so if somebody grabs the  lock somebody else grabs the lock and  it's already taken then they have to  wait and they sit there waiting until  until this guy says yes I'm gonna  release it okay so so what we'll do now  is we'll make each straw each slot be a  struct with a mutex l and a pointer head  to the slot context so it's going to be  the same data structure we had before  but now I'm gonna have not just the  pointer from the slot but I'll also have  I also have a a lock in that position  okay and so the idea of in the code now  is that before I access the lock before  I access the list I'm going to lock that  that list in the table by locking the  slot then I'll do the things that I need  to do and then I'll unlock it and now  anything else can go on because you know  the what's happening is the reason we're  getting into trouble is because we got  some sort of interleaving of operations  and our goal is to make sure that it's  either doing this or doing this and  never this okay to make sure that so  that each thing each piece of code is  restoring the invariant of correctness  after it executes the pointer swaps them  very in this case is that the elements  are in a list okay and and so you want  to restore that with each one okay so so  mutex is this is one way you can use new  mutexes to implement atomicity okay so  now let's just go back this is this is  so who has who has seen you know mutexes  before it's at pretty much everybody  yeah okay good I hope that this is not  brand-new for too many of you if it is  brand-new that's great but what I'm  trying to do is make it so let's go back  a little bit and recall in this class  our discussion of determinacy races so  remember that determinacy race occurs  when you have to logically parallel  instructions that access the same memory  location and at least one of them  performs all right okay so mutex locks  can guarantee that critical sections  behave atomically but the resulting code  is inherently non-deterministic because  you've got a you know we had a race bug  there right we had two things trying to  access the same slot but that may be  what I  want right I want to have a shared hash  table maybe okay for these for these  things so I want something where there  is a race but I just don't want to have  the anomalies that arise in this case  you know the race bug caused things and  I can solve that with atomicity okay you  know if you have no determinacy races it  means that the program is deterministic  on that input and that it always behaves  the same okay and remember also that if  a determinacy race exists in an  ostensibly deterministic program okay  then it guarantees to find a race now if  you put in mutexes you still have a  non-deterministic program you still have  a race because you have two things that  are logically parallel that are both  accessing the lock that's a rate that's  a determinacy race okay if you have two  things that are in parallel they're both  accessing the lock  that's a determinacy race it may be a  safe correct one but it is a determinacy  race okay and so any codes that use  locks are non-deterministic by intention  and they're going to invalidate the  silks and guarantee of finding those  race bugs okay so you will end up with  races in your code if you're not careful  okay and so this is one reason it's  important to have some way of turning  off non determinism to detect stuff  because what you don't want is that  whole rash of false positives saying oh  you raced on gathering this lock nor do  you want to ignore that and then  discover that a race has popped up  somewhere else okay now some people feel  that so this is basically a talking  about having a data race okay and a data  race is similar to the definition of  determinacy race okay but it says that  you have to logically parallel  instructions and they don't hold locks  in common  and then it's the same definition if  they access the same memory location one  of them performs or right then you have  a then you have a data race bug okay but  if they if they have the locks in common  okay if they both have acquired at least  one lock that's the same then you don't  have a data race okay because that means  that you've now successfully protected  the atomicity but it is still  non-deterministic and there is a  determinacy race just no data race and  that's the big distinction between data  races and determinacy races and on quiz  2 you better know the difference between  data races and determinacy races okay  because they are different okay so a  program may have no determinant may have  no data races that doesn't mean that it  doesn't have a determinacy race in fact  if it's got any locks it probably has a  determinacy race okay okay so so one of  the things is if I have no data races  does that mean I have no bugs suppose I  have no data races in my code does that  mean I have no bugs this is quite an  obvious answer just bye-bye quiz  moonship right so what might happen  think about a little bit what might  happen how could I have no data races  and yet there's still be a bug even  though and assuming it's a correct piece  of code otherwise okay in other words  when it runs serially or whatever it's  correct how could I end up having code  no data races but still have a bug  [Music]  yeah but that doesn't mean it's bad  right yep  might still be a different yep  okay uh yeah let me give you an example  okay  which is more to the point here is a way  of making sure that I have no data race  which is I lock before I follow the  table slot value and then I unlock then  I lock again and then I set the value  okay so I haven't prevented the  atomicity right now I've got an atom  isset II violation but I have no data  races okay because I never have two  things that are going to access things  at the same time is protected by the  lock okay but it doesn't solve my  atomicity so there's a you know there's  you know you can definitely have no data  races but that doesn't mean you have no  bugs okay but usually what it happens is  if you have no data races then usually  the programmer actually got this code  right okay it's one of these things  we're demonstrating no data races is in  fact you know a very positive thing in  your code it doesn't mean the program I  did right but most of the time the  reason they're putting in the in the  locks is to provide atomicity for  something and they usually get it right  they don't always get it right in fact  Java for example had a very famous bug  early on in the way that it specified  locking such that you know the you could  look at the length of a string and then  modify it and then you would end up with  a race bug because somebody else could  swoop in in between so they thought they  were being providing atomicity and they  didn't okay  so there's there's another set of issues  here having to do with benign races now  there's some people who argue that no  races are  are no determinacy races are benign and  they make academic statements that I  find quite compelling actually what  they're what they say about about races  and whether races are benign but  nevertheless the literature also  continues to use the term benign race  for this kind of example so suppose we  want to identify what are is the set of  digits that perform that occurred in  this in in some array so here's an array  with a bunch of values in it okay each  one being a digit from zero to nine so I  could write a little piece of code that  runs through my digits array of length  10 and sets the number of digits I've  seen so far of each value to be 0 okay  and now I go through and I'm gonna do  this in parallel and I'm going to set  every time I see a value a of I suppose  a of I is 3 I set the location of a 3 to  be 1 and otherwise and now otherwise  it's zero because that's what I had it  before so here's the kind of thing I  have so for example I can have both of  those sixes are in parallel going to  access the location 6 to set it to 1 but  they're both setting it to 1 it doesn't  really matter what order they do it in  you're gonna get the same value there 1  ok and so there's a race maybe we don't  too much care about that race okay  because they're both setting the same  value we're not going to get an  incorrect value well not exactly  we might get on some architecture on the  Intel architectures you won't get an  incorrect value on x86 but there are  codes where the elements of the array  values are not set atomically so for  example on the mips architecture  in order to set a bite to be a  particular value you have to you have to  fetch the word mask out set the word and  then store it back in set the the bite  and then store it back into the word and  so if there are two guys who are  basically operating on that same word  location they will have a race even  though in the code it looks like they're  just setting bites does that make sense  okay so nasty nasty bugs okay that's  that's why you should never do  non-deterministic programming okay  unless you have to  okay so silks and allows you to turn off  race detection for intentional races  okay so if you really meant there to be  a race as in this case you can turn it  off  this is dangerous but practical turns  out usually you're not turning it off  for because here's what can happen you  can turn it off and yet then there's  something else which is using that same  stuff and now you're running silks and  without you know having turned it off  for exactly what your race might be  they're better solutions so in Intel's  silk screen there's the notion of fake  locks we just have not yet implemented  it in in the open silk compiler so we  will and in silks and we'll eventually  get to doing that and then people who  take this class in the future will have  an easier time with that okay because  we'll be able to check for that as well  okay so any questions about these  notions so you can see the notions of  races can get quite hairy and you know  make it quite difficult to do your  debugging and in fact even can confound  your tools that are supposed to be  helping you to get correct code okay  all a name of performance okay but we  like performance okay any questions  yeah  yeah so how can some architectures cause  some error so here's the thing is that  if I have a let's say a byte array it  may be that this is stored int as as a  set of let's say 4 byte words okay and  so although you may write that a of 0  gets 1 what it does is it says let me  fetch these 4 values because there is no  byte set instruction on some  architectures it can only set in this  case 32-bit words so it fetches the  values it then you know into a register  it then sets the value in the register  ok by masking so it doesn't set the  other things here and then it stores it  back right so that it has a 1 here but  what if somebody at the same time is  storing into this location they will  fetch it into their own register set  their byte mascot etc and now my right  back is going to we're gonna have a lost  update in the in the right backs does  that make sense ok good  very good question ya know I went  through that orally a little bit quicker  than maybe I should have ok great  so let's talk a little bit about  implementation I always like to take  things down one level below what you  necessarily need to know in order to do  things but it's helpful to sort of see  how these things are implemented and  because then that gives you a better  sense at a higher level what the what  your capabilities are and how things are  actually working underneath so let's  talk about how in the mutex is so here  first of all to understand there are  lots of different mutexes if you look at  an operating system they may have a half  a dozen or more different  you Texas different locks that can  provide mutual exclusion or parameters  that can be set for what kind of mute  exes so the first basic difference in  most things is whether the mutex is  yielding or spinning so a yielding mutex  returns control to the operating system  when it blocks when a program tries to  get active when it tries to get access  when a thread tries to get access to a  given low given lock if it is blocked it  doesn't just sit there and keep and  spinning where you're basically spinning  means I just sit there checking it and  checking it and checking it and checking  in ok instead what it does is it says oh  I'm doing useless work here let me go  and return control of the operating  system maybe there's another thread that  can run at the same time and therefore  I'll give by switching myself out by  yielding my my scheduling quantum I will  get better efficiency overall because  somebody some other thread that is  capable of running can run at that point  ok so is that clear the distinction  between spinning and and yielding ok  another one is whether the mutex is  reentrant or non re-entrant a reentrant  mutex allows a thread that is already  holding a lock to acquire it again ok a  non reentrant one deadlocks if the  thread attempts to require a mutex that  already holds ok so I grab a lock and  now I go to a piece of code and it says  grab that lock so very simple I can  check to see whether I have if I want to  be reentrant I can check do I have that  lock already and if I do then I don't  actually have to acquire it I just keep  going  but that's extra overhead it's faster  for me to have a non reentrant lock ok  where I just simply grab the lock and if  somebody's got it  including me then it's a dead lock but  now if there's the possibility that I  could react wire a lock then that might  not be safe  okay you have to worry about the  programmer has to worry about that now  is that clear that one okay  and then final basic property of mutexes  is whether they're fair or unfair okay  so here's the thing with it's easiest to  think about it in the context of  spinning  I have several threads that basically  came to the same lock and we decide  they're gonna spin they're just going to  sit there continually checking waiting  for that lock to be free okay so when  finally the guy who has it unlocks it  you know maybe I've got half a dozen  threads sitting there one of them wins  okay and which one wins well if they're  spinning it could be any one of them  then it has one and so the issue that  can go on is you could have what's  called a starvation problem okay where  some guy is sitting there for a really  long time waiting while everybody else  is continually grabbing locks out from  under his or her nose okay so with a  fair mutex basically what you do is you  go for the one that's been waiting the  longest essentially and so therefore you  never have to wait more than for however  many things were there when you got  there before you're able to go question  why  [Music]  it can be better because you may freeze  out or service okay if there's something  that's you know you may never get to do  the thing you want to do okay because  there's something else always  interfering with the ability for that  part of the program to make progress  this tends to be more of an issue in in  concurrent programming where we have  different programs that are trying to  accomplish different tasks and you want  to accomplish both tasks okay  does not come across in parallel  programming mostly we deal with unfair  often unfair spinning locks because  they're the cheapest and we just trust  that a we're not going to do any have  any critical regions we write our code  so we don't have critical regions that  are really long so nobody ever has to  wait a very long time but indeed dealing  with a contention issue as we talked  about last week you know can can make a  difference okay good so here's an  implementation of a simple spinning  mutex in assembly language okay  so the first thing it does is it checks  to see if the the mutex is free if it's  value is 0 so it compares the value of  the mutex to 0 and if it is 0 it says oh  the it's free let me go get it it then  to get the mutex what it does is it  moves one into the it basically moves  one into a register and then it  exchanges the mutex with that register  EAX ok and then it compares to see  whether or not it actually got the mutex  and if it didn't then it goes back up to  the top and starts again ok and then the  other branch is at the at the top there  it does this pause and this apparently  is due to a bug in x86 that they end up  having to put this pause instruction in  there and then otherwise you jump to the  where the spin mutex is and and go go  again  okay and then once you've done the  critical section when you're done you  free it by just setting it to zero so  the question here is so the exchange  instruction is a is a is an atomic  exchange so it takes the register and  the memory value and it swaps them and  you can't have anything come in so one  of the things that might have you  confused a little bit here is wait a  second I check to see if the mutex is  free and then I tried to get it and to  test if I was successful why why can't I  just start out by essentially going to  get mutex right I mean why do I need why  don't need any of the code between spin  mutex and get mutex  right so if I just started with getting  new text I would move one in I would  exchange check to see if I could get it  if I have it fine then I execute the  internet if not okay I go back and and  try again so why you know because if  somebody has it by the way the value  that I'm gonna get is going to be one  and that's what I swapped in so I  haven't changed anything I go back and I  check again so so why do I need that  first part yeah yeah maybe it's faster  okay  so indeed it's because it's faster even  though you're executing extra code it's  faster tell me why it's faster and this  tape will take you you have to think a  little bit about the cache protocols and  the invalidation issue so why is it  gonna be faster yeah okay good say more  [Music]  yeah so so it turns out the exchange  operation is like a right and so in  order to do a right what do I need to do  for the for the cash line that it's on  to bring it in but what how does it have  to be brought in remember the cash lines  had let's let's a Matt you have to  invalidate it on the other ones and you  have to hold it in what state  remember the cash lines have you know if  we take a look at just a simplified  protocol right where the MSI protocol  [Music]  yeah you have to have it in the EMA MSI  or an ESI you have to bring it in in  modified or at least exclusive state  right so exclusive is for the Emmy ESI  protocol we we mentioned that but we  didn't really do it mostly we just went  but I have to bring it in and modified  where I guarantee there are no other  copies so if I've got two guys that are  polling on this location  they're both continually invalidating  each other and you create a whole bunch  of traffic on the memory network that's  going to slow everything down okay  whereas if I do the first one what state  do I get it in then you get it in shared  State what does the other guy get it in  shared state and now I keep going just  having it in spinning in my own local  cache not generating any local traffic  until the until somebody releases the  Falak in which case it invalidate Sall  those and now you can actually get it a  little bit of a storm after the fact  there in fact locks where you don't even  get a storm after the after the fact but  called mcs locks but but this kind of  lock is for most practical purposes just  fine okay so so to everybody follow that  description of what's going on there so  that first code for correctness purposes  not important for performance it is  important okay isn't it great that you  guys can read assembly language right  okay now suppose that this is a spinning  mutex suppose that I want to do a  yielding mutex how does this code have  to change  okay so this is a spinning one just  keeps checking instead I want to return  control of the operating system okay so  how does this code change if I do that  yeah  [Music]  like that yeah exactly  okay so instead of doing that that you  know  pause instruction which the  documentation on this is not very clear  I'm love to have the inside scoop on why  they really had to do the pause there  but in any case you take that no op that  they want to have in there and your  place it with it just a call to the  yield which allows the operating systems  to schedule something else and then when  it's your turn again it resumes from  that point okay so that's the yield so  so that's the difference in the  implementation essentially between a  spinning mutex and a yielding mutex okay  now there's another kind of mutex that  is kind of cool which is called a  competitive mutex so think about it this  way  I've competing goals one is to I want to  get the mutex as quickly as possible  after it's released right I don't want  if if it's unlocked I don't want to sit  there for a really long time and before  I actually acquire okay and - yeah but I  don't want to sit there spinning for a  really long time and then you know  because as long I'm doing that I'm  taking up cycles and not accomplishing  anything let me turn it over to some  other thread that can can use the cycles  more effectively so there at those two  goals how can I get the best of both  worlds here something that's close to  the best of both worlds that's not  absolutely the best of all flows but  it's close to the best of both worlds  what strategy could I do so I want to  claim it very soon so so the point is  that the spinning mutex achieves goal  one and the yielding mutex achieves goal  two so how can i what can I do to get  both goals yeah  [Music]  thank you  so you're saying use message-passing to  inform the waiting threads I'm thinking  something a lot simpler  okay in this context so because the  message passing you're gonna have to go  through to do message passing properly  you actually need to use mutex as a  setter to implement it so you want to be  a little bit careful you know about  about that but but interesting idea yeah  using an interrupt how would you do that  yeah so typically implement interrupts  you also need to have some mutual  exclusions to do it properly but I mean  hardware will support that  that's pretty heavy-handed as well this  is actually a very simple solution who's  everybody I'm seeing familiar hands I  want to see so unfamiliar hands who's  got an unfamiliar hand okay I see you  raised your left hand that time instead  of your right hand okay yeah  it hard to measure that right how would  you write code to measure that yeah  mm-hmm yeah good okay good  [Music]  why doesn't it have a because if I yield  okay so what's the how often does if I  contact switch how often is it going to  be that I how long am I going to have to  wait typically before I am scheduled  again okay when when a when a code  yields to the operating system how often  does the operating system normally do  context switching do you know what's  that what's the rate at which it  contacts switches for the different  multiplexing of threads that it does on  to on to the available processors what's  the rate at which it it shifts oh this  is what okay that's gonna be on the quiz  okay this is a new numeracy thing yeah  how do you know how frequently  not quite but but your your you're not  off by more than an order of magnitude  okay so what are the typical rates that  you that the system does contact  switching okay so from human times it's  the blink of an eye right so it's  actually around 10 milliseconds so it  would does 100 times a second some of  them do some do 60 times a second okay  that's how often it switches okay now  let's say it's a hundred times the  second 10 milliseconds so you're pretty  close okay 10 milliseconds okay how many  orders of magnitude is that from the  execution of a simple instruction okay  so we're going at more than a gigahertz  right and so a gigahertz is 10 to the  ninth and we're talking ten to the minus  nine and we're talking ten to the minus  two so that's you know 10 million  instruction opportunities that we miss  if we switch out and of course we  probably only switch out for half hour  you know where are you along the the  thing so you're only switching out maybe  for half assuming nothing else is going  on there but that's that means you're  not grabbing the lock quickly after it's  released because you've got ten million  instructions that are going to execute  before you're gonna have a chance to  come back in and grab it okay  so that's why a yielding one is does not  grab it quickly  okay we're spinning is like we're  executing this stuff at the rate of  gigahertz right checking again checking  and checking again okay so why so what's  what's the strategy here what can I do  yeah  hey what a good idea okay okay spin for  a while and then yield okay so the idea  being hey if if the if the lock is  really soon then I will be able to grab  it immediately because I'm spinning and  if it takes a long time for the lock to  yield well I will yield eventually so  yeah but okay how long does spin how  long shall I spin sure  [Music]  yeah basically as long as a context  switch takes okay as long as it takes to  go out and come back okay and if you do  that then you never wait more than twice  the optimal time okay this is  competitive analysis which the  theoreticians have gone off there's  brilliant work in competitive analysis  so the idea here is that if the mutex is  released while you're spinning then the  strategy is optimal right because you  just sat there spinning and as soon as  was there you got it on the next cycle  if the mutex is released after the yield  you've already spun for equal to that so  you'll come back and get it at within it  most a factor of two okay this is by the  way that the way this shows up in the  Theory literature if you're introduced  it's called the ski rental problem okay  and here's the idea you're gonna go you  know your friends are persuading you to  go try skiing okay right snow skiing  right right okay and so you say gee  should I buy the equipment or should I  rent after all you may discover it that  you rent and then I mean you buy it and  then you break your leg and never want  to go back okay well then if you've  bought it's been very expensive and if  you've rented well then then you're  probably better off on the other hand if  it turns out you like it  you're now accumulating the costs going  forward okay and so the question is well  what's your strategy and the idea is  well let's look at what renting costs  and what buying costs let me rent until  it's equal to the cost of buying and  then buy and then I'm within a factor of  two of having spent the optimal amount  of of money for because then if I break  my leg after that well at least I you  know you know I got you know I I I  didn't spend more than a factor of two  and if I get it before it I've spent  optimally  okay yeah  yeah yeah 10 milliseconds yeah so spin  for 10 milliseconds okay and then switch  so now the point is that that when you  come back in okay the other job is gonna  run for 10 milliseconds or whatever okay  so if you get switched out then if the  lock is released you're gonna be done in  20 milliseconds right and so you'll be  within a factor of 2 and if it happened  if they lock out release before then  you're right there to grab it okay now  it turns out that there's a really  clever randomized algorithm I love this  rat algorithm okay from 1994 that  achieves a competitive ratio of e over e  minus 1 using a randomized strategy and  I'll encourage you to you know those of  you who have a theoretical bent to go  take a look at that it's very clever  ok so basically have some probability of  at every step of whether you at that  point decide to yield you know or  continue spinning ok and by using a  randomized strategy you can actually get  this to Yi over u minus 1 ok questions  about this so this is sort of some of  the basics I'm glad we went over some of  that because everybody should know these  basic numbers about what things cost  because otherwise you don't know where  to spend it so context switching time is  on the order of 10 milliseconds how long  is the disk access compared to you know  yeah what's the disk access  150 cycles hmm that's an that that would  be accessing DRAM okay  accessing DRAM if you if it wasn't in  cache might be 150 cycles okay  so two orders of magnitude or so okay so  what what about a disk access how long  is that take yeah  yes several milliseconds so ten  milliseconds that are five milliseconds  depend upon how fast your disk is but  once again it's on the order of  milliseconds so it's helpful to know  some of these numbers because otherwise  you know where are you spending your  time especially we're sort of doing  performance engineering you know in the  small basically looking within the  product you know within a multi-core  processor most performance engineering  is on all the stuff on the outside  dealing with networking and file systems  and stuff where where things are really  costly and where you actually have a lot  of time you can write fast piece of code  that can figure out how you should best  deal with these slow parts of your  system okay so those are all sort of  good numbers to know you'll probably see  some of them on quiz 2 okay okay  deadlock I mentioned deadlock earlier  let's talk about what deadlock is and  understand this once again I expect some  of you have seen this but I still want  to go through it because it's hugely  important material and this is the issue  that holding more than one lock at a  time can be dangerous okay so imagine  that thread one says I'm gonna lock a  lock B execute the crudo section unlock  B unlock a where a and B or mutexes and  thread two does something very similar  it locks B and locks a then it does the  critical section then it unlocks a and  then unlocks B okay so what can happen  here so we you know thread one locks a  thread two locks B thread  one can't go and lock be because thread  two has it thread two can't go and lock  a because thread one has it so they sit  there blocked I don't care if they're  spinning or yielding they're not going  anywhere okay so this is the ultimate  loss of performance right it's like it's  incorrect okay it's like you're stuck  you've deadlocked okay now there's  there's three basic conditions for  deadlock everybody understands this  right there's there any anybody who has  a question because just okay it's  there's three conditions you need for  deadlock the first one is mutual  exclusion that you're going to have  exclusive control over the resources the  second is non preemption you don't  release your resources you hold until  you finish using them okay and and three  is circular waiting you have a cycle of  threads in which each thread is blocked  waiting for resources held by the next  one in this case the resource is the  lock okay and so if you remove any one  of these constraints okay you can come  up with solutions that won't deadlock so  for example it could be that when I try  to acquire a lock if somebody else has  him I take it away  okay you know that could be one thing  now there may get into other issues  which is like well but what if he's  actually doing real work okay  or whatever so all of these things have  things or you know I don't insist that  it be mutual exclusion except that's the  kind of problem that we're trying to  solve so these are generally the three  things that are necessary in order to in  order to have a deadlock situation now  in any discussion of deadlock you have  to talk about dining philosophers when I  was an undergraduate and I graduated in  1975 from Yale humanities school  I was taught the dining philosophers  right because after all philosophy is  what you find at humanity schools I mean  we have philosophy department - don't  get me wrong but at Yale the humanities  is huge and so philosophy I guess they  thought this was would appeal to the  people who were not real techies in the  background I sort of like I was a techie  in the midst of all these you know  non-technical people dining philosophers  is a story of deadlock told by Tony  Hoare based on an examination question  by Edsger Dijkstra okay  and it's been embellished too over the  years by many many many retailers okay  and I like the Chinese version of this  okay there's versions where they use  Forks but I'm going to this is going to  be their dining I'm going to say that  they are eating noodles with chopsticks  and they're in philosophers seated  around the table and II and between  every plate of noodles there's a  chopstick and so in order to eat the  noodles they need two chopsticks okay  which to me sounds very natural okay and  so here's the code for philosopher I so  he's a philosopher so he starts by  thinking for a while okay and then he  gets hungry he or she gets hungry so the  philosopher grabs the chopstick on the  law right on the Left sorry and then it  grabs the one on the on the right but it  had you know so which is I plus one but  it has to do that mod in because if it's  the last one you got to go around and  grab the first one then eats and then it  unlocks the two chopsticks and now they  can be used by the other dining  philosophers because they don't think  much about sanitation and  and so forth right so they're because  they're they're too busy thinking right  but but what happens what's wrong with  this solution what happens what can  happen for this it's very simple I need  two chopsticks I grab one I grab the  other I eat one day what happens yeah  yeah they're all it and they grab one to  the left and now they go to the right  it's not there and they starve okay you  know one day they got all the things so  we have the starving philosophers  problem so so be motivated by this  problem yeah question yeah so if you are  willing to preempt then that would be  preemption as I say it's got to be non  pre-emptive in order for deadlock to  occur in this case yeah but you also  have to worry in those cases could be oh  well if I couldn't get both let me put  them both down but then you can have a  thing that's called livelock okay so  they all pick up their left they see the  right ones busy so they put it down so  somebody else can have it they look  around oh okay let me pick out one no no  okay okay and so they still starve even  though they've done that so in that kind  of situation you could put in a time  delay you could say let me let everybody  pick a random number have a randomized  scheme so that we're not you know so  there are other solutions if you don't  insist on non preemption I'm going to  give you one where we have non  preemption but we still avoid deadlock  and it's to go for that cyclic problem  so here's the idea  suppose that we can linearly order the  mutexes okay so I pick some order of the  mutexes so that whenever a thread holds  a mutex else of I and attempts to act a  lock another mutex l sub J we have that  in this linear order L sub I comes  before L sub J then you can't have a  deadlock okay so in this case for the  dining philosophers it would for example  number the number of the chopsticks you  know from 1 to N and then or 0 to n  minus 1 whatever and then grab the  smaller one and then grab the larger one  then then it says then you would never  have a deadlock  okay and so here's the proof you know I  like proofs okay  proofs are really important so I'm going  to show you that if you do that you  couldn't have a cycle of waiting so  suppose you had a cycle of waiting we  were in a situation where everybody is  holding chopsticks and you know one that  one of them is waiting for another one  which is waiting for now all the way  around to the first one that's what we  need for deadlock to occur so let me  just look at what's the largest mutex on  the cycle let's call that L max okay and  suppose that it's waiting on a mutex L  held by the next thread in the cycle  well then we have a something that's  bigger than the maximum one okay and so  that contradicts the fact that I grabbed  them whenever I grabbed them I do it in  order okay so very simple very simple  proof that you can't have deadlock if  you grab them according to a linear  order okay and so for this particular  problem what I do is instead of grabbing  the one on the left and one on the right  as I say you grab the smaller of the two  and then grab the larger of the two and  then you're guaranteed to have no  deadlock does that make sense now if  you're going to use locks in silk you  have to realize that in the operating  sin the runtime system of silk they're  doing they're using locks you can't see  them they're encapsulated as we talked  about the non determinism in silk is  capsulated is still going on underneath  the color covers and if you start  introducing your own non determinism  through the use of locks you can run  into trouble if you're not careful and  let me give you an example okay this is  a situation you can deadlock your  program in silk okay with just one lock  okay  so here's an example of a code that does  that so main spawns off foo and foo  basically locks the lock L and then  unlocks it and meanwhile  after it spawns our food the  continuation goes and it locks L itself  and then does a sink and then it unlocks  it so what happens here we sort of have  a situation like this where where the  locking I've done with an open bracket  and an unlock a release I'm doing with a  closed bracket so I'm spawning off foo  which is the lower part there and  locking and unlocking and up above I'm  locking and unlocking so what can happen  here I can go and I basically spawn off  the child but then I lock and now the  child goes and it says oh I can't you  know foo is going to wait here because  it can't grab the lock as it's owned by  the by main ok and now we get to the  point where main is has to wait for the  sink okay and the child is never going  to complete because I hold the resource  that the child needs to complete okay so  don't hold mutexes across silk sinks  that's the lesson there there are  actually places you can but if you don't  hold them across that then you won't run  into this particular problem a good  strategy is only holding mutexes within  strands okay so there's no parallelism  so you you have it bounded and also  that's a good idea generally because you  want to hold mutex as a short amount of  time as you possibly can  so for example if you have a big  calculation and then you want to sign  something atomically don't put the big  calculation inside the critical region  move the calculation outside the  critical region do the calculation you  need to do and then acquire the locks  just to do the interaction you need to  set a value okay so that's and then  you'll have a lot faster code okay  because you're not holding up other  threads for a long time and always try  to avoid non-deterministic programming  but that's not always possible okay I  want to so any questions about that then  I want to go on a really interesting  topic because it's really  you know recent research level topic and  that's to talk about transactional  memory who's heard this term before  anybody okay so the idea is to have  database transactions that you have  things like database transaction or the  atomicity is happening implicitly you  don't specify locks you just say this is  a critical region okay don't interrupt  me while I do this critical region the  system works everything out okay here's  a good example of where it might be  useful suppose we want to do a  concurrent graph computation okay and so  you take people involved in parallel and  distributed computing and at MIT and you  say okay I want to do Gaussian  elimination on this graph now you guys  are sure most of you know Gaussian  elimination from the matrix context do  you know what it means in a graph  context so if you have a sparse matrix  you actually have a graph and Gaussian  elimination is a way of manipulating the  graph and you get exactly the same  behavior as you get in the dense one so  I'll show you what it what it is you  basically pick somebody to eliminate  okay  okay and now what you do is look at all  this all the this vertexes neighbors  okay  those guys and what you do is you  eliminate that vertex bye-bye okay  and you interconnect all the neighbors  with all the edges that don't already  exist and that's Gaussian elimination if  you think of it in terms of matrix  fashion the question is if you have a  sparse matrix where are you going to get  fill-in what are the places that you  need to update when you do a pivot in  Gaussian elimination in a matrix okay so  that's the basic notion of graph of  doing Gaussian elimination but now we  want to deal with the concurrency okay  and the problem occurs is if I want to  eliminate two nodes at the same time  okay because now they're adjacent each  other and if I just do what i Express  there's going to be all kinds of  atomicity violations etc by the way the  reason I'm picking these two folks is  because they're going to a better place  okay so so how do you deal with this and  so in transactional memory what I want  to be able to do is just simply say okay  here's the thing that I need to be  atomic and so if I look at this code  it's basically saying you know who are  my neighbors and then let me identify  all of the edges that need to be removed  the ones that I just showed you they  know that we're moved now let me get rid  of the element you know V and now for  all of the neighbors of U let us let us  add in the edge between the neighbor and  you know between the pairs of neighbors  okay so that's basically what it's doing  okay and I'd like to just say that's  atomic okay and so the idea is that if I  express that as a transaction then the  ID  is that on the transaction commit all  the memory updates in the critical  region appear to take it happen at once  however in transaction member the idea  is rather than forcing it go forward I  can have the transactions abort okay so  if I get a conflict I'll abort one and  restart it and then the restarted  transaction may take a different code  path because after all I may have  restructured the graph underneath and so  it's may do something different the  second time through the first it may  also abort again and so forth so when  you study transaction transactional  memory let me just do a couple of -  there's one is a conflict that's when  you have two transactions that are you  know they can't both complete one of  them has to be aborted okay and aborting  by the way is once again violating the  non you know you know the non  pre-emptive nature here we're going to  preempt one of them by by keeping all  the states so I can roll a state back  and restart it from scratch so  contention resolution is deciding which  of the two conflicting transit actions  to wait or to abort and restart and  under what conditions you do that so the  resolution manager has to figure out  what happens in the case of contention  and then forward progress is avoiding  deadlock of course but also livelock and  starvation you want to make sure that  you're going to make because what you  don't want to have happen for example is  that two transactions keep aborting each  other and you never make forward  progress and throughput well you'd like  to run as many transactions as  concurrently as possible so I'm going to  show you an algorithm for doing this  it's a really simple algorithm okay it  happens to be one that I discovered just  a couple of years ago okay and I was  surprised that it did not appear in the  literature and so I wrote a very short  paper on it because it's because what  happens for a lot of people is they you  know in if they discover there's a lot  of aborting they say oh well let's grab  a global lock  and then if everybody grabs a global  lock you can do this sort of thing you  can't deadlock with a single lock if if  you're not also doing things like you  know silk sync or whatever but but in  any case if you have just a single lock  everybody falls back to the single lock  and then you have no concurrency in your  program no performance until everybody  gets through the difficult time so this  is an algorithm that doesn't require a  global lock okay so it assumes that  transactional memory system will log the  reads and writes that's typically true  of any transaction maybe you log what  reads and writes you're doing so that  you can either abort and rollback or you  can when you abort you or else you  sandbox things and then atomically  commit them okay and so we have all the  mechanisms for boarding and rolling back  these are all very interesting in their  own right and restarting and this is  going to basically use a lock based  approach that uses two ideas one is the  notion of what's called a final finite  ownership array and then another is a  thing called release sort require and  let me explain those two things and I'll  show you really quickly how this  beautiful algorithm works okay so you  have an array of anti starvation mutual  exclusion locks so these are ones that  are going to be fair okay so that so  that you're always going for the oldest  one and you can do in a acquire we're  also going to add in a try acquire tell  me whether if I tried to acquire I would  get it that is megive it to me if I  don't get it  don't wait just tell me that I didn't  get it okay and then release and there's  an owner function that map's  all of the function H that map's my  universe of memory locations to the  indexes in this finite ownership array  this lock array okay so the lock has  length array has length n has n slots in  it to lock a location X in the set of  all possible  you actually require lock of H of X so  you can think of H as a hash function  but doesn't have to be a fair hash  function or whatever any function will  do okay and then there yes there'll be  some advantages to picking some  functions or another one okay so in  rather than actually locking the  location or locking the object I lock a  location that I that essentially I hash  to from that object so if two guys are  trying to grab the same location they  will both grab the same lock because  they've got the same hash function but I  may have inadvertently where if I were  locking the objects themselves okay I  wouldn't have them both trying to  acquire the same lock that might happen  in this algorithm okay so here's the  idea it's called the first idea is  called a release sort and reacquires so  that's the ownership array part that I  just explained now here's the release  sort a quat reacquire before you access  a memory location X simply try to grab  lock of X greedily and if you if you  have a conflict okay so if you don't  have a conflict you get it you just  simply try to get it if you can that's  great if not then what I'm going to do  is roll back the transaction but don't  release the locks I hold and then  release all the locks with index is  greater than H of X and then I'm gonna  choir the lock that I want and now at  that point I've released all the bigger  locks so I'm acquiring the next lock and  then I acquire the released locks in  sorted order so I go through all the  locks I release and I really min sorted  order okay and then I start my  transaction over again I try again so  what happens each time through this  process  I'm always whenever trying to acquire a  lock I'm only holding locks that are  smaller but each time that I restart I  have one more lock that I didn't used to  have before I restart my transaction  which I've acquired in the order in the  linear order from you know from the  in that ownership array from zero to n  minus one okay and so here's the  algorithm I'll let you guys look at it  in more detail because I see our our  time is up and it's actually fun to take  a look at and we'll put the paper online  there's one other topic that I wanted to  go through here which is which you  should know about is this locking  anomaly called convoying and this was  actually a bug that we had performance  bug that we had in our original MIT silk  so it's kind of a neat one to see and  how we resolved it and that's it

2
00:00:03,189 --> 00:00:05,769
the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  okay so today we're gonna do some really  cool stuff having to do with  non-deterministic parallel programming  this is where the course starts to get  hard because non determinism is really  nasty we'll talk about a little bit it's  really nasty parallel computing as you  know is pretty easy right you know it's  just working span easy stuff right it  makes sense can measure these things can  learn some skills around them and so  forth but non determinism is is is nasty  really nasty so first let's talk about  what we mean by determinism so we say  that a program is deterministic on a  given input if every memory location is  updated with the sequence the same  sequence of values in every execution so  the program always behaves the same and  you may end up if it's a parallel  program having different memory  locations updated in different orders I  may do a and then B versus updating B  and then a but if I look at a single  memory location a say I'm always  updating a with the same sequence of  values okay there are lots of  definitions of determinism this is not  the only one okay there's some where  people say well it's only matters if the  output is always the same okay and there  are others where you say not only does  it have to be the same but every right  to a location has to be in the same  order globally okay that turns out to be  actually pretty hard because if you have  parallel computing you're not going to  get them all updated the same unless you  only have one  you know processor executing  instructions okay and so we'll talk  about this we'll talk a little bit more  about this kind of thing so so why  what's the big advantage of  deterministic programs why should we  care whether it's deterministic or  non-deterministic sure  it's repeatable okay so what  [Music]  [Music]  why is that  because sometimes that's what you want  okay that doesn't say okay so if I mean  there's a lot of things I might  sometimes want how why is that important  to want that yeah yeah makes it easier  to debug that's probably the number one  reason debugging okay if it does the  same thing every time right then if you  have a bug you can run it again you  expect to see the bug again right so  every time you run through hey I get the  same bug okay but if it's  non-deterministic I get a bug and now I  go to look for it and the bug is nowhere  to be found  makes debugging a lot harder there are  other reasons for wanting repeatability  so your answer is actually a broader  correct answer but but the the big  advantage is in the specific application  of repeatability to debugging okay so  here's the golden rule of parallel  programming okay  never write non-deterministic parallel  programs okay they can exhibit anomalous  behaviors and it's hard to debug them  okay so never ever write  non-deterministic programs unfortunately  this is one of these things that is kind  of hard in practice to do so why might  you want to write a non-deterministic  program even though even when famous  masters in the area of performance  engineering okay with with highly  credentialed you know numerous awards  and so forth tell you you shouldn't  write non-deterministic programs why  might you want to do it anyway  yeah yeah you might get better  performance that's a that's one of the  big ones that's one of the big ones and  sometimes you can't  okay the nature of the problem is maybe  that it's that it's non determinacy you  may have asynchronous inputs coming in  and so forth so so this is the Golden  Rule we also have a silver rule silver  rule says never write non-deterministic  parallel programs but if you must always  devise a test strategy to manage the non  determinism okay so this gets into you  better have some way of handling how  you're gonna tell what's going on if you  have a bug okay so what are some of the  typical test strategies that you could  use that would manage the non  determinism okay so imagine you've got a  parallel program and it's it's got races  in it and so forth  you know and its operating  non-deterministically what and that's  okay if it's everything's going right  how would you you find a bug in the  program how are you what are you going  to do  what kinds of ideas do you have yeah  [Music]  yeah you could turn off the  non-determinism okay you put a switch in  there that says well I know the source  of this non-deterministic behavior let  me do that let me give you an example of  that for security reasons these days  when you allocate memory its allocated  to different locations on different runs  of the program its allocated in random  places they want to randomize the  addresses when you call malloc okay that  means that you can end up with different  behaviors okay from one to run and that  can compromise your your performance but  there turns out that there is a compiler  switch and if you run it in debug mode  it'll always deliver the the results of  malloc in in deterministic locations  okay where the locations the of the  things your mouth looking are you know  are repeatable okay so that's that's  good okay because there's support they  said yes we have to randomize okay for  security reasons so that people can't  deterministically exploit buffer  overflow errors for example but you know  I don't want to have to do that every  time okay so you know I I don't want to  randomize you know every time I run I  won't have the option of making it so  that that randomization is turned off  okay so that's a good one  what's another one that that can be done  you're full of good ideas this is like  let's try somebody else or not okay  that's good I like that I like that what  are some other ideas what else can you  do to handle non determinism okay you  got a program and it's yeah yeah yeah  yeah if you have random numbers you use  the same seed in some sense that's a  kind of of same thing if you're turning  off non determinism okay but that's a  great one you know there are other  places for example if you read you know  if you do get time of day you know for  something in your program for something  you could have an option where it'll put  in a particular fixed value there so you  can make sure that it doesn't you know  even a serial program isn't  non-deterministic okay so that's good  but I also consider that to be that's  another great example of turning off non  determinism okay what other things can  you do yeah  [Music]  ah yep you can do record replay for some  things is that what you're saying is  that what you mean or am i okay so  record replay says you run it through  you can run it through with random  numbers and then but it's recording  those things so that when you when you  run it again you can instead of using  the random numbers that you you know new  random numbers it uses the ones that you  used to use okay so that's a record  replay thing is that what you're saying  are you saying something else yeah okay  good  okay so this so that's using using some  tools there's actually a lot of  strategies let me just move on in answer  so another thing you can do is  encapsulate the non-determinism so  that's actually done in the silk runtime  system already the runtime system is  using a random scheduling strategy but  you don't see that it's random in the  execution of your code if you don't if  you have no race conditions in your code  its encapsulated okay so that so that  the you know in the platform so the  platform is going to guarantee you  deterministic results even though  underneath the covers it's doing  non-deterministic things okay you can  also substitute a deterministic  alternative okay sometimes there's  there's a way of computing something  that is non-deterministic but in debug  mode let me not use the  non-deterministic one okay and you can  also use analysis tools which can tell  you things about your program and which  you can can control things so there's a  lot of things so whenever you have a  non-deterministic program you want to  find some way of controlling and often  the non determinism is over in this  corner but your bug is over in this  corner so if you can turn this thing off  in some way or encapsulate it or you  know otherwise control the non  determinism over there now you have a  better chance of catching the stuff over  here  okay that's going to be particularly  important in project four when we get to  it  because that's going to be actually  going to be doing non-deterministic  programming for a game playing program  and one of the one of the things is that  the processors are in this case keeping  the game positions together and so the  if one processor stores something into a  into what's called the transposition  table which is a essentially a big hash  table of positions at scene another one  can see that value and change its  behavior and so one of the things you  want to be able do is turn off  transposition table okay so that so that  you don't take advantage that  performance advantage but now you can  debug the search code or you can debug  the evaluation code and so forth you can  also do things like you know unit  testing so you know whether or not a  particular piece is correct that might  have you know so that so that you can  test this thing separately from the rest  of your system which may have non  determinism and way this is a major  thing so so you never write them but if  you have to you always want to have some  test strategy okay and so for the people  who are not watching this video and who  are you know not in class today they are  going to be sorely hampered by not  knowing this lesson when they go into  the fourth project okay so so what we're  gonna do is now we're going to talk  about how to do non-deterministic  programming okay so this is you know  there's always there's always some part  of your code you know which has a skull  and crossbones  like you know you have this abstraction  it's beautiful and you can design etc  and then somewhere there's this really  ugly thing that nobody should know and  you put the skull and crossbones on that  and only experts go in well anyway  that's the that's the barrier we're  crossing here okay and we're going to  start out by talking about something  that you've probably seen in  your other classes mutual exclusion and  atomicity so I'm going to use the  example of the hash table so here's a  typical hash table it's got collisions  resolved by chaining so you have a bunch  of linked lists you hash to a particular  slot in the table and then you chase  down the linked list to find the value  and so for example if I'm going to  insert X which has a key value of 81 you  know what I do is figure out which slot  I go to by by hashing the key and then  in this case I made it be the last one  so the animations could be easier but if  it were in the middle and so so now what  do I do is I make the the pointer of X  go to the you know first element of that  list and then I make the slot value now  point to X and that effectively in with  a constant number of operations inserts  X into the hash table and in particular  into the linked list in the slot that  it's supposed to be okay this is all  familiar right okay so now what happens  when you have multiple parallel  instructions that are accessing the same  the same locations okay so so here we  have two threads one inserting X and one  inserting Y and X goes it does its thing  it hashes to there and it then sets the  next pointer to be the to add itself  into the list and then there's this  other thing going on in parallel which  effectively says oh I'm going to hash  oh we're going to the same slot it  doesn't know that somebody's already  there and so then it decides it's going  to put itself in as the first element  of the list and then it sets the value  of y that sets the value of the slot to  point to Y and then along comes X  finishing off what it's doing and it  points the value to X and you can see  that we have a race bug here a really  nasty one because we've just destroyed  the integrity of our of our system we  now have in particular y is sort of  floating not in the list when it's  supposed to be in the list right so the  standard solution to this is to make  some of these instructions be atomic ok  so and what that means is the rest of  the system can never view them as being  partially executed okay so they're  either all have been executed or none of  them have been executed at any point in  time as far as the as far as the rest of  the system is concerned okay and the  part of code that is within the atomic  regions called the critical section and  it can typically a critical section of  code is someplace that should not be  being executed by two things at the same  time so the standard solution to  atomicity is to use what's called a  mutex lock or a mutual exclusion lock  okay and it's basically an object with a  lock and unlock member functions and an  attempt by a thread to lock an already  locked mutex causes the thread to block  okay that is wait until the mutex is  unlocked okay so if somebody grabs the  lock somebody else grabs the lock and  it's already taken then they have to  wait and they sit there waiting until  until this guy says yes I'm gonna  release it okay so so what we'll do now  is we'll make each straw each slot be a  struct with a mutex l and a pointer head  to the slot context so it's going to be  the same data structure we had before  but now I'm gonna have not just the  pointer from the slot but I'll also have  I also have a a lock in that position  okay and so the idea of in the code now  is that before I access the lock before  I access the list I'm going to lock that  that list in the table by locking the  slot then I'll do the things that I need  to do and then I'll unlock it and now  anything else can go on because you know  the what's happening is the reason we're  getting into trouble is because we got  some sort of interleaving of operations  and our goal is to make sure that it's  either doing this or doing this and  never this okay to make sure that so  that each thing each piece of code is  restoring the invariant of correctness  after it executes the pointer swaps them  very in this case is that the elements  are in a list okay and and so you want  to restore that with each one okay so so  mutex is this is one way you can use new  mutexes to implement atomicity okay so  now let's just go back this is this is  so who has who has seen you know mutexes  before it's at pretty much everybody  yeah okay good I hope that this is not  brand-new for too many of you if it is  brand-new that's great but what I'm  trying to do is make it so let's go back  a little bit and recall in this class  our discussion of determinacy races so  remember that determinacy race occurs  when you have to logically parallel  instructions that access the same memory  location and at least one of them  performs all right okay so mutex locks  can guarantee that critical sections  behave atomically but the resulting code  is inherently non-deterministic because  you've got a you know we had a race bug  there right we had two things trying to  access the same slot but that may be  what I  want right I want to have a shared hash  table maybe okay for these for these  things so I want something where there  is a race but I just don't want to have  the anomalies that arise in this case  you know the race bug caused things and  I can solve that with atomicity okay you  know if you have no determinacy races it  means that the program is deterministic  on that input and that it always behaves  the same okay and remember also that if  a determinacy race exists in an  ostensibly deterministic program okay  then it guarantees to find a race now if  you put in mutexes you still have a  non-deterministic program you still have  a race because you have two things that  are logically parallel that are both  accessing the lock that's a rate that's  a determinacy race okay if you have two  things that are in parallel they're both  accessing the lock  that's a determinacy race it may be a  safe correct one but it is a determinacy  race okay and so any codes that use  locks are non-deterministic by intention  and they're going to invalidate the  silks and guarantee of finding those  race bugs okay so you will end up with  races in your code if you're not careful  okay and so this is one reason it's  important to have some way of turning  off non determinism to detect stuff  because what you don't want is that  whole rash of false positives saying oh  you raced on gathering this lock nor do  you want to ignore that and then  discover that a race has popped up  somewhere else okay now some people feel  that so this is basically a talking  about having a data race okay and a data  race is similar to the definition of  determinacy race okay but it says that  you have to logically parallel  instructions and they don't hold locks  in common  and then it's the same definition if  they access the same memory location one  of them performs or right then you have  a then you have a data race bug okay but  if they if they have the locks in common  okay if they both have acquired at least  one lock that's the same then you don't  have a data race okay because that means  that you've now successfully protected  the atomicity but it is still  non-deterministic and there is a  determinacy race just no data race and  that's the big distinction between data  races and determinacy races and on quiz  2 you better know the difference between  data races and determinacy races okay  because they are different okay so a  program may have no determinant may have  no data races that doesn't mean that it  doesn't have a determinacy race in fact  if it's got any locks it probably has a  determinacy race okay okay so so one of  the things is if I have no data races  does that mean I have no bugs suppose I  have no data races in my code does that  mean I have no bugs this is quite an  obvious answer just bye-bye quiz  moonship right so what might happen  think about a little bit what might  happen how could I have no data races  and yet there's still be a bug even  though and assuming it's a correct piece  of code otherwise okay in other words  when it runs serially or whatever it's  correct how could I end up having code  no data races but still have a bug  [Music]  yeah but that doesn't mean it's bad  right yep  might still be a different yep  okay uh yeah let me give you an example  okay  which is more to the point here is a way  of making sure that I have no data race  which is I lock before I follow the  table slot value and then I unlock then  I lock again and then I set the value  okay so I haven't prevented the  atomicity right now I've got an atom  isset II violation but I have no data  races okay because I never have two  things that are going to access things  at the same time is protected by the  lock okay but it doesn't solve my  atomicity so there's a you know there's  you know you can definitely have no data  races but that doesn't mean you have no  bugs okay but usually what it happens is  if you have no data races then usually  the programmer actually got this code  right okay it's one of these things  we're demonstrating no data races is in  fact you know a very positive thing in  your code it doesn't mean the program I  did right but most of the time the  reason they're putting in the in the  locks is to provide atomicity for  something and they usually get it right  they don't always get it right in fact  Java for example had a very famous bug  early on in the way that it specified  locking such that you know the you could  look at the length of a string and then  modify it and then you would end up with  a race bug because somebody else could  swoop in in between so they thought they  were being providing atomicity and they  didn't okay  so there's there's another set of issues  here having to do with benign races now  there's some people who argue that no  races are  are no determinacy races are benign and  they make academic statements that I  find quite compelling actually what  they're what they say about about races  and whether races are benign but  nevertheless the literature also  continues to use the term benign race  for this kind of example so suppose we  want to identify what are is the set of  digits that perform that occurred in  this in in some array so here's an array  with a bunch of values in it okay each  one being a digit from zero to nine so I  could write a little piece of code that  runs through my digits array of length  10 and sets the number of digits I've  seen so far of each value to be 0 okay  and now I go through and I'm gonna do  this in parallel and I'm going to set  every time I see a value a of I suppose  a of I is 3 I set the location of a 3 to  be 1 and otherwise and now otherwise  it's zero because that's what I had it  before so here's the kind of thing I  have so for example I can have both of  those sixes are in parallel going to  access the location 6 to set it to 1 but  they're both setting it to 1 it doesn't  really matter what order they do it in  you're gonna get the same value there 1  ok and so there's a race maybe we don't  too much care about that race okay  because they're both setting the same  value we're not going to get an  incorrect value well not exactly  we might get on some architecture on the  Intel architectures you won't get an  incorrect value on x86 but there are  codes where the elements of the array  values are not set atomically so for  example on the mips architecture  in order to set a bite to be a  particular value you have to you have to  fetch the word mask out set the word and  then store it back in set the the bite  and then store it back into the word and  so if there are two guys who are  basically operating on that same word  location they will have a race even  though in the code it looks like they're  just setting bites does that make sense  okay so nasty nasty bugs okay that's  that's why you should never do  non-deterministic programming okay  unless you have to  okay so silks and allows you to turn off  race detection for intentional races  okay so if you really meant there to be  a race as in this case you can turn it  off  this is dangerous but practical turns  out usually you're not turning it off  for because here's what can happen you  can turn it off and yet then there's  something else which is using that same  stuff and now you're running silks and  without you know having turned it off  for exactly what your race might be  they're better solutions so in Intel's  silk screen there's the notion of fake  locks we just have not yet implemented  it in in the open silk compiler so we  will and in silks and we'll eventually  get to doing that and then people who  take this class in the future will have  an easier time with that okay because  we'll be able to check for that as well  okay so any questions about these  notions so you can see the notions of  races can get quite hairy and you know  make it quite difficult to do your  debugging and in fact even can confound  your tools that are supposed to be  helping you to get correct code okay  all a name of performance okay but we  like performance okay any questions  yeah  yeah so how can some architectures cause  some error so here's the thing is that  if I have a let's say a byte array it  may be that this is stored int as as a  set of let's say 4 byte words okay and  so although you may write that a of 0  gets 1 what it does is it says let me  fetch these 4 values because there is no  byte set instruction on some  architectures it can only set in this  case 32-bit words so it fetches the  values it then you know into a register  it then sets the value in the register  ok by masking so it doesn't set the  other things here and then it stores it  back right so that it has a 1 here but  what if somebody at the same time is  storing into this location they will  fetch it into their own register set  their byte mascot etc and now my right  back is going to we're gonna have a lost  update in the in the right backs does  that make sense ok good  very good question ya know I went  through that orally a little bit quicker  than maybe I should have ok great  so let's talk a little bit about  implementation I always like to take  things down one level below what you  necessarily need to know in order to do  things but it's helpful to sort of see  how these things are implemented and  because then that gives you a better  sense at a higher level what the what  your capabilities are and how things are  actually working underneath so let's  talk about how in the mutex is so here  first of all to understand there are  lots of different mutexes if you look at  an operating system they may have a half  a dozen or more different  you Texas different locks that can  provide mutual exclusion or parameters  that can be set for what kind of mute  exes so the first basic difference in  most things is whether the mutex is  yielding or spinning so a yielding mutex  returns control to the operating system  when it blocks when a program tries to  get active when it tries to get access  when a thread tries to get access to a  given low given lock if it is blocked it  doesn't just sit there and keep and  spinning where you're basically spinning  means I just sit there checking it and  checking it and checking it and checking  in ok instead what it does is it says oh  I'm doing useless work here let me go  and return control of the operating  system maybe there's another thread that  can run at the same time and therefore  I'll give by switching myself out by  yielding my my scheduling quantum I will  get better efficiency overall because  somebody some other thread that is  capable of running can run at that point  ok so is that clear the distinction  between spinning and and yielding ok  another one is whether the mutex is  reentrant or non re-entrant a reentrant  mutex allows a thread that is already  holding a lock to acquire it again ok a  non reentrant one deadlocks if the  thread attempts to require a mutex that  already holds ok so I grab a lock and  now I go to a piece of code and it says  grab that lock so very simple I can  check to see whether I have if I want to  be reentrant I can check do I have that  lock already and if I do then I don't  actually have to acquire it I just keep  going  but that's extra overhead it's faster  for me to have a non reentrant lock ok  where I just simply grab the lock and if  somebody's got it  including me then it's a dead lock but  now if there's the possibility that I  could react wire a lock then that might  not be safe  okay you have to worry about the  programmer has to worry about that now  is that clear that one okay  and then final basic property of mutexes  is whether they're fair or unfair okay  so here's the thing with it's easiest to  think about it in the context of  spinning  I have several threads that basically  came to the same lock and we decide  they're gonna spin they're just going to  sit there continually checking waiting  for that lock to be free okay so when  finally the guy who has it unlocks it  you know maybe I've got half a dozen  threads sitting there one of them wins  okay and which one wins well if they're  spinning it could be any one of them  then it has one and so the issue that  can go on is you could have what's  called a starvation problem okay where  some guy is sitting there for a really  long time waiting while everybody else  is continually grabbing locks out from  under his or her nose okay so with a  fair mutex basically what you do is you  go for the one that's been waiting the  longest essentially and so therefore you  never have to wait more than for however  many things were there when you got  there before you're able to go question  why  [Music]  it can be better because you may freeze  out or service okay if there's something  that's you know you may never get to do  the thing you want to do okay because  there's something else always  interfering with the ability for that  part of the program to make progress  this tends to be more of an issue in in  concurrent programming where we have  different programs that are trying to  accomplish different tasks and you want  to accomplish both tasks okay  does not come across in parallel  programming mostly we deal with unfair  often unfair spinning locks because  they're the cheapest and we just trust  that a we're not going to do any have  any critical regions we write our code  so we don't have critical regions that  are really long so nobody ever has to  wait a very long time but indeed dealing  with a contention issue as we talked  about last week you know can can make a  difference okay good so here's an  implementation of a simple spinning  mutex in assembly language okay  so the first thing it does is it checks  to see if the the mutex is free if it's  value is 0 so it compares the value of  the mutex to 0 and if it is 0 it says oh  the it's free let me go get it it then  to get the mutex what it does is it  moves one into the it basically moves  one into a register and then it  exchanges the mutex with that register  EAX ok and then it compares to see  whether or not it actually got the mutex  and if it didn't then it goes back up to  the top and starts again ok and then the  other branch is at the at the top there  it does this pause and this apparently  is due to a bug in x86 that they end up  having to put this pause instruction in  there and then otherwise you jump to the  where the spin mutex is and and go go  again  okay and then once you've done the  critical section when you're done you  free it by just setting it to zero so  the question here is so the exchange  instruction is a is a is an atomic  exchange so it takes the register and  the memory value and it swaps them and  you can't have anything come in so one  of the things that might have you  confused a little bit here is wait a  second I check to see if the mutex is  free and then I tried to get it and to  test if I was successful why why can't I  just start out by essentially going to  get mutex right I mean why do I need why  don't need any of the code between spin  mutex and get mutex  right so if I just started with getting  new text I would move one in I would  exchange check to see if I could get it  if I have it fine then I execute the  internet if not okay I go back and and  try again so why you know because if  somebody has it by the way the value  that I'm gonna get is going to be one  and that's what I swapped in so I  haven't changed anything I go back and I  check again so so why do I need that  first part yeah yeah maybe it's faster  okay  so indeed it's because it's faster even  though you're executing extra code it's  faster tell me why it's faster and this  tape will take you you have to think a  little bit about the cache protocols and  the invalidation issue so why is it  gonna be faster yeah okay good say more  [Music]  yeah so so it turns out the exchange  operation is like a right and so in  order to do a right what do I need to do  for the for the cash line that it's on  to bring it in but what how does it have  to be brought in remember the cash lines  had let's let's a Matt you have to  invalidate it on the other ones and you  have to hold it in what state  remember the cash lines have you know if  we take a look at just a simplified  protocol right where the MSI protocol  [Music]  yeah you have to have it in the EMA MSI  or an ESI you have to bring it in in  modified or at least exclusive state  right so exclusive is for the Emmy ESI  protocol we we mentioned that but we  didn't really do it mostly we just went  but I have to bring it in and modified  where I guarantee there are no other  copies so if I've got two guys that are  polling on this location  they're both continually invalidating  each other and you create a whole bunch  of traffic on the memory network that's  going to slow everything down okay  whereas if I do the first one what state  do I get it in then you get it in shared  State what does the other guy get it in  shared state and now I keep going just  having it in spinning in my own local  cache not generating any local traffic  until the until somebody releases the  Falak in which case it invalidate Sall  those and now you can actually get it a  little bit of a storm after the fact  there in fact locks where you don't even  get a storm after the after the fact but  called mcs locks but but this kind of  lock is for most practical purposes just  fine okay so so to everybody follow that  description of what's going on there so  that first code for correctness purposes  not important for performance it is  important okay isn't it great that you  guys can read assembly language right  okay now suppose that this is a spinning  mutex suppose that I want to do a  yielding mutex how does this code have  to change  okay so this is a spinning one just  keeps checking instead I want to return  control of the operating system okay so  how does this code change if I do that  yeah  [Music]  like that yeah exactly  okay so instead of doing that that you  know  pause instruction which the  documentation on this is not very clear  I'm love to have the inside scoop on why  they really had to do the pause there  but in any case you take that no op that  they want to have in there and your  place it with it just a call to the  yield which allows the operating systems  to schedule something else and then when  it's your turn again it resumes from  that point okay so that's the yield so  so that's the difference in the  implementation essentially between a  spinning mutex and a yielding mutex okay  now there's another kind of mutex that  is kind of cool which is called a  competitive mutex so think about it this  way  I've competing goals one is to I want to  get the mutex as quickly as possible  after it's released right I don't want  if if it's unlocked I don't want to sit  there for a really long time and before  I actually acquire okay and - yeah but I  don't want to sit there spinning for a  really long time and then you know  because as long I'm doing that I'm  taking up cycles and not accomplishing  anything let me turn it over to some  other thread that can can use the cycles  more effectively so there at those two  goals how can I get the best of both  worlds here something that's close to  the best of both worlds that's not  absolutely the best of all flows but  it's close to the best of both worlds  what strategy could I do so I want to  claim it very soon so so the point is  that the spinning mutex achieves goal  one and the yielding mutex achieves goal  two so how can i what can I do to get  both goals yeah  [Music]  thank you  so you're saying use message-passing to  inform the waiting threads I'm thinking  something a lot simpler  okay in this context so because the  message passing you're gonna have to go  through to do message passing properly  you actually need to use mutex as a  setter to implement it so you want to be  a little bit careful you know about  about that but but interesting idea yeah  using an interrupt how would you do that  yeah so typically implement interrupts  you also need to have some mutual  exclusions to do it properly but I mean  hardware will support that  that's pretty heavy-handed as well this  is actually a very simple solution who's  everybody I'm seeing familiar hands I  want to see so unfamiliar hands who's  got an unfamiliar hand okay I see you  raised your left hand that time instead  of your right hand okay yeah  it hard to measure that right how would  you write code to measure that yeah  mm-hmm yeah good okay good  [Music]  why doesn't it have a because if I yield  okay so what's the how often does if I  contact switch how often is it going to  be that I how long am I going to have to  wait typically before I am scheduled  again okay when when a when a code  yields to the operating system how often  does the operating system normally do  context switching do you know what's  that what's the rate at which it  contacts switches for the different  multiplexing of threads that it does on  to on to the available processors what's  the rate at which it it shifts oh this  is what okay that's gonna be on the quiz  okay this is a new numeracy thing yeah  how do you know how frequently  not quite but but your your you're not  off by more than an order of magnitude  okay so what are the typical rates that  you that the system does contact  switching okay so from human times it's  the blink of an eye right so it's  actually around 10 milliseconds so it  would does 100 times a second some of  them do some do 60 times a second okay  that's how often it switches okay now  let's say it's a hundred times the  second 10 milliseconds so you're pretty  close okay 10 milliseconds okay how many  orders of magnitude is that from the  execution of a simple instruction okay  so we're going at more than a gigahertz  right and so a gigahertz is 10 to the  ninth and we're talking ten to the minus  nine and we're talking ten to the minus  two so that's you know 10 million  instruction opportunities that we miss  if we switch out and of course we  probably only switch out for half hour  you know where are you along the the  thing so you're only switching out maybe  for half assuming nothing else is going  on there but that's that means you're  not grabbing the lock quickly after it's  released because you've got ten million  instructions that are going to execute  before you're gonna have a chance to  come back in and grab it okay  so that's why a yielding one is does not  grab it quickly  okay we're spinning is like we're  executing this stuff at the rate of  gigahertz right checking again checking  and checking again okay so why so what's  what's the strategy here what can I do  yeah  hey what a good idea okay okay spin for  a while and then yield okay so the idea  being hey if if the if the lock is  really soon then I will be able to grab  it immediately because I'm spinning and  if it takes a long time for the lock to  yield well I will yield eventually so  yeah but okay how long does spin how  long shall I spin sure  [Music]  yeah basically as long as a context  switch takes okay as long as it takes to  go out and come back okay and if you do  that then you never wait more than twice  the optimal time okay this is  competitive analysis which the  theoreticians have gone off there's  brilliant work in competitive analysis  so the idea here is that if the mutex is  released while you're spinning then the  strategy is optimal right because you  just sat there spinning and as soon as  was there you got it on the next cycle  if the mutex is released after the yield  you've already spun for equal to that so  you'll come back and get it at within it  most a factor of two okay this is by the  way that the way this shows up in the  Theory literature if you're introduced  it's called the ski rental problem okay  and here's the idea you're gonna go you  know your friends are persuading you to  go try skiing okay right snow skiing  right right okay and so you say gee  should I buy the equipment or should I  rent after all you may discover it that  you rent and then I mean you buy it and  then you break your leg and never want  to go back okay well then if you've  bought it's been very expensive and if  you've rented well then then you're  probably better off on the other hand if  it turns out you like it  you're now accumulating the costs going  forward okay and so the question is well  what's your strategy and the idea is  well let's look at what renting costs  and what buying costs let me rent until  it's equal to the cost of buying and  then buy and then I'm within a factor of  two of having spent the optimal amount  of of money for because then if I break  my leg after that well at least I you  know you know I got you know I I I  didn't spend more than a factor of two  and if I get it before it I've spent  optimally  okay yeah  yeah yeah 10 milliseconds yeah so spin  for 10 milliseconds okay and then switch  so now the point is that that when you  come back in okay the other job is gonna  run for 10 milliseconds or whatever okay  so if you get switched out then if the  lock is released you're gonna be done in  20 milliseconds right and so you'll be  within a factor of 2 and if it happened  if they lock out release before then  you're right there to grab it okay now  it turns out that there's a really  clever randomized algorithm I love this  rat algorithm okay from 1994 that  achieves a competitive ratio of e over e  minus 1 using a randomized strategy and  I'll encourage you to you know those of  you who have a theoretical bent to go  take a look at that it's very clever  ok so basically have some probability of  at every step of whether you at that  point decide to yield you know or  continue spinning ok and by using a  randomized strategy you can actually get  this to Yi over u minus 1 ok questions  about this so this is sort of some of  the basics I'm glad we went over some of  that because everybody should know these  basic numbers about what things cost  because otherwise you don't know where  to spend it so context switching time is  on the order of 10 milliseconds how long  is the disk access compared to you know  yeah what's the disk access  150 cycles hmm that's an that that would  be accessing DRAM okay  accessing DRAM if you if it wasn't in  cache might be 150 cycles okay  so two orders of magnitude or so okay so  what what about a disk access how long  is that take yeah  yes several milliseconds so ten  milliseconds that are five milliseconds  depend upon how fast your disk is but  once again it's on the order of  milliseconds so it's helpful to know  some of these numbers because otherwise  you know where are you spending your  time especially we're sort of doing  performance engineering you know in the  small basically looking within the  product you know within a multi-core  processor most performance engineering  is on all the stuff on the outside  dealing with networking and file systems  and stuff where where things are really  costly and where you actually have a lot  of time you can write fast piece of code  that can figure out how you should best  deal with these slow parts of your  system okay so those are all sort of  good numbers to know you'll probably see  some of them on quiz 2 okay okay  deadlock I mentioned deadlock earlier  let's talk about what deadlock is and  understand this once again I expect some  of you have seen this but I still want  to go through it because it's hugely  important material and this is the issue  that holding more than one lock at a  time can be dangerous okay so imagine  that thread one says I'm gonna lock a  lock B execute the crudo section unlock  B unlock a where a and B or mutexes and  thread two does something very similar  it locks B and locks a then it does the  critical section then it unlocks a and  then unlocks B okay so what can happen  here so we you know thread one locks a  thread two locks B thread  one can't go and lock be because thread  two has it thread two can't go and lock  a because thread one has it so they sit  there blocked I don't care if they're  spinning or yielding they're not going  anywhere okay so this is the ultimate  loss of performance right it's like it's  incorrect okay it's like you're stuck  you've deadlocked okay now there's  there's three basic conditions for  deadlock everybody understands this  right there's there any anybody who has  a question because just okay it's  there's three conditions you need for  deadlock the first one is mutual  exclusion that you're going to have  exclusive control over the resources the  second is non preemption you don't  release your resources you hold until  you finish using them okay and and three  is circular waiting you have a cycle of  threads in which each thread is blocked  waiting for resources held by the next  one in this case the resource is the  lock okay and so if you remove any one  of these constraints okay you can come  up with solutions that won't deadlock so  for example it could be that when I try  to acquire a lock if somebody else has  him I take it away  okay you know that could be one thing  now there may get into other issues  which is like well but what if he's  actually doing real work okay  or whatever so all of these things have  things or you know I don't insist that  it be mutual exclusion except that's the  kind of problem that we're trying to  solve so these are generally the three  things that are necessary in order to in  order to have a deadlock situation now  in any discussion of deadlock you have  to talk about dining philosophers when I  was an undergraduate and I graduated in  1975 from Yale humanities school  I was taught the dining philosophers  right because after all philosophy is  what you find at humanity schools I mean  we have philosophy department - don't  get me wrong but at Yale the humanities  is huge and so philosophy I guess they  thought this was would appeal to the  people who were not real techies in the  background I sort of like I was a techie  in the midst of all these you know  non-technical people dining philosophers  is a story of deadlock told by Tony  Hoare based on an examination question  by Edsger Dijkstra okay  and it's been embellished too over the  years by many many many retailers okay  and I like the Chinese version of this  okay there's versions where they use  Forks but I'm going to this is going to  be their dining I'm going to say that  they are eating noodles with chopsticks  and they're in philosophers seated  around the table and II and between  every plate of noodles there's a  chopstick and so in order to eat the  noodles they need two chopsticks okay  which to me sounds very natural okay and  so here's the code for philosopher I so  he's a philosopher so he starts by  thinking for a while okay and then he  gets hungry he or she gets hungry so the  philosopher grabs the chopstick on the  law right on the Left sorry and then it  grabs the one on the on the right but it  had you know so which is I plus one but  it has to do that mod in because if it's  the last one you got to go around and  grab the first one then eats and then it  unlocks the two chopsticks and now they  can be used by the other dining  philosophers because they don't think  much about sanitation and  and so forth right so they're because  they're they're too busy thinking right  but but what happens what's wrong with  this solution what happens what can  happen for this it's very simple I need  two chopsticks I grab one I grab the  other I eat one day what happens yeah  yeah they're all it and they grab one to  the left and now they go to the right  it's not there and they starve okay you  know one day they got all the things so  we have the starving philosophers  problem so so be motivated by this  problem yeah question yeah so if you are  willing to preempt then that would be  preemption as I say it's got to be non  pre-emptive in order for deadlock to  occur in this case yeah but you also  have to worry in those cases could be oh  well if I couldn't get both let me put  them both down but then you can have a  thing that's called livelock okay so  they all pick up their left they see the  right ones busy so they put it down so  somebody else can have it they look  around oh okay let me pick out one no no  okay okay and so they still starve even  though they've done that so in that kind  of situation you could put in a time  delay you could say let me let everybody  pick a random number have a randomized  scheme so that we're not you know so  there are other solutions if you don't  insist on non preemption I'm going to  give you one where we have non  preemption but we still avoid deadlock  and it's to go for that cyclic problem  so here's the idea  suppose that we can linearly order the  mutexes okay so I pick some order of the  mutexes so that whenever a thread holds  a mutex else of I and attempts to act a  lock another mutex l sub J we have that  in this linear order L sub I comes  before L sub J then you can't have a  deadlock okay so in this case for the  dining philosophers it would for example  number the number of the chopsticks you  know from 1 to N and then or 0 to n  minus 1 whatever and then grab the  smaller one and then grab the larger one  then then it says then you would never  have a deadlock  okay and so here's the proof you know I  like proofs okay  proofs are really important so I'm going  to show you that if you do that you  couldn't have a cycle of waiting so  suppose you had a cycle of waiting we  were in a situation where everybody is  holding chopsticks and you know one that  one of them is waiting for another one  which is waiting for now all the way  around to the first one that's what we  need for deadlock to occur so let me  just look at what's the largest mutex on  the cycle let's call that L max okay and  suppose that it's waiting on a mutex L  held by the next thread in the cycle  well then we have a something that's  bigger than the maximum one okay and so  that contradicts the fact that I grabbed  them whenever I grabbed them I do it in  order okay so very simple very simple  proof that you can't have deadlock if  you grab them according to a linear  order okay and so for this particular  problem what I do is instead of grabbing  the one on the left and one on the right  as I say you grab the smaller of the two  and then grab the larger of the two and  then you're guaranteed to have no  deadlock does that make sense now if  you're going to use locks in silk you  have to realize that in the operating  sin the runtime system of silk they're  doing they're using locks you can't see  them they're encapsulated as we talked  about the non determinism in silk is  capsulated is still going on underneath  the color covers and if you start  introducing your own non determinism  through the use of locks you can run  into trouble if you're not careful and  let me give you an example okay this is  a situation you can deadlock your  program in silk okay with just one lock  okay  so here's an example of a code that does  that so main spawns off foo and foo  basically locks the lock L and then  unlocks it and meanwhile  after it spawns our food the  continuation goes and it locks L itself  and then does a sink and then it unlocks  it so what happens here we sort of have  a situation like this where where the  locking I've done with an open bracket  and an unlock a release I'm doing with a  closed bracket so I'm spawning off foo  which is the lower part there and  locking and unlocking and up above I'm  locking and unlocking so what can happen  here I can go and I basically spawn off  the child but then I lock and now the  child goes and it says oh I can't you  know foo is going to wait here because  it can't grab the lock as it's owned by  the by main ok and now we get to the  point where main is has to wait for the  sink okay and the child is never going  to complete because I hold the resource  that the child needs to complete okay so  don't hold mutexes across silk sinks  that's the lesson there there are  actually places you can but if you don't  hold them across that then you won't run  into this particular problem a good  strategy is only holding mutexes within  strands okay so there's no parallelism  so you you have it bounded and also  that's a good idea generally because you  want to hold mutex as a short amount of  time as you possibly can  so for example if you have a big  calculation and then you want to sign  something atomically don't put the big  calculation inside the critical region  move the calculation outside the  critical region do the calculation you  need to do and then acquire the locks  just to do the interaction you need to  set a value okay so that's and then  you'll have a lot faster code okay  because you're not holding up other  threads for a long time and always try  to avoid non-deterministic programming  but that's not always possible okay I  want to so any questions about that then  I want to go on a really interesting  topic because it's really  you know recent research level topic and  that's to talk about transactional  memory who's heard this term before  anybody okay so the idea is to have  database transactions that you have  things like database transaction or the  atomicity is happening implicitly you  don't specify locks you just say this is  a critical region okay don't interrupt  me while I do this critical region the  system works everything out okay here's  a good example of where it might be  useful suppose we want to do a  concurrent graph computation okay and so  you take people involved in parallel and  distributed computing and at MIT and you  say okay I want to do Gaussian  elimination on this graph now you guys  are sure most of you know Gaussian  elimination from the matrix context do  you know what it means in a graph  context so if you have a sparse matrix  you actually have a graph and Gaussian  elimination is a way of manipulating the  graph and you get exactly the same  behavior as you get in the dense one so  I'll show you what it what it is you  basically pick somebody to eliminate  okay  okay and now what you do is look at all  this all the this vertexes neighbors  okay  those guys and what you do is you  eliminate that vertex bye-bye okay  and you interconnect all the neighbors  with all the edges that don't already  exist and that's Gaussian elimination if  you think of it in terms of matrix  fashion the question is if you have a  sparse matrix where are you going to get  fill-in what are the places that you  need to update when you do a pivot in  Gaussian elimination in a matrix okay so  that's the basic notion of graph of  doing Gaussian elimination but now we  want to deal with the concurrency okay  and the problem occurs is if I want to  eliminate two nodes at the same time  okay because now they're adjacent each  other and if I just do what i Express  there's going to be all kinds of  atomicity violations etc by the way the  reason I'm picking these two folks is  because they're going to a better place  okay so so how do you deal with this and  so in transactional memory what I want  to be able to do is just simply say okay  here's the thing that I need to be  atomic and so if I look at this code  it's basically saying you know who are  my neighbors and then let me identify  all of the edges that need to be removed  the ones that I just showed you they  know that we're moved now let me get rid  of the element you know V and now for  all of the neighbors of U let us let us  add in the edge between the neighbor and  you know between the pairs of neighbors  okay so that's basically what it's doing  okay and I'd like to just say that's  atomic okay and so the idea is that if I  express that as a transaction then the  ID  is that on the transaction commit all  the memory updates in the critical  region appear to take it happen at once  however in transaction member the idea  is rather than forcing it go forward I  can have the transactions abort okay so  if I get a conflict I'll abort one and  restart it and then the restarted  transaction may take a different code  path because after all I may have  restructured the graph underneath and so  it's may do something different the  second time through the first it may  also abort again and so forth so when  you study transaction transactional  memory let me just do a couple of -  there's one is a conflict that's when  you have two transactions that are you  know they can't both complete one of  them has to be aborted okay and aborting  by the way is once again violating the  non you know you know the non  pre-emptive nature here we're going to  preempt one of them by by keeping all  the states so I can roll a state back  and restart it from scratch so  contention resolution is deciding which  of the two conflicting transit actions  to wait or to abort and restart and  under what conditions you do that so the  resolution manager has to figure out  what happens in the case of contention  and then forward progress is avoiding  deadlock of course but also livelock and  starvation you want to make sure that  you're going to make because what you  don't want to have happen for example is  that two transactions keep aborting each  other and you never make forward  progress and throughput well you'd like  to run as many transactions as  concurrently as possible so I'm going to  show you an algorithm for doing this  it's a really simple algorithm okay it  happens to be one that I discovered just  a couple of years ago okay and I was  surprised that it did not appear in the  literature and so I wrote a very short  paper on it because it's because what  happens for a lot of people is they you  know in if they discover there's a lot  of aborting they say oh well let's grab  a global lock  and then if everybody grabs a global  lock you can do this sort of thing you  can't deadlock with a single lock if if  you're not also doing things like you  know silk sync or whatever but but in  any case if you have just a single lock  everybody falls back to the single lock  and then you have no concurrency in your  program no performance until everybody  gets through the difficult time so this  is an algorithm that doesn't require a  global lock okay so it assumes that  transactional memory system will log the  reads and writes that's typically true  of any transaction maybe you log what  reads and writes you're doing so that  you can either abort and rollback or you  can when you abort you or else you  sandbox things and then atomically  commit them okay and so we have all the  mechanisms for boarding and rolling back  these are all very interesting in their  own right and restarting and this is  going to basically use a lock based  approach that uses two ideas one is the  notion of what's called a final finite  ownership array and then another is a  thing called release sort require and  let me explain those two things and I'll  show you really quickly how this  beautiful algorithm works okay so you  have an array of anti starvation mutual  exclusion locks so these are ones that  are going to be fair okay so that so  that you're always going for the oldest  one and you can do in a acquire we're  also going to add in a try acquire tell  me whether if I tried to acquire I would  get it that is megive it to me if I  don't get it  don't wait just tell me that I didn't  get it okay and then release and there's  an owner function that map's  all of the function H that map's my  universe of memory locations to the  indexes in this finite ownership array  this lock array okay so the lock has  length array has length n has n slots in  it to lock a location X in the set of  all possible  you actually require lock of H of X so  you can think of H as a hash function  but doesn't have to be a fair hash  function or whatever any function will  do okay and then there yes there'll be  some advantages to picking some  functions or another one okay so in  rather than actually locking the  location or locking the object I lock a  location that I that essentially I hash  to from that object so if two guys are  trying to grab the same location they  will both grab the same lock because  they've got the same hash function but I  may have inadvertently where if I were  locking the objects themselves okay I  wouldn't have them both trying to  acquire the same lock that might happen  in this algorithm okay so here's the  idea it's called the first idea is  called a release sort and reacquires so  that's the ownership array part that I  just explained now here's the release  sort a quat reacquire before you access  a memory location X simply try to grab  lock of X greedily and if you if you  have a conflict okay so if you don't  have a conflict you get it you just  simply try to get it if you can that's  great if not then what I'm going to do  is roll back the transaction but don't  release the locks I hold and then  release all the locks with index is  greater than H of X and then I'm gonna  choir the lock that I want and now at  that point I've released all the bigger  locks so I'm acquiring the next lock and  then I acquire the released locks in  sorted order so I go through all the  locks I release and I really min sorted  order okay and then I start my  transaction over again I try again so  what happens each time through this  process  I'm always whenever trying to acquire a  lock I'm only holding locks that are  smaller but each time that I restart I  have one more lock that I didn't used to  have before I restart my transaction  which I've acquired in the order in the  linear order from you know from the  in that ownership array from zero to n  minus one okay and so here's the  algorithm I'll let you guys look at it  in more detail because I see our our  time is up and it's actually fun to take  a look at and we'll put the paper online  there's one other topic that I wanted to  go through here which is which you  should know about is this locking  anomaly called convoying and this was  actually a bug that we had performance  bug that we had in our original MIT silk  so it's kind of a neat one to see and  how we resolved it and that's it
 

3
00:00:05,769 --> 00:00:08,019

 

4
00:00:08,019 --> 00:00:09,850

 

5
00:00:09,850 --> 00:00:10,930

 

6
00:00:10,930 --> 00:00:13,120

 

7
00:00:13,120 --> 00:00:15,160

 

8
00:00:15,160 --> 00:00:22,980

 

9
00:00:22,980 --> 00:00:27,099

 

10
00:00:27,099 --> 00:00:31,960

 

11
00:00:31,960 --> 00:00:33,670

 

12
00:00:33,670 --> 00:00:35,170

 

13
00:00:35,170 --> 00:00:45,189

 

14
00:00:45,189 --> 00:00:47,110

 

15
00:00:47,110 --> 00:00:50,649

 

16
00:00:50,649 --> 00:00:52,719

 

17
00:00:52,719 --> 00:00:58,209

 

18
00:00:58,209 --> 00:01:02,020

 

19
00:01:02,020 --> 00:01:03,729

 

20
00:01:03,729 --> 00:01:09,940

 

21
00:01:09,940 --> 00:01:11,770

 

22
00:01:11,770 --> 00:01:14,770

 

23
00:01:14,770 --> 00:01:17,290

 

24
00:01:17,290 --> 00:01:20,830

 

25
00:01:20,830 --> 00:01:22,690

 

26
00:01:22,690 --> 00:01:29,850

 

27
00:01:29,850 --> 00:01:34,560

 

28
00:01:34,560 --> 00:01:37,030

 

29
00:01:37,030 --> 00:01:38,469

 

30
00:01:38,469 --> 00:01:42,010

 

31
00:01:42,010 --> 00:01:47,830

 

32
00:01:47,830 --> 00:01:50,080

 

33
00:01:50,080 --> 00:01:53,319

 

34
00:01:53,319 --> 00:01:56,139

 

35
00:01:56,139 --> 00:02:00,730

 

36
00:02:00,730 --> 00:02:02,679

 

37
00:02:02,679 --> 00:02:05,760

 

38
00:02:05,760 --> 00:02:08,380

 

39
00:02:08,380 --> 00:02:11,800

 

40
00:02:11,800 --> 00:02:14,860

 

41
00:02:14,860 --> 00:02:18,640

 

42
00:02:18,640 --> 00:02:21,490

 

43
00:02:21,490 --> 00:02:25,840

 

44
00:02:25,840 --> 00:02:28,449

 

45
00:02:28,449 --> 00:02:30,939

 

46
00:02:30,939 --> 00:02:33,550

 

47
00:02:33,550 --> 00:02:36,030

 

48
00:02:36,030 --> 00:02:38,740

 

49
00:02:38,740 --> 00:02:41,830

 

50
00:02:41,830 --> 00:02:43,120

 

51
00:02:43,120 --> 00:02:46,949

 

52
00:02:46,949 --> 00:02:51,360

 

53
00:02:51,360 --> 00:02:55,840

 

54
00:02:55,840 --> 00:02:57,460

 

55
00:02:57,460 --> 00:03:04,400

 

56
00:03:04,400 --> 00:03:06,810

 

57
00:03:06,810 --> 00:03:06,820

 

58
00:03:06,820 --> 00:03:12,770


59
00:03:12,770 --> 00:03:12,780

 

60
00:03:12,780 --> 00:03:17,300


61
00:03:17,300 --> 00:03:26,580

 

62
00:03:26,580 --> 00:03:28,690

 

63
00:03:28,690 --> 00:03:33,069

 

64
00:03:33,069 --> 00:03:34,059

 

65
00:03:34,059 --> 00:03:36,699

 

66
00:03:36,699 --> 00:03:43,360

 

67
00:03:43,360 --> 00:03:45,039

 

68
00:03:45,039 --> 00:03:48,220

 

69
00:03:48,220 --> 00:03:52,330

 

70
00:03:52,330 --> 00:03:55,270

 

71
00:03:55,270 --> 00:03:58,780

 

72
00:03:58,780 --> 00:04:00,670

 

73
00:04:00,670 --> 00:04:03,729

 

74
00:04:03,729 --> 00:04:07,899

 

75
00:04:07,899 --> 00:04:10,449

 

76
00:04:10,449 --> 00:04:11,259

 

77
00:04:11,259 --> 00:04:13,990

 

78
00:04:13,990 --> 00:04:15,520

 

79
00:04:15,520 --> 00:04:18,550

 

80
00:04:18,550 --> 00:04:21,849

 

81
00:04:21,849 --> 00:04:24,249

 

82
00:04:24,249 --> 00:04:28,960

 

83
00:04:28,960 --> 00:04:31,089

 

84
00:04:31,089 --> 00:04:33,219

 

85
00:04:33,219 --> 00:04:37,180

 

86
00:04:37,180 --> 00:04:42,370

 

87
00:04:42,370 --> 00:04:45,330

 

88
00:04:45,330 --> 00:04:48,420

 

89
00:04:48,420 --> 00:04:54,850

 

90
00:04:54,850 --> 00:04:58,779

 

91
00:04:58,779 --> 00:05:05,529

 

92
00:05:05,529 --> 00:05:07,420

 

93
00:05:07,420 --> 00:05:13,800

 

94
00:05:13,800 --> 00:05:17,140

 

95
00:05:17,140 --> 00:05:21,159

 

96
00:05:21,159 --> 00:05:25,990

 

97
00:05:25,990 --> 00:05:29,020

 

98
00:05:29,020 --> 00:05:31,960

 

99
00:05:31,960 --> 00:05:40,730

 

100
00:05:40,730 --> 00:05:43,980

 

101
00:05:43,980 --> 00:05:47,670

 

102
00:05:47,670 --> 00:05:50,249

 

103
00:05:50,249 --> 00:05:51,890

 

104
00:05:51,890 --> 00:05:54,689

 

105
00:05:54,689 --> 00:05:57,809

 

106
00:05:57,809 --> 00:06:01,290

 

107
00:06:01,290 --> 00:06:08,969

 

108
00:06:08,969 --> 00:06:12,659

 

109
00:06:12,659 --> 00:06:15,089

 

110
00:06:15,089 --> 00:06:19,830

 

111
00:06:19,830 --> 00:06:22,679

 

112
00:06:22,679 --> 00:06:26,700

 

113
00:06:26,700 --> 00:06:29,459

 

114
00:06:29,459 --> 00:06:32,309

 

115
00:06:32,309 --> 00:06:35,339

 

116
00:06:35,339 --> 00:06:38,159

 

117
00:06:38,159 --> 00:06:42,360

 

118
00:06:42,360 --> 00:06:45,360

 

119
00:06:45,360 --> 00:06:49,499

 

120
00:06:49,499 --> 00:06:50,640

 

121
00:06:50,640 --> 00:06:52,559

 

122
00:06:52,559 --> 00:06:56,820

 

123
00:06:56,820 --> 00:06:58,980

 

124
00:06:58,980 --> 00:07:01,260

 

125
00:07:01,260 --> 00:07:02,519

 

126
00:07:02,519 --> 00:07:05,089

 

127
00:07:05,089 --> 00:07:07,910

 

128
00:07:07,910 --> 00:07:07,920

 

129
00:07:07,920 --> 00:07:11,580


130
00:07:11,580 --> 00:07:13,590

 

131
00:07:13,590 --> 00:07:16,830

 

132
00:07:16,830 --> 00:07:18,480

 

133
00:07:18,480 --> 00:07:21,150

 

134
00:07:21,150 --> 00:07:22,740

 

135
00:07:22,740 --> 00:07:27,500

 

136
00:07:27,500 --> 00:07:31,440

 

137
00:07:31,440 --> 00:07:33,750

 

138
00:07:33,750 --> 00:07:36,390

 

139
00:07:36,390 --> 00:07:38,250

 

140
00:07:38,250 --> 00:07:41,280

 

141
00:07:41,280 --> 00:07:45,140

 

142
00:07:45,140 --> 00:07:49,379

 

143
00:07:49,379 --> 00:07:54,210

 

144
00:07:54,210 --> 00:07:57,600

 

145
00:07:57,600 --> 00:08:00,020

 

146
00:08:00,020 --> 00:08:06,150

 

147
00:08:06,150 --> 00:08:11,340

 

148
00:08:11,340 --> 00:08:14,100

 

149
00:08:14,100 --> 00:08:16,469

 

150
00:08:16,469 --> 00:08:19,260

 

151
00:08:19,260 --> 00:08:21,330

 

152
00:08:21,330 --> 00:08:24,390

 

153
00:08:24,390 --> 00:08:27,590

 

154
00:08:27,590 --> 00:08:29,940

 

155
00:08:29,940 --> 00:08:33,450

 

156
00:08:33,450 --> 00:08:34,980

 

157
00:08:34,980 --> 00:08:39,839

 

158
00:08:39,839 --> 00:08:42,149

 

159
00:08:42,149 --> 00:08:43,890

 

160
00:08:43,890 --> 00:08:45,540

 

161
00:08:45,540 --> 00:08:47,130

 

162
00:08:47,130 --> 00:08:56,430

 

163
00:08:56,430 --> 00:08:58,960

 

164
00:08:58,960 --> 00:09:01,210

 

165
00:09:01,210 --> 00:09:03,700

 

166
00:09:03,700 --> 00:09:06,130

 

167
00:09:06,130 --> 00:09:09,730

 

168
00:09:09,730 --> 00:09:16,650

 

169
00:09:16,650 --> 00:09:18,910

 

170
00:09:18,910 --> 00:09:20,710

 

171
00:09:20,710 --> 00:09:24,970

 

172
00:09:24,970 --> 00:09:27,250

 

173
00:09:27,250 --> 00:09:28,480

 

174
00:09:28,480 --> 00:09:30,610

 

175
00:09:30,610 --> 00:09:35,380

 

176
00:09:35,380 --> 00:09:36,880

 

177
00:09:36,880 --> 00:09:39,250

 

178
00:09:39,250 --> 00:09:41,590

 

179
00:09:41,590 --> 00:09:43,690

 

180
00:09:43,690 --> 00:09:45,120

 

181
00:09:45,120 --> 00:09:48,190

 

182
00:09:48,190 --> 00:09:50,170

 

183
00:09:50,170 --> 00:09:52,180

 

184
00:09:52,180 --> 00:09:55,390

 

185
00:09:55,390 --> 00:09:58,730

 

186
00:09:58,730 --> 00:09:58,740

 

187
00:09:58,740 --> 00:10:03,570


188
00:10:03,570 --> 00:10:09,930

 

189
00:10:09,930 --> 00:10:12,000

 

190
00:10:12,000 --> 00:10:15,450

 

191
00:10:15,450 --> 00:10:17,700

 

192
00:10:17,700 --> 00:10:19,440

 

193
00:10:19,440 --> 00:10:21,930

 

194
00:10:21,930 --> 00:10:25,290

 

195
00:10:25,290 --> 00:10:27,210

 

196
00:10:27,210 --> 00:10:29,070

 

197
00:10:29,070 --> 00:10:30,810

 

198
00:10:30,810 --> 00:10:33,300

 

199
00:10:33,300 --> 00:10:34,710

 

200
00:10:34,710 --> 00:10:36,540

 

201
00:10:36,540 --> 00:10:36,550

 

202
00:10:36,550 --> 00:10:37,020


203
00:10:37,020 --> 00:10:41,160

 

204
00:10:41,160 --> 00:10:42,330

 

205
00:10:42,330 --> 00:10:45,090

 

206
00:10:45,090 --> 00:10:46,740

 

207
00:10:46,740 --> 00:10:49,260

 

208
00:10:49,260 --> 00:10:51,120

 

209
00:10:51,120 --> 00:10:53,670

 

210
00:10:53,670 --> 00:10:58,680

 

211
00:10:58,680 --> 00:11:00,540

 

212
00:11:00,540 --> 00:11:03,780

 

213
00:11:03,780 --> 00:11:06,170

 

214
00:11:06,170 --> 00:11:10,440

 

215
00:11:10,440 --> 00:11:13,710

 

216
00:11:13,710 --> 00:11:15,270

 

217
00:11:15,270 --> 00:11:16,830

 

218
00:11:16,830 --> 00:11:18,600

 

219
00:11:18,600 --> 00:11:22,500

 

220
00:11:22,500 --> 00:11:24,090

 

221
00:11:24,090 --> 00:11:26,850

 

222
00:11:26,850 --> 00:11:29,190

 

223
00:11:29,190 --> 00:11:34,140

 

224
00:11:34,140 --> 00:11:35,970

 

225
00:11:35,970 --> 00:11:40,050

 

226
00:11:40,050 --> 00:11:42,330

 

227
00:11:42,330 --> 00:11:46,230

 

228
00:11:46,230 --> 00:11:50,130

 

229
00:11:50,130 --> 00:11:51,720

 

230
00:11:51,720 --> 00:11:53,910

 

231
00:11:53,910 --> 00:11:58,230

 

232
00:11:58,230 --> 00:12:00,030

 

233
00:12:00,030 --> 00:12:01,680

 

234
00:12:01,680 --> 00:12:04,410

 

235
00:12:04,410 --> 00:12:10,560

 

236
00:12:10,560 --> 00:12:12,150

 

237
00:12:12,150 --> 00:12:13,590

 

238
00:12:13,590 --> 00:12:15,360

 

239
00:12:15,360 --> 00:12:15,370

 

240
00:12:15,370 --> 00:12:16,570


241
00:12:16,570 --> 00:12:18,250

 

242
00:12:18,250 --> 00:12:20,530

 

243
00:12:20,530 --> 00:12:20,540

 

244
00:12:20,540 --> 00:12:20,829


245
00:12:20,829 --> 00:12:22,180

 

246
00:12:22,180 --> 00:12:23,440

 

247
00:12:23,440 --> 00:12:26,079

 

248
00:12:26,079 --> 00:12:30,160

 

249
00:12:30,160 --> 00:12:33,160

 

250
00:12:33,160 --> 00:12:38,380

 

251
00:12:38,380 --> 00:12:41,430

 

252
00:12:41,430 --> 00:12:43,569

 

253
00:12:43,569 --> 00:12:46,810

 

254
00:12:46,810 --> 00:12:49,210

 

255
00:12:49,210 --> 00:12:51,579

 

256
00:12:51,579 --> 00:12:53,620

 

257
00:12:53,620 --> 00:12:54,550

 

258
00:12:54,550 --> 00:12:57,699

 

259
00:12:57,699 --> 00:12:59,170

 

260
00:12:59,170 --> 00:13:01,180

 

261
00:13:01,180 --> 00:13:03,430

 

262
00:13:03,430 --> 00:13:07,030

 

263
00:13:07,030 --> 00:13:08,769

 

264
00:13:08,769 --> 00:13:10,660

 

265
00:13:10,660 --> 00:13:13,960

 

266
00:13:13,960 --> 00:13:18,040

 

267
00:13:18,040 --> 00:13:19,600

 

268
00:13:19,600 --> 00:13:20,980

 

269
00:13:20,980 --> 00:13:22,870

 

270
00:13:22,870 --> 00:13:25,780

 

271
00:13:25,780 --> 00:13:29,199

 

272
00:13:29,199 --> 00:13:33,069

 

273
00:13:33,069 --> 00:13:35,139

 

274
00:13:35,139 --> 00:13:38,530

 

275
00:13:38,530 --> 00:13:42,449

 

276
00:13:42,449 --> 00:13:44,980

 

277
00:13:44,980 --> 00:13:53,319

 

278
00:13:53,319 --> 00:13:57,340

 

279
00:13:57,340 --> 00:13:58,569

 

280
00:13:58,569 --> 00:14:02,710

 

281
00:14:02,710 --> 00:14:05,439

 

282
00:14:05,439 --> 00:14:07,150

 

283
00:14:07,150 --> 00:14:08,079

 

284
00:14:08,079 --> 00:14:10,269

 

285
00:14:10,269 --> 00:14:12,310

 

286
00:14:12,310 --> 00:14:14,710

 

287
00:14:14,710 --> 00:14:17,410

 

288
00:14:17,410 --> 00:14:19,300

 

289
00:14:19,300 --> 00:14:21,490

 

290
00:14:21,490 --> 00:14:23,259

 

291
00:14:23,259 --> 00:14:26,110

 

292
00:14:26,110 --> 00:14:27,639

 

293
00:14:27,639 --> 00:14:29,710

 

294
00:14:29,710 --> 00:14:31,749

 

295
00:14:31,749 --> 00:14:35,860

 

296
00:14:35,860 --> 00:14:43,360

 

297
00:14:43,360 --> 00:14:45,730

 

298
00:14:45,730 --> 00:14:47,470

 

299
00:14:47,470 --> 00:14:50,470

 

300
00:14:50,470 --> 00:14:52,240

 

301
00:14:52,240 --> 00:14:54,879

 

302
00:14:54,879 --> 00:14:57,340

 

303
00:14:57,340 --> 00:15:02,590

 

304
00:15:02,590 --> 00:15:04,449

 

305
00:15:04,449 --> 00:15:10,179

 

306
00:15:10,179 --> 00:15:12,579

 

307
00:15:12,579 --> 00:15:16,300

 

308
00:15:16,300 --> 00:15:20,290

 

309
00:15:20,290 --> 00:15:25,030

 

310
00:15:25,030 --> 00:15:28,540

 

311
00:15:28,540 --> 00:15:31,840

 

312
00:15:31,840 --> 00:15:35,889

 

313
00:15:35,889 --> 00:15:38,790

 

314
00:15:38,790 --> 00:15:42,369

 

315
00:15:42,369 --> 00:15:44,740

 

316
00:15:44,740 --> 00:15:46,420

 

317
00:15:46,420 --> 00:15:50,470

 

318
00:15:50,470 --> 00:15:55,420

 

319
00:15:55,420 --> 00:15:59,280

 

320
00:15:59,280 --> 00:16:07,869

 

321
00:16:07,869 --> 00:16:10,900

 

322
00:16:10,900 --> 00:16:15,759

 

323
00:16:15,759 --> 00:16:19,379

 

324
00:16:19,379 --> 00:16:26,350

 

325
00:16:26,350 --> 00:16:28,540

 

326
00:16:28,540 --> 00:16:30,249

 

327
00:16:30,249 --> 00:16:32,619

 

328
00:16:32,619 --> 00:16:34,569

 

329
00:16:34,569 --> 00:16:36,280

 

330
00:16:36,280 --> 00:16:39,160

 

331
00:16:39,160 --> 00:16:43,720

 

332
00:16:43,720 --> 00:16:47,740

 

333
00:16:47,740 --> 00:16:50,980

 

334
00:16:50,980 --> 00:16:53,620

 

335
00:16:53,620 --> 00:16:55,570

 

336
00:16:55,570 --> 00:16:58,360

 

337
00:16:58,360 --> 00:17:04,000

 

338
00:17:04,000 --> 00:17:05,980

 

339
00:17:05,980 --> 00:17:08,949

 

340
00:17:08,949 --> 00:17:11,860

 

341
00:17:11,860 --> 00:17:14,710

 

342
00:17:14,710 --> 00:17:20,560

 

343
00:17:20,560 --> 00:17:22,329

 

344
00:17:22,329 --> 00:17:25,210

 

345
00:17:25,210 --> 00:17:30,100

 

346
00:17:30,100 --> 00:17:31,870

 

347
00:17:31,870 --> 00:17:34,990

 

348
00:17:34,990 --> 00:17:37,210

 

349
00:17:37,210 --> 00:17:39,370

 

350
00:17:39,370 --> 00:17:42,790

 

351
00:17:42,790 --> 00:17:46,330

 

352
00:17:46,330 --> 00:17:49,600

 

353
00:17:49,600 --> 00:17:52,390

 

354
00:17:52,390 --> 00:17:54,610

 

355
00:17:54,610 --> 00:17:56,460

 

356
00:17:56,460 --> 00:17:59,260

 

357
00:17:59,260 --> 00:18:02,920

 

358
00:18:02,920 --> 00:18:04,690

 

359
00:18:04,690 --> 00:18:07,230

 

360
00:18:07,230 --> 00:18:10,330

 

361
00:18:10,330 --> 00:18:12,460

 

362
00:18:12,460 --> 00:18:15,490

 

363
00:18:15,490 --> 00:18:18,390

 

364
00:18:18,390 --> 00:18:22,090

 

365
00:18:22,090 --> 00:18:25,720

 

366
00:18:25,720 --> 00:18:28,090

 

367
00:18:28,090 --> 00:18:30,190

 

368
00:18:30,190 --> 00:18:32,400

 

369
00:18:32,400 --> 00:18:34,930

 

370
00:18:34,930 --> 00:18:38,800

 

371
00:18:38,800 --> 00:18:42,160

 

372
00:18:42,160 --> 00:18:46,090

 

373
00:18:46,090 --> 00:18:48,430

 

374
00:18:48,430 --> 00:18:49,720

 

375
00:18:49,720 --> 00:18:51,550

 

376
00:18:51,550 --> 00:18:54,130

 

377
00:18:54,130 --> 00:19:01,380

 

378
00:19:01,380 --> 00:19:07,330

 

379
00:19:07,330 --> 00:19:10,060

 

380
00:19:10,060 --> 00:19:14,970

 

381
00:19:14,970 --> 00:19:18,100

 

382
00:19:18,100 --> 00:19:20,860

 

383
00:19:20,860 --> 00:19:23,530

 

384
00:19:23,530 --> 00:19:25,600

 

385
00:19:25,600 --> 00:19:29,560

 

386
00:19:29,560 --> 00:19:31,060

 

387
00:19:31,060 --> 00:19:32,980

 

388
00:19:32,980 --> 00:19:35,500

 

389
00:19:35,500 --> 00:19:37,510

 

390
00:19:37,510 --> 00:19:41,260

 

391
00:19:41,260 --> 00:19:44,530

 

392
00:19:44,530 --> 00:19:47,010

 

393
00:19:47,010 --> 00:19:50,350

 

394
00:19:50,350 --> 00:19:52,210

 

395
00:19:52,210 --> 00:19:55,840

 

396
00:19:55,840 --> 00:20:00,789

 

397
00:20:00,789 --> 00:20:03,460

 

398
00:20:03,460 --> 00:20:07,480

 

399
00:20:07,480 --> 00:20:12,669

 

400
00:20:12,669 --> 00:20:17,950

 

401
00:20:17,950 --> 00:20:19,900

 

402
00:20:19,900 --> 00:20:23,110

 

403
00:20:23,110 --> 00:20:25,419

 

404
00:20:25,419 --> 00:20:27,280

 

405
00:20:27,280 --> 00:20:29,799

 

406
00:20:29,799 --> 00:20:31,570

 

407
00:20:31,570 --> 00:20:34,270

 

408
00:20:34,270 --> 00:20:36,310

 

409
00:20:36,310 --> 00:20:38,260

 

410
00:20:38,260 --> 00:20:40,060

 

411
00:20:40,060 --> 00:20:44,049

 

412
00:20:44,049 --> 00:20:47,650

 

413
00:20:47,650 --> 00:20:49,690

 

414
00:20:49,690 --> 00:20:56,740

 

415
00:20:56,740 --> 00:21:00,520

 

416
00:21:00,520 --> 00:21:02,560

 

417
00:21:02,560 --> 00:21:04,210

 

418
00:21:04,210 --> 00:21:07,240

 

419
00:21:07,240 --> 00:21:07,900

 

420
00:21:07,900 --> 00:21:10,420

 

421
00:21:10,420 --> 00:21:13,300

 

422
00:21:13,300 --> 00:21:15,940

 

423
00:21:15,940 --> 00:21:17,770

 

424
00:21:17,770 --> 00:21:20,170

 

425
00:21:20,170 --> 00:21:22,810

 

426
00:21:22,810 --> 00:21:29,200

 

427
00:21:29,200 --> 00:21:32,500

 

428
00:21:32,500 --> 00:21:35,290

 

429
00:21:35,290 --> 00:21:38,590

 

430
00:21:38,590 --> 00:21:43,000

 

431
00:21:43,000 --> 00:21:44,920

 

432
00:21:44,920 --> 00:21:48,220

 

433
00:21:48,220 --> 00:21:51,760

 

434
00:21:51,760 --> 00:21:54,220

 

435
00:21:54,220 --> 00:21:56,170

 

436
00:21:56,170 --> 00:21:59,080

 

437
00:21:59,080 --> 00:22:01,120

 

438
00:22:01,120 --> 00:22:03,850

 

439
00:22:03,850 --> 00:22:07,660

 

440
00:22:07,660 --> 00:22:08,920

 

441
00:22:08,920 --> 00:22:10,150

 

442
00:22:10,150 --> 00:22:12,640

 

443
00:22:12,640 --> 00:22:16,510

 

444
00:22:16,510 --> 00:22:20,710

 

445
00:22:20,710 --> 00:22:23,560

 

446
00:22:23,560 --> 00:22:25,750

 

447
00:22:25,750 --> 00:22:28,630

 

448
00:22:28,630 --> 00:22:32,890

 

449
00:22:32,890 --> 00:22:35,520

 

450
00:22:35,520 --> 00:22:37,990

 

451
00:22:37,990 --> 00:22:39,970

 

452
00:22:39,970 --> 00:22:42,250

 

453
00:22:42,250 --> 00:22:43,690

 

454
00:22:43,690 --> 00:22:47,380

 

455
00:22:47,380 --> 00:22:50,560

 

456
00:22:50,560 --> 00:22:52,150

 

457
00:22:52,150 --> 00:22:53,680

 

458
00:22:53,680 --> 00:22:57,310

 

459
00:22:57,310 --> 00:23:01,180

 

460
00:23:01,180 --> 00:23:05,020

 

461
00:23:05,020 --> 00:23:09,250

 

462
00:23:09,250 --> 00:23:12,550

 

463
00:23:12,550 --> 00:23:13,990

 

464
00:23:13,990 --> 00:23:18,040

 

465
00:23:18,040 --> 00:23:19,550

 

466
00:23:19,550 --> 00:23:22,250

 

467
00:23:22,250 --> 00:23:24,050

 

468
00:23:24,050 --> 00:23:26,240

 

469
00:23:26,240 --> 00:23:31,250

 

470
00:23:31,250 --> 00:23:35,090

 

471
00:23:35,090 --> 00:23:37,670

 

472
00:23:37,670 --> 00:23:41,630

 

473
00:23:41,630 --> 00:23:44,510

 

474
00:23:44,510 --> 00:23:46,130

 

475
00:23:46,130 --> 00:23:50,030

 

476
00:23:50,030 --> 00:23:51,950

 

477
00:23:51,950 --> 00:23:54,710

 

478
00:23:54,710 --> 00:23:56,960

 

479
00:23:56,960 --> 00:23:59,180

 

480
00:23:59,180 --> 00:24:01,730

 

481
00:24:01,730 --> 00:24:05,030

 

482
00:24:05,030 --> 00:24:07,940

 

483
00:24:07,940 --> 00:24:10,310

 

484
00:24:10,310 --> 00:24:12,710

 

485
00:24:12,710 --> 00:24:14,540

 

486
00:24:14,540 --> 00:24:17,450

 

487
00:24:17,450 --> 00:24:25,700

 

488
00:24:25,700 --> 00:24:27,590

 

489
00:24:27,590 --> 00:24:30,710

 

490
00:24:30,710 --> 00:24:35,210

 

491
00:24:35,210 --> 00:24:37,870

 

492
00:24:37,870 --> 00:24:41,450

 

493
00:24:41,450 --> 00:24:48,640

 

494
00:24:48,640 --> 00:24:50,570

 

495
00:24:50,570 --> 00:24:56,810

 

496
00:24:56,810 --> 00:24:58,730

 

497
00:24:58,730 --> 00:25:01,070

 

498
00:25:01,070 --> 00:25:03,710

 

499
00:25:03,710 --> 00:25:05,120

 

500
00:25:05,120 --> 00:25:10,190

 

501
00:25:10,190 --> 00:25:23,540

 

502
00:25:23,540 --> 00:25:23,550

 

503
00:25:23,550 --> 00:25:26,490


504
00:25:26,490 --> 00:25:28,570

 

505
00:25:28,570 --> 00:25:38,820

 

506
00:25:38,820 --> 00:25:44,490

 

507
00:25:44,490 --> 00:25:50,150

 

508
00:25:50,150 --> 00:25:50,160

 

509
00:25:50,160 --> 00:25:51,300


510
00:25:51,300 --> 00:25:53,940

 

511
00:25:53,940 --> 00:26:00,060

 

512
00:26:00,060 --> 00:26:06,560

 

513
00:26:06,560 --> 00:26:11,250

 

514
00:26:11,250 --> 00:26:13,940

 

515
00:26:13,940 --> 00:26:15,930

 

516
00:26:15,930 --> 00:26:17,880

 

517
00:26:17,880 --> 00:26:19,860

 

518
00:26:19,860 --> 00:26:24,300

 

519
00:26:24,300 --> 00:26:26,880

 

520
00:26:26,880 --> 00:26:28,350

 

521
00:26:28,350 --> 00:26:32,730

 

522
00:26:32,730 --> 00:26:38,220

 

523
00:26:38,220 --> 00:26:41,040

 

524
00:26:41,040 --> 00:26:43,050

 

525
00:26:43,050 --> 00:26:49,230

 

526
00:26:49,230 --> 00:26:55,830

 

527
00:26:55,830 --> 00:26:57,600

 

528
00:26:57,600 --> 00:27:01,110

 

529
00:27:01,110 --> 00:27:03,930

 

530
00:27:03,930 --> 00:27:06,210

 

531
00:27:06,210 --> 00:27:08,910

 

532
00:27:08,910 --> 00:27:10,710

 

533
00:27:10,710 --> 00:27:12,030

 

534
00:27:12,030 --> 00:27:14,850

 

535
00:27:14,850 --> 00:27:16,260

 

536
00:27:16,260 --> 00:27:18,270

 

537
00:27:18,270 --> 00:27:20,850

 

538
00:27:20,850 --> 00:27:26,630

 

539
00:27:26,630 --> 00:27:30,690

 

540
00:27:30,690 --> 00:27:33,210

 

541
00:27:33,210 --> 00:27:35,730

 

542
00:27:35,730 --> 00:27:37,320

 

543
00:27:37,320 --> 00:27:39,660

 

544
00:27:39,660 --> 00:27:41,700

 

545
00:27:41,700 --> 00:27:44,390

 

546
00:27:44,390 --> 00:27:50,430

 

547
00:27:50,430 --> 00:27:54,120

 

548
00:27:54,120 --> 00:27:55,710

 

549
00:27:55,710 --> 00:27:57,570

 

550
00:27:57,570 --> 00:28:03,500

 

551
00:28:03,500 --> 00:28:07,110

 

552
00:28:07,110 --> 00:28:09,180

 

553
00:28:09,180 --> 00:28:13,080

 

554
00:28:13,080 --> 00:28:15,030

 

555
00:28:15,030 --> 00:28:18,300

 

556
00:28:18,300 --> 00:28:20,370

 

557
00:28:20,370 --> 00:28:22,470

 

558
00:28:22,470 --> 00:28:25,380

 

559
00:28:25,380 --> 00:28:27,510

 

560
00:28:27,510 --> 00:28:31,290

 

561
00:28:31,290 --> 00:28:34,320

 

562
00:28:34,320 --> 00:28:37,140

 

563
00:28:37,140 --> 00:28:38,430

 

564
00:28:38,430 --> 00:28:43,830

 

565
00:28:43,830 --> 00:28:46,530

 

566
00:28:46,530 --> 00:28:51,300

 

567
00:28:51,300 --> 00:28:53,670

 

568
00:28:53,670 --> 00:28:59,060

 

569
00:28:59,060 --> 00:29:03,720

 

570
00:29:03,720 --> 00:29:08,820

 

571
00:29:08,820 --> 00:29:12,180

 

572
00:29:12,180 --> 00:29:14,970

 

573
00:29:14,970 --> 00:29:18,480

 

574
00:29:18,480 --> 00:29:20,490

 

575
00:29:20,490 --> 00:29:23,040

 

576
00:29:23,040 --> 00:29:29,010

 

577
00:29:29,010 --> 00:29:30,600

 

578
00:29:30,600 --> 00:29:32,610

 

579
00:29:32,610 --> 00:29:35,780

 

580
00:29:35,780 --> 00:29:41,550

 

581
00:29:41,550 --> 00:29:44,280

 

582
00:29:44,280 --> 00:29:45,570

 

583
00:29:45,570 --> 00:29:46,590

 

584
00:29:46,590 --> 00:29:50,610

 

585
00:29:50,610 --> 00:29:52,590

 

586
00:29:52,590 --> 00:29:54,390

 

587
00:29:54,390 --> 00:29:57,780

 

588
00:29:57,780 --> 00:30:04,050

 

589
00:30:04,050 --> 00:30:08,820

 

590
00:30:08,820 --> 00:30:11,160

 

591
00:30:11,160 --> 00:30:13,289

 

592
00:30:13,289 --> 00:30:15,870

 

593
00:30:15,870 --> 00:30:19,169

 

594
00:30:19,169 --> 00:30:21,690

 

595
00:30:21,690 --> 00:30:23,820

 

596
00:30:23,820 --> 00:30:25,890

 

597
00:30:25,890 --> 00:30:30,299

 

598
00:30:30,299 --> 00:30:32,610

 

599
00:30:32,610 --> 00:30:34,080

 

600
00:30:34,080 --> 00:30:36,810

 

601
00:30:36,810 --> 00:30:40,950

 

602
00:30:40,950 --> 00:30:43,440

 

603
00:30:43,440 --> 00:30:46,080

 

604
00:30:46,080 --> 00:30:47,659

 

605
00:30:47,659 --> 00:30:54,570

 

606
00:30:54,570 --> 00:30:58,310

 

607
00:30:58,310 --> 00:31:00,720

 

608
00:31:00,720 --> 00:31:02,669

 

609
00:31:02,669 --> 00:31:02,679

 

610
00:31:02,679 --> 00:31:03,150


611
00:31:03,150 --> 00:31:07,590

 

612
00:31:07,590 --> 00:31:09,900

 

613
00:31:09,900 --> 00:31:11,190

 

614
00:31:11,190 --> 00:31:12,870

 

615
00:31:12,870 --> 00:31:14,640

 

616
00:31:14,640 --> 00:31:16,950

 

617
00:31:16,950 --> 00:31:20,039

 

618
00:31:20,039 --> 00:31:21,960

 

619
00:31:21,960 --> 00:31:24,570

 

620
00:31:24,570 --> 00:31:27,210

 

621
00:31:27,210 --> 00:31:29,549

 

622
00:31:29,549 --> 00:31:34,380

 

623
00:31:34,380 --> 00:31:36,480

 

624
00:31:36,480 --> 00:31:39,570

 

625
00:31:39,570 --> 00:31:41,220

 

626
00:31:41,220 --> 00:31:43,230

 

627
00:31:43,230 --> 00:31:44,760

 

628
00:31:44,760 --> 00:31:47,430

 

629
00:31:47,430 --> 00:31:49,590

 

630
00:31:49,590 --> 00:31:55,530

 

631
00:31:55,530 --> 00:31:57,630

 

632
00:31:57,630 --> 00:32:02,159

 

633
00:32:02,159 --> 00:32:03,870

 

634
00:32:03,870 --> 00:32:07,289

 

635
00:32:07,289 --> 00:32:10,620

 

636
00:32:10,620 --> 00:32:14,940

 

637
00:32:14,940 --> 00:32:14,950

 

638
00:32:14,950 --> 00:32:24,029


639
00:32:24,029 --> 00:32:26,680

 

640
00:32:26,680 --> 00:32:29,049

 

641
00:32:29,049 --> 00:32:39,039

 

642
00:32:39,039 --> 00:32:41,619

 

643
00:32:41,619 --> 00:32:49,649

 

644
00:32:49,649 --> 00:32:54,810

 

645
00:32:54,810 --> 00:32:58,930

 

646
00:32:58,930 --> 00:33:03,279

 

647
00:33:03,279 --> 00:33:05,649

 

648
00:33:05,649 --> 00:33:08,859

 

649
00:33:08,859 --> 00:33:12,249

 

650
00:33:12,249 --> 00:33:16,779

 

651
00:33:16,779 --> 00:33:19,980

 

652
00:33:19,980 --> 00:33:23,350

 

653
00:33:23,350 --> 00:33:25,570

 

654
00:33:25,570 --> 00:33:29,320

 

655
00:33:29,320 --> 00:33:30,999

 

656
00:33:30,999 --> 00:33:34,149

 

657
00:33:34,149 --> 00:33:37,960

 

658
00:33:37,960 --> 00:33:41,230

 

659
00:33:41,230 --> 00:33:44,019

 

660
00:33:44,019 --> 00:33:47,139

 

661
00:33:47,139 --> 00:33:50,169

 

662
00:33:50,169 --> 00:33:52,899

 

663
00:33:52,899 --> 00:33:54,249

 

664
00:33:54,249 --> 00:33:59,049

 

665
00:33:59,049 --> 00:34:00,820

 

666
00:34:00,820 --> 00:34:02,499

 

667
00:34:02,499 --> 00:34:04,570

 

668
00:34:04,570 --> 00:34:06,700

 

669
00:34:06,700 --> 00:34:08,770

 

670
00:34:08,770 --> 00:34:12,659

 

671
00:34:12,659 --> 00:34:14,619

 

672
00:34:14,619 --> 00:34:18,549

 

673
00:34:18,549 --> 00:34:20,260

 

674
00:34:20,260 --> 00:34:22,990

 

675
00:34:22,990 --> 00:34:25,149

 

676
00:34:25,149 --> 00:34:26,680

 

677
00:34:26,680 --> 00:34:28,899

 

678
00:34:28,899 --> 00:34:31,329

 

679
00:34:31,329 --> 00:34:32,950

 

680
00:34:32,950 --> 00:34:35,349

 

681
00:34:35,349 --> 00:34:40,300

 

682
00:34:40,300 --> 00:34:45,070

 

683
00:34:45,070 --> 00:34:48,099

 

684
00:34:48,099 --> 00:34:50,770

 

685
00:34:50,770 --> 00:34:55,659

 

686
00:34:55,659 --> 00:34:57,970

 

687
00:34:57,970 --> 00:35:00,460

 

688
00:35:00,460 --> 00:35:02,440

 

689
00:35:02,440 --> 00:35:04,210

 

690
00:35:04,210 --> 00:35:07,510

 

691
00:35:07,510 --> 00:35:10,570

 

692
00:35:10,570 --> 00:35:12,970

 

693
00:35:12,970 --> 00:35:15,070

 

694
00:35:15,070 --> 00:35:16,300

 

695
00:35:16,300 --> 00:35:19,480

 

696
00:35:19,480 --> 00:35:22,300

 

697
00:35:22,300 --> 00:35:24,400

 

698
00:35:24,400 --> 00:35:26,050

 

699
00:35:26,050 --> 00:35:28,540

 

700
00:35:28,540 --> 00:35:31,510

 

701
00:35:31,510 --> 00:35:36,010

 

702
00:35:36,010 --> 00:35:37,870

 

703
00:35:37,870 --> 00:35:39,790

 

704
00:35:39,790 --> 00:35:41,560

 

705
00:35:41,560 --> 00:35:43,420

 

706
00:35:43,420 --> 00:35:47,700

 

707
00:35:47,700 --> 00:35:51,430

 

708
00:35:51,430 --> 00:35:54,880

 

709
00:35:54,880 --> 00:35:56,980

 

710
00:35:56,980 --> 00:36:01,120

 

711
00:36:01,120 --> 00:36:04,780

 

712
00:36:04,780 --> 00:36:06,820

 

713
00:36:06,820 --> 00:36:10,470

 

714
00:36:10,470 --> 00:36:13,570

 

715
00:36:13,570 --> 00:36:16,990

 

716
00:36:16,990 --> 00:36:18,820

 

717
00:36:18,820 --> 00:36:20,950

 

718
00:36:20,950 --> 00:36:24,370

 

719
00:36:24,370 --> 00:36:25,750

 

720
00:36:25,750 --> 00:36:25,760

 

721
00:36:25,760 --> 00:36:26,320


722
00:36:26,320 --> 00:36:29,560

 

723
00:36:29,560 --> 00:36:33,220

 

724
00:36:33,220 --> 00:36:35,260

 

725
00:36:35,260 --> 00:36:36,190

 

726
00:36:36,190 --> 00:36:38,650

 

727
00:36:38,650 --> 00:36:42,190

 

728
00:36:42,190 --> 00:36:44,290

 

729
00:36:44,290 --> 00:36:45,340

 

730
00:36:45,340 --> 00:36:47,510

 

731
00:36:47,510 --> 00:36:49,280

 

732
00:36:49,280 --> 00:36:52,550

 

733
00:36:52,550 --> 00:36:57,380

 

734
00:36:57,380 --> 00:37:00,260

 

735
00:37:00,260 --> 00:37:03,530

 

736
00:37:03,530 --> 00:37:04,880

 

737
00:37:04,880 --> 00:37:04,890

 

738
00:37:04,890 --> 00:37:05,480


739
00:37:05,480 --> 00:37:09,980

 

740
00:37:09,980 --> 00:37:13,970

 

741
00:37:13,970 --> 00:37:15,080

 

742
00:37:15,080 --> 00:37:17,060

 

743
00:37:17,060 --> 00:37:22,130

 

744
00:37:22,130 --> 00:37:25,690

 

745
00:37:25,690 --> 00:37:27,890

 

746
00:37:27,890 --> 00:37:31,600

 

747
00:37:31,600 --> 00:37:36,860

 

748
00:37:36,860 --> 00:37:38,950

 

749
00:37:38,950 --> 00:37:44,480

 

750
00:37:44,480 --> 00:37:46,070

 

751
00:37:46,070 --> 00:37:48,980

 

752
00:37:48,980 --> 00:37:50,720

 

753
00:37:50,720 --> 00:37:54,620

 

754
00:37:54,620 --> 00:37:57,230

 

755
00:37:57,230 --> 00:38:02,600

 

756
00:38:02,600 --> 00:38:05,390

 

757
00:38:05,390 --> 00:38:07,610

 

758
00:38:07,610 --> 00:38:11,060

 

759
00:38:11,060 --> 00:38:13,640

 

760
00:38:13,640 --> 00:38:15,410

 

761
00:38:15,410 --> 00:38:20,730

 

762
00:38:20,730 --> 00:38:20,740

 

763
00:38:20,740 --> 00:38:22,080


764
00:38:22,080 --> 00:38:22,090

 

765
00:38:22,090 --> 00:38:24,650


766
00:38:24,650 --> 00:38:26,630

 

767
00:38:26,630 --> 00:38:30,170

 

768
00:38:30,170 --> 00:38:32,750

 

769
00:38:32,750 --> 00:38:35,809

 

770
00:38:35,809 --> 00:38:37,069

 

771
00:38:37,069 --> 00:38:38,780

 

772
00:38:38,780 --> 00:38:40,359

 

773
00:38:40,359 --> 00:38:45,140

 

774
00:38:45,140 --> 00:38:46,849

 

775
00:38:46,849 --> 00:38:48,589

 

776
00:38:48,589 --> 00:38:51,440

 

777
00:38:51,440 --> 00:38:54,400

 

778
00:38:54,400 --> 00:38:57,500

 

779
00:38:57,500 --> 00:39:00,760

 

780
00:39:00,760 --> 00:39:04,039

 

781
00:39:04,039 --> 00:39:07,760

 

782
00:39:07,760 --> 00:39:10,490

 

783
00:39:10,490 --> 00:39:12,170

 

784
00:39:12,170 --> 00:39:13,549

 

785
00:39:13,549 --> 00:39:15,319

 

786
00:39:15,319 --> 00:39:18,349

 

787
00:39:18,349 --> 00:39:20,750

 

788
00:39:20,750 --> 00:39:22,940

 

789
00:39:22,940 --> 00:39:27,440

 

790
00:39:27,440 --> 00:39:29,329

 

791
00:39:29,329 --> 00:39:33,740

 

792
00:39:33,740 --> 00:39:37,370

 

793
00:39:37,370 --> 00:39:39,380

 

794
00:39:39,380 --> 00:39:41,870

 

795
00:39:41,870 --> 00:39:45,770

 

796
00:39:45,770 --> 00:39:48,980

 

797
00:39:48,980 --> 00:39:52,720

 

798
00:39:52,720 --> 00:39:59,240

 

799
00:39:59,240 --> 00:40:01,039

 

800
00:40:01,039 --> 00:40:04,700

 

801
00:40:04,700 --> 00:40:10,250

 

802
00:40:10,250 --> 00:40:12,440

 

803
00:40:12,440 --> 00:40:15,799

 

804
00:40:15,799 --> 00:40:19,280

 

805
00:40:19,280 --> 00:40:21,920

 

806
00:40:21,920 --> 00:40:24,289

 

807
00:40:24,289 --> 00:40:28,039

 

808
00:40:28,039 --> 00:40:29,930

 

809
00:40:29,930 --> 00:40:32,240

 

810
00:40:32,240 --> 00:40:35,480

 

811
00:40:35,480 --> 00:40:35,490

 

812
00:40:35,490 --> 00:40:36,440


813
00:40:36,440 --> 00:40:38,150

 

814
00:40:38,150 --> 00:40:40,099

 

815
00:40:40,099 --> 00:40:42,470

 

816
00:40:42,470 --> 00:40:46,150

 

817
00:40:46,150 --> 00:40:56,150

 

818
00:40:56,150 --> 00:40:58,520

 

819
00:40:58,520 --> 00:41:00,410

 

820
00:41:00,410 --> 00:41:03,980

 

821
00:41:03,980 --> 00:41:05,480

 

822
00:41:05,480 --> 00:41:06,680

 

823
00:41:06,680 --> 00:41:08,540

 

824
00:41:08,540 --> 00:41:11,390

 

825
00:41:11,390 --> 00:41:15,620

 

826
00:41:15,620 --> 00:41:19,000

 

827
00:41:19,000 --> 00:41:24,980

 

828
00:41:24,980 --> 00:41:28,160

 

829
00:41:28,160 --> 00:41:35,330

 

830
00:41:35,330 --> 00:41:38,400

 

831
00:41:38,400 --> 00:41:40,230

 

832
00:41:40,230 --> 00:41:42,570

 

833
00:41:42,570 --> 00:41:46,890

 

834
00:41:46,890 --> 00:41:53,839

 

835
00:41:53,839 --> 00:42:01,770

 

836
00:42:01,770 --> 00:42:03,720

 

837
00:42:03,720 --> 00:42:05,010

 

838
00:42:05,010 --> 00:42:06,900

 

839
00:42:06,900 --> 00:42:09,510

 

840
00:42:09,510 --> 00:42:11,970

 

841
00:42:11,970 --> 00:42:19,140

 

842
00:42:19,140 --> 00:42:19,150

 

843
00:42:19,150 --> 00:42:19,770


844
00:42:19,770 --> 00:42:22,800

 

845
00:42:22,800 --> 00:42:25,200

 

846
00:42:25,200 --> 00:42:27,810

 

847
00:42:27,810 --> 00:42:29,460

 

848
00:42:29,460 --> 00:42:32,790

 

849
00:42:32,790 --> 00:42:36,270

 

850
00:42:36,270 --> 00:42:48,910

 

851
00:42:48,910 --> 00:42:48,920

 

852
00:42:48,920 --> 00:42:59,550


853
00:42:59,550 --> 00:43:02,380

 

854
00:43:02,380 --> 00:43:05,650

 

855
00:43:05,650 --> 00:43:08,170

 

856
00:43:08,170 --> 00:43:13,020

 

857
00:43:13,020 --> 00:43:16,000

 

858
00:43:16,000 --> 00:43:17,620

 

859
00:43:17,620 --> 00:43:21,400

 

860
00:43:21,400 --> 00:43:22,900

 

861
00:43:22,900 --> 00:43:25,380

 

862
00:43:25,380 --> 00:43:28,180

 

863
00:43:28,180 --> 00:43:29,770

 

864
00:43:29,770 --> 00:43:38,680

 

865
00:43:38,680 --> 00:43:38,690

 

866
00:43:38,690 --> 00:43:40,980


867
00:43:40,980 --> 00:43:44,040

 

868
00:43:44,040 --> 00:43:47,240

 

869
00:43:47,240 --> 00:43:50,910

 

870
00:43:50,910 --> 00:43:53,310

 

871
00:43:53,310 --> 00:43:55,260

 

872
00:43:55,260 --> 00:43:56,760

 

873
00:43:56,760 --> 00:43:58,650

 

874
00:43:58,650 --> 00:44:00,240

 

875
00:44:00,240 --> 00:44:02,670

 

876
00:44:02,670 --> 00:44:04,370

 

877
00:44:04,370 --> 00:44:07,079

 

878
00:44:07,079 --> 00:44:09,030

 

879
00:44:09,030 --> 00:44:12,390

 

880
00:44:12,390 --> 00:44:14,579

 

881
00:44:14,579 --> 00:44:17,040

 

882
00:44:17,040 --> 00:44:20,310

 

883
00:44:20,310 --> 00:44:22,190

 

884
00:44:22,190 --> 00:44:25,140

 

885
00:44:25,140 --> 00:44:27,329

 

886
00:44:27,329 --> 00:44:29,990

 

887
00:44:29,990 --> 00:44:35,910

 

888
00:44:35,910 --> 00:44:39,510

 

889
00:44:39,510 --> 00:44:41,700

 

890
00:44:41,700 --> 00:44:43,260

 

891
00:44:43,260 --> 00:44:44,849

 

892
00:44:44,849 --> 00:44:47,640

 

893
00:44:47,640 --> 00:44:50,849

 

894
00:44:50,849 --> 00:44:53,609

 

895
00:44:53,609 --> 00:44:59,099

 

896
00:44:59,099 --> 00:45:01,410

 

897
00:45:01,410 --> 00:45:03,900

 

898
00:45:03,900 --> 00:45:05,730

 

899
00:45:05,730 --> 00:45:09,930

 

900
00:45:09,930 --> 00:45:17,180

 

901
00:45:17,180 --> 00:45:22,290

 

902
00:45:22,290 --> 00:45:24,950

 

903
00:45:24,950 --> 00:45:27,450

 

904
00:45:27,450 --> 00:45:33,000

 

905
00:45:33,000 --> 00:45:35,230

 

906
00:45:35,230 --> 00:45:37,090

 

907
00:45:37,090 --> 00:45:41,140

 

908
00:45:41,140 --> 00:45:43,450

 

909
00:45:43,450 --> 00:45:43,460

 

910
00:45:43,460 --> 00:45:48,040


911
00:45:48,040 --> 00:45:48,050

 

912
00:45:48,050 --> 00:45:50,100


913
00:45:50,100 --> 00:45:54,630

 

914
00:45:54,630 --> 00:45:59,070

 

915
00:45:59,070 --> 00:45:59,080

 

916
00:45:59,080 --> 00:45:59,430


917
00:45:59,430 --> 00:46:02,370

 

918
00:46:02,370 --> 00:46:04,710

 

919
00:46:04,710 --> 00:46:08,190

 

920
00:46:08,190 --> 00:46:10,190

 

921
00:46:10,190 --> 00:46:14,160

 

922
00:46:14,160 --> 00:46:16,290

 

923
00:46:16,290 --> 00:46:18,210

 

924
00:46:18,210 --> 00:46:22,110

 

925
00:46:22,110 --> 00:46:24,210

 

926
00:46:24,210 --> 00:46:26,460

 

927
00:46:26,460 --> 00:46:31,940

 

928
00:46:31,940 --> 00:46:34,200

 

929
00:46:34,200 --> 00:46:35,700

 

930
00:46:35,700 --> 00:46:40,910

 

931
00:46:40,910 --> 00:46:43,800

 

932
00:46:43,800 --> 00:46:46,740

 

933
00:46:46,740 --> 00:46:49,680

 

934
00:46:49,680 --> 00:46:49,690

 

935
00:46:49,690 --> 00:46:50,570


936
00:46:50,570 --> 00:46:56,280

 

937
00:46:56,280 --> 00:46:58,130

 

938
00:46:58,130 --> 00:47:01,410

 

939
00:47:01,410 --> 00:47:04,350

 

940
00:47:04,350 --> 00:47:08,280

 

941
00:47:08,280 --> 00:47:13,620

 

942
00:47:13,620 --> 00:47:15,210

 

943
00:47:15,210 --> 00:47:20,370

 

944
00:47:20,370 --> 00:47:21,810

 

945
00:47:21,810 --> 00:47:23,700

 

946
00:47:23,700 --> 00:47:25,650

 

947
00:47:25,650 --> 00:47:28,950

 

948
00:47:28,950 --> 00:47:32,160

 

949
00:47:32,160 --> 00:47:35,520

 

950
00:47:35,520 --> 00:47:41,100

 

951
00:47:41,100 --> 00:47:42,470

 

952
00:47:42,470 --> 00:47:44,340

 

953
00:47:44,340 --> 00:47:49,080

 

954
00:47:49,080 --> 00:47:52,230

 

955
00:47:52,230 --> 00:47:54,390

 

956
00:47:54,390 --> 00:47:57,480

 

957
00:47:57,480 --> 00:48:01,980

 

958
00:48:01,980 --> 00:48:06,720

 

959
00:48:06,720 --> 00:48:18,200

 

960
00:48:18,200 --> 00:48:18,210

 

961
00:48:18,210 --> 00:48:20,109


962
00:48:20,109 --> 00:48:23,160

 

963
00:48:23,160 --> 00:48:26,960

 

964
00:48:26,960 --> 00:48:32,640

 

965
00:48:32,640 --> 00:48:34,250

 

966
00:48:34,250 --> 00:48:38,430

 

967
00:48:38,430 --> 00:48:39,750

 

968
00:48:39,750 --> 00:48:42,120

 

969
00:48:42,120 --> 00:48:45,270

 

970
00:48:45,270 --> 00:48:46,800

 

971
00:48:46,800 --> 00:48:49,260

 

972
00:48:49,260 --> 00:48:57,290

 

973
00:48:57,290 --> 00:49:08,720

 

974
00:49:08,720 --> 00:49:11,250

 

975
00:49:11,250 --> 00:49:13,620

 

976
00:49:13,620 --> 00:49:16,620

 

977
00:49:16,620 --> 00:49:17,970

 

978
00:49:17,970 --> 00:49:20,640

 

979
00:49:20,640 --> 00:49:28,820

 

980
00:49:28,820 --> 00:49:31,320

 

981
00:49:31,320 --> 00:49:33,390

 

982
00:49:33,390 --> 00:49:37,710

 

983
00:49:37,710 --> 00:49:39,570

 

984
00:49:39,570 --> 00:49:48,120

 

985
00:49:48,120 --> 00:49:51,100

 

986
00:49:51,100 --> 00:49:55,800

 

987
00:49:55,800 --> 00:50:06,140

 

988
00:50:06,140 --> 00:50:06,150

 

989
00:50:06,150 --> 00:50:10,180


990
00:50:10,180 --> 00:50:20,640

 

991
00:50:20,640 --> 00:50:24,940

 

992
00:50:24,940 --> 00:50:27,160

 

993
00:50:27,160 --> 00:50:29,680

 

994
00:50:29,680 --> 00:50:33,549

 

995
00:50:33,549 --> 00:50:37,120

 

996
00:50:37,120 --> 00:50:39,400

 

997
00:50:39,400 --> 00:50:41,200

 

998
00:50:41,200 --> 00:50:43,870

 

999
00:50:43,870 --> 00:50:45,190

 

1000
00:50:45,190 --> 00:50:47,309

 

1001
00:50:47,309 --> 00:50:49,720

 

1002
00:50:49,720 --> 00:50:53,890

 

1003
00:50:53,890 --> 00:50:57,460

 

1004
00:50:57,460 --> 00:51:01,019

 

1005
00:51:01,019 --> 00:51:04,240

 

1006
00:51:04,240 --> 00:51:10,079

 

1007
00:51:10,079 --> 00:51:14,910

 

1008
00:51:14,910 --> 00:51:18,710

 

1009
00:51:18,710 --> 00:51:21,690

 

1010
00:51:21,690 --> 00:51:24,089

 

1011
00:51:24,089 --> 00:51:28,109

 

1012
00:51:28,109 --> 00:51:31,319

 

1013
00:51:31,319 --> 00:51:32,940

 

1014
00:51:32,940 --> 00:51:34,980

 

1015
00:51:34,980 --> 00:51:38,160

 

1016
00:51:38,160 --> 00:51:41,990

 

1017
00:51:41,990 --> 00:51:43,980

 

1018
00:51:43,980 --> 00:51:45,660

 

1019
00:51:45,660 --> 00:51:49,170

 

1020
00:51:49,170 --> 00:51:52,620

 

1021
00:51:52,620 --> 00:51:56,790

 

1022
00:51:56,790 --> 00:51:59,960

 

1023
00:51:59,960 --> 00:52:04,050

 

1024
00:52:04,050 --> 00:52:06,420

 

1025
00:52:06,420 --> 00:52:08,640

 

1026
00:52:08,640 --> 00:52:15,349

 

1027
00:52:15,349 --> 00:52:17,579

 

1028
00:52:17,579 --> 00:52:19,829

 

1029
00:52:19,829 --> 00:52:21,420

 

1030
00:52:21,420 --> 00:52:23,819

 

1031
00:52:23,819 --> 00:52:25,170

 

1032
00:52:25,170 --> 00:52:26,970

 

1033
00:52:26,970 --> 00:52:29,069

 

1034
00:52:29,069 --> 00:52:31,800

 

1035
00:52:31,800 --> 00:52:34,650

 

1036
00:52:34,650 --> 00:52:36,180

 

1037
00:52:36,180 --> 00:52:39,420

 

1038
00:52:39,420 --> 00:52:41,220

 

1039
00:52:41,220 --> 00:52:45,540

 

1040
00:52:45,540 --> 00:52:47,630

 

1041
00:52:47,630 --> 00:52:50,099

 

1042
00:52:50,099 --> 00:52:51,480

 

1043
00:52:51,480 --> 00:52:53,760

 

1044
00:52:53,760 --> 00:52:57,690

 

1045
00:52:57,690 --> 00:53:00,420

 

1046
00:53:00,420 --> 00:53:00,430

 

1047
00:53:00,430 --> 00:53:05,710


1048
00:53:05,710 --> 00:53:11,650

 

1049
00:53:11,650 --> 00:53:15,310

 

1050
00:53:15,310 --> 00:53:20,680

 

1051
00:53:20,680 --> 00:53:25,690

 

1052
00:53:25,690 --> 00:53:28,270

 

1053
00:53:28,270 --> 00:53:31,420

 

1054
00:53:31,420 --> 00:53:33,490

 

1055
00:53:33,490 --> 00:53:42,610

 

1056
00:53:42,610 --> 00:53:46,600

 

1057
00:53:46,600 --> 00:53:46,610

 

1058
00:53:46,610 --> 00:53:51,010


1059
00:53:51,010 --> 00:53:53,810

 

1060
00:53:53,810 --> 00:53:55,940

 

1061
00:53:55,940 --> 00:53:59,840

 

1062
00:53:59,840 --> 00:54:04,760

 

1063
00:54:04,760 --> 00:54:08,060

 

1064
00:54:08,060 --> 00:54:10,460

 

1065
00:54:10,460 --> 00:54:12,350

 

1066
00:54:12,350 --> 00:54:14,990

 

1067
00:54:14,990 --> 00:54:17,900

 

1068
00:54:17,900 --> 00:54:20,930

 

1069
00:54:20,930 --> 00:54:26,000

 

1070
00:54:26,000 --> 00:54:27,890

 

1071
00:54:27,890 --> 00:54:30,140

 

1072
00:54:30,140 --> 00:54:33,590

 

1073
00:54:33,590 --> 00:54:37,640

 

1074
00:54:37,640 --> 00:54:40,160

 

1075
00:54:40,160 --> 00:54:44,270

 

1076
00:54:44,270 --> 00:54:45,980

 

1077
00:54:45,980 --> 00:54:47,450

 

1078
00:54:47,450 --> 00:54:50,780

 

1079
00:54:50,780 --> 00:54:53,420

 

1080
00:54:53,420 --> 00:54:55,130

 

1081
00:54:55,130 --> 00:54:57,830

 

1082
00:54:57,830 --> 00:55:02,920

 

1083
00:55:02,920 --> 00:55:05,810

 

1084
00:55:05,810 --> 00:55:10,070

 

1085
00:55:10,070 --> 00:55:12,710

 

1086
00:55:12,710 --> 00:55:14,660

 

1087
00:55:14,660 --> 00:55:17,210

 

1088
00:55:17,210 --> 00:55:19,070

 

1089
00:55:19,070 --> 00:55:21,380

 

1090
00:55:21,380 --> 00:55:22,940

 

1091
00:55:22,940 --> 00:55:24,260

 

1092
00:55:24,260 --> 00:55:26,600

 

1093
00:55:26,600 --> 00:55:29,810

 

1094
00:55:29,810 --> 00:55:32,750

 

1095
00:55:32,750 --> 00:55:35,570

 

1096
00:55:35,570 --> 00:55:37,610

 

1097
00:55:37,610 --> 00:55:42,070

 

1098
00:55:42,070 --> 00:55:45,140

 

1099
00:55:45,140 --> 00:55:47,390

 

1100
00:55:47,390 --> 00:55:50,840

 

1101
00:55:50,840 --> 00:55:53,630

 

1102
00:55:53,630 --> 00:55:58,940

 

1103
00:55:58,940 --> 00:56:00,590

 

1104
00:56:00,590 --> 00:56:02,060

 

1105
00:56:02,060 --> 00:56:02,070

 

1106
00:56:02,070 --> 00:56:02,770


1107
00:56:02,770 --> 00:56:09,030

 

1108
00:56:09,030 --> 00:56:15,480

 

1109
00:56:15,480 --> 00:56:18,960

 

1110
00:56:18,960 --> 00:56:22,230

 

1111
00:56:22,230 --> 00:56:25,140

 

1112
00:56:25,140 --> 00:56:29,510

 

1113
00:56:29,510 --> 00:56:33,210

 

1114
00:56:33,210 --> 00:56:35,460

 

1115
00:56:35,460 --> 00:56:40,170

 

1116
00:56:40,170 --> 00:56:42,090

 

1117
00:56:42,090 --> 00:56:44,250

 

1118
00:56:44,250 --> 00:56:47,610

 

1119
00:56:47,610 --> 00:56:49,410

 

1120
00:56:49,410 --> 00:56:51,570

 

1121
00:56:51,570 --> 00:56:56,640

 

1122
00:56:56,640 --> 00:56:59,670

 

1123
00:56:59,670 --> 00:57:02,340

 

1124
00:57:02,340 --> 00:57:04,860

 

1125
00:57:04,860 --> 00:57:08,220

 

1126
00:57:08,220 --> 00:57:10,250

 

1127
00:57:10,250 --> 00:57:13,590

 

1128
00:57:13,590 --> 00:57:16,620

 

1129
00:57:16,620 --> 00:57:21,600

 

1130
00:57:21,600 --> 00:57:25,110

 

1131
00:57:25,110 --> 00:57:26,820

 

1132
00:57:26,820 --> 00:57:33,000

 

1133
00:57:33,000 --> 00:57:35,400

 

1134
00:57:35,400 --> 00:57:36,960

 

1135
00:57:36,960 --> 00:57:38,610

 

1136
00:57:38,610 --> 00:57:40,980

 

1137
00:57:40,980 --> 00:57:42,300

 

1138
00:57:42,300 --> 00:57:44,670

 

1139
00:57:44,670 --> 00:57:46,860

 

1140
00:57:46,860 --> 00:57:53,190

 

1141
00:57:53,190 --> 00:57:57,930

 

1142
00:57:57,930 --> 00:58:05,739

 

1143
00:58:05,739 --> 00:58:09,180

 

1144
00:58:09,180 --> 00:58:11,799

 

1145
00:58:11,799 --> 00:58:14,979

 

1146
00:58:14,979 --> 00:58:18,009

 

1147
00:58:18,009 --> 00:58:19,989

 

1148
00:58:19,989 --> 00:58:22,349

 

1149
00:58:22,349 --> 00:58:25,239

 

1150
00:58:25,239 --> 00:58:26,920

 

1151
00:58:26,920 --> 00:58:28,809

 

1152
00:58:28,809 --> 00:58:29,859

 

1153
00:58:29,859 --> 00:58:32,229

 

1154
00:58:32,229 --> 00:58:33,870

 

1155
00:58:33,870 --> 00:58:36,190

 

1156
00:58:36,190 --> 00:58:38,160

 

1157
00:58:38,160 --> 00:58:41,380

 

1158
00:58:41,380 --> 00:58:43,539

 

1159
00:58:43,539 --> 00:58:45,160

 

1160
00:58:45,160 --> 00:58:47,559

 

1161
00:58:47,559 --> 00:58:49,019

 

1162
00:58:49,019 --> 00:58:51,819

 

1163
00:58:51,819 --> 00:58:53,410

 

1164
00:58:53,410 --> 00:58:56,079

 

1165
00:58:56,079 --> 00:58:58,089

 

1166
00:58:58,089 --> 00:59:00,519

 

1167
00:59:00,519 --> 00:59:02,739

 

1168
00:59:02,739 --> 00:59:06,009

 

1169
00:59:06,009 --> 00:59:08,229

 

1170
00:59:08,229 --> 00:59:16,079

 

1171
00:59:16,079 --> 00:59:18,700

 

1172
00:59:18,700 --> 00:59:22,239

 

1173
00:59:22,239 --> 00:59:26,109

 

1174
00:59:26,109 --> 00:59:29,319

 

1175
00:59:29,319 --> 00:59:30,549

 

1176
00:59:30,549 --> 00:59:34,059

 

1177
00:59:34,059 --> 00:59:35,920

 

1178
00:59:35,920 --> 00:59:39,430

 

1179
00:59:39,430 --> 00:59:42,039

 

1180
00:59:42,039 --> 00:59:45,339

 

1181
00:59:45,339 --> 00:59:48,390

 

1182
00:59:48,390 --> 00:59:51,039

 

1183
00:59:51,039 --> 00:59:54,400

 

1184
00:59:54,400 --> 00:59:56,319

 

1185
00:59:56,319 --> 00:59:59,469

 

1186
00:59:59,469 --> 01:00:02,670

 

1187
01:00:02,670 --> 01:00:09,440

 

1188
01:00:09,440 --> 01:00:12,499

 

1189
01:00:12,499 --> 01:00:14,479

 

1190
01:00:14,479 --> 01:00:17,479

 

1191
01:00:17,479 --> 01:00:19,729

 

1192
01:00:19,729 --> 01:00:22,460

 

1193
01:00:22,460 --> 01:00:25,279

 

1194
01:00:25,279 --> 01:00:29,089

 

1195
01:00:29,089 --> 01:00:31,370

 

1196
01:00:31,370 --> 01:00:34,729

 

1197
01:00:34,729 --> 01:00:38,059

 

1198
01:00:38,059 --> 01:00:39,650

 

1199
01:00:39,650 --> 01:00:42,140

 

1200
01:00:42,140 --> 01:00:45,880

 

1201
01:00:45,880 --> 01:00:48,170

 

1202
01:00:48,170 --> 01:00:49,880

 

1203
01:00:49,880 --> 01:00:51,559

 

1204
01:00:51,559 --> 01:00:54,200

 

1205
01:00:54,200 --> 01:00:56,930

 

1206
01:00:56,930 --> 01:00:59,660

 

1207
01:00:59,660 --> 01:01:03,859

 

1208
01:01:03,859 --> 01:01:06,259

 

1209
01:01:06,259 --> 01:01:08,029

 

1210
01:01:08,029 --> 01:01:10,309

 

1211
01:01:10,309 --> 01:01:11,779

 

1212
01:01:11,779 --> 01:01:15,739

 

1213
01:01:15,739 --> 01:01:19,249

 

1214
01:01:19,249 --> 01:01:21,799

 

1215
01:01:21,799 --> 01:01:23,420

 

1216
01:01:23,420 --> 01:01:26,630

 

1217
01:01:26,630 --> 01:01:28,839

 

1218
01:01:28,839 --> 01:01:32,329

 

1219
01:01:32,329 --> 01:01:33,950

 

1220
01:01:33,950 --> 01:01:35,329

 

1221
01:01:35,329 --> 01:01:38,539

 

1222
01:01:38,539 --> 01:01:40,579

 

1223
01:01:40,579 --> 01:01:44,900

 

1224
01:01:44,900 --> 01:01:46,519

 

1225
01:01:46,519 --> 01:01:48,650

 

1226
01:01:48,650 --> 01:01:51,019

 

1227
01:01:51,019 --> 01:01:55,400

 

1228
01:01:55,400 --> 01:01:58,910

 

1229
01:01:58,910 --> 01:02:01,579

 

1230
01:02:01,579 --> 01:02:04,160

 

1231
01:02:04,160 --> 01:02:09,309

 

1232
01:02:09,309 --> 01:02:17,120

 

1233
01:02:17,120 --> 01:02:19,640

 

1234
01:02:19,640 --> 01:02:23,150

 

1235
01:02:23,150 --> 01:02:26,540

 

1236
01:02:26,540 --> 01:02:28,130

 

1237
01:02:28,130 --> 01:02:30,140

 

1238
01:02:30,140 --> 01:02:34,250

 

1239
01:02:34,250 --> 01:02:35,870

 

1240
01:02:35,870 --> 01:02:38,120

 

1241
01:02:38,120 --> 01:02:40,790

 

1242
01:02:40,790 --> 01:02:42,560

 

1243
01:02:42,560 --> 01:02:45,260

 

1244
01:02:45,260 --> 01:02:49,550

 

1245
01:02:49,550 --> 01:02:53,930

 

1246
01:02:53,930 --> 01:02:56,180

 

1247
01:02:56,180 --> 01:02:58,010

 

1248
01:02:58,010 --> 01:03:00,920

 

1249
01:03:00,920 --> 01:03:03,410

 

1250
01:03:03,410 --> 01:03:05,180

 

1251
01:03:05,180 --> 01:03:07,520

 

1252
01:03:07,520 --> 01:03:09,740

 

1253
01:03:09,740 --> 01:03:12,020

 

1254
01:03:12,020 --> 01:03:14,600

 

1255
01:03:14,600 --> 01:03:16,940

 

1256
01:03:16,940 --> 01:03:19,310

 

1257
01:03:19,310 --> 01:03:23,900

 

1258
01:03:23,900 --> 01:03:27,380

 

1259
01:03:27,380 --> 01:03:30,500

 

1260
01:03:30,500 --> 01:03:35,840

 

1261
01:03:35,840 --> 01:03:37,520

 

1262
01:03:37,520 --> 01:03:42,500

 

1263
01:03:42,500 --> 01:03:50,750

 

1264
01:03:50,750 --> 01:03:52,700

 

1265
01:03:52,700 --> 01:03:58,130

 

1266
01:03:58,130 --> 01:04:01,610

 

1267
01:04:01,610 --> 01:04:03,410

 

1268
01:04:03,410 --> 01:04:06,710

 

1269
01:04:06,710 --> 01:04:08,510

 

1270
01:04:08,510 --> 01:04:11,840

 

1271
01:04:11,840 --> 01:04:13,880

 

1272
01:04:13,880 --> 01:04:16,910

 

1273
01:04:16,910 --> 01:04:19,490

 

1274
01:04:19,490 --> 01:04:23,410

 

1275
01:04:23,410 --> 01:04:25,480

 

1276
01:04:25,480 --> 01:04:27,069

 

1277
01:04:27,069 --> 01:04:31,450

 

1278
01:04:31,450 --> 01:04:33,849

 

1279
01:04:33,849 --> 01:04:35,799

 

1280
01:04:35,799 --> 01:04:38,170

 

1281
01:04:38,170 --> 01:04:48,960

 

1282
01:04:48,960 --> 01:04:51,480

 

1283
01:04:51,480 --> 01:04:53,760

 

1284
01:04:53,760 --> 01:04:57,480

 

1285
01:04:57,480 --> 01:04:59,609

 

1286
01:04:59,609 --> 01:05:02,420

 

1287
01:05:02,420 --> 01:05:09,150

 

1288
01:05:09,150 --> 01:05:15,900

 

1289
01:05:15,900 --> 01:05:18,210

 

1290
01:05:18,210 --> 01:05:20,099

 

1291
01:05:20,099 --> 01:05:21,750

 

1292
01:05:21,750 --> 01:05:24,120

 

1293
01:05:24,120 --> 01:05:26,010

 

1294
01:05:26,010 --> 01:05:28,230

 

1295
01:05:28,230 --> 01:05:30,540

 

1296
01:05:30,540 --> 01:05:34,829

 

1297
01:05:34,829 --> 01:05:36,930

 

1298
01:05:36,930 --> 01:05:38,700

 

1299
01:05:38,700 --> 01:05:40,260

 

1300
01:05:40,260 --> 01:05:44,040

 

1301
01:05:44,040 --> 01:05:47,520

 

1302
01:05:47,520 --> 01:05:50,579

 

1303
01:05:50,579 --> 01:05:52,230

 

1304
01:05:52,230 --> 01:05:54,390

 

1305
01:05:54,390 --> 01:05:56,579

 

1306
01:05:56,579 --> 01:05:59,700

 

1307
01:05:59,700 --> 01:06:01,650

 

1308
01:06:01,650 --> 01:06:04,380

 

1309
01:06:04,380 --> 01:06:05,819

 

1310
01:06:05,819 --> 01:06:08,550

 

1311
01:06:08,550 --> 01:06:11,370

 

1312
01:06:11,370 --> 01:06:13,400

 

1313
01:06:13,400 --> 01:06:15,930

 

1314
01:06:15,930 --> 01:06:18,960

 

1315
01:06:18,960 --> 01:06:21,510

 

1316
01:06:21,510 --> 01:06:25,530

 

1317
01:06:25,530 --> 01:06:29,280

 

1318
01:06:29,280 --> 01:06:32,309

 

1319
01:06:32,309 --> 01:06:35,280

 

1320
01:06:35,280 --> 01:06:39,390

 

1321
01:06:39,390 --> 01:06:41,510

 

1322
01:06:41,510 --> 01:06:46,890

 

1323
01:06:46,890 --> 01:06:49,950

 

1324
01:06:49,950 --> 01:06:51,900

 

1325
01:06:51,900 --> 01:06:54,150

 

1326
01:06:54,150 --> 01:06:56,370

 

1327
01:06:56,370 --> 01:06:57,830

 

1328
01:06:57,830 --> 01:07:00,380

 

1329
01:07:00,380 --> 01:07:03,290

 

1330
01:07:03,290 --> 01:07:06,860

 

1331
01:07:06,860 --> 01:07:08,060

 

1332
01:07:08,060 --> 01:07:09,470

 

1333
01:07:09,470 --> 01:07:12,170

 

1334
01:07:12,170 --> 01:07:13,910

 

1335
01:07:13,910 --> 01:07:17,000

 

1336
01:07:17,000 --> 01:07:18,530

 

1337
01:07:18,530 --> 01:07:19,670

 

1338
01:07:19,670 --> 01:07:21,110

 

1339
01:07:21,110 --> 01:07:23,930

 

1340
01:07:23,930 --> 01:07:28,640

 

1341
01:07:28,640 --> 01:07:31,820

 

1342
01:07:31,820 --> 01:07:34,490

 

1343
01:07:34,490 --> 01:07:37,210

 

1344
01:07:37,210 --> 01:07:40,880

 

1345
01:07:40,880 --> 01:07:45,050

 

1346
01:07:45,050 --> 01:07:48,260

 

1347
01:07:48,260 --> 01:07:51,200

 

1348
01:07:51,200 --> 01:07:54,140

 

1349
01:07:54,140 --> 01:07:56,720

 

1350
01:07:56,720 --> 01:07:58,430

 

1351
01:07:58,430 --> 01:08:02,110

 

1352
01:08:02,110 --> 01:08:05,870

 

1353
01:08:05,870 --> 01:08:07,370

 

1354
01:08:07,370 --> 01:08:09,590

 

1355
01:08:09,590 --> 01:08:11,570

 

1356
01:08:11,570 --> 01:08:13,070

 

1357
01:08:13,070 --> 01:08:19,190

 

1358
01:08:19,190 --> 01:08:21,770

 

1359
01:08:21,770 --> 01:08:23,780

 

1360
01:08:23,780 --> 01:08:26,840

 

1361
01:08:26,840 --> 01:08:30,230

 

1362
01:08:30,230 --> 01:08:33,020

 

1363
01:08:33,020 --> 01:08:34,700

 

1364
01:08:34,700 --> 01:08:36,770

 

1365
01:08:36,770 --> 01:08:38,780

 

1366
01:08:38,780 --> 01:08:41,900

 

1367
01:08:41,900 --> 01:08:43,820

 

1368
01:08:43,820 --> 01:08:45,410

 

1369
01:08:45,410 --> 01:08:49,760

 

1370
01:08:49,760 --> 01:08:51,980

 

1371
01:08:51,980 --> 01:08:56,110

 

1372
01:08:56,110 --> 01:08:56,120

 

1373
01:08:56,120 --> 01:08:57,290


1374
01:08:57,290 --> 01:08:59,900

 

1375
01:08:59,900 --> 01:09:03,860

 

1376
01:09:03,860 --> 01:09:08,720

 

1377
01:09:08,720 --> 01:09:11,780

 

1378
01:09:11,780 --> 01:09:13,370

 

1379
01:09:13,370 --> 01:09:15,920

 

1380
01:09:15,920 --> 01:09:18,590

 

1381
01:09:18,590 --> 01:09:22,309

 

1382
01:09:22,309 --> 01:09:26,559

 

1383
01:09:26,559 --> 01:09:29,090

 

1384
01:09:29,090 --> 01:09:31,099

 

1385
01:09:31,099 --> 01:09:34,309

 

1386
01:09:34,309 --> 01:09:37,030

 

1387
01:09:37,030 --> 01:09:39,950

 

1388
01:09:39,950 --> 01:09:41,870

 

1389
01:09:41,870 --> 01:09:45,890

 

1390
01:09:45,890 --> 01:09:49,760

 

1391
01:09:49,760 --> 01:09:53,480

 

1392
01:09:53,480 --> 01:09:55,130

 

1393
01:09:55,130 --> 01:09:57,440

 

1394
01:09:57,440 --> 01:10:02,600

 

1395
01:10:02,600 --> 01:10:05,750

 

1396
01:10:05,750 --> 01:10:11,570

 

1397
01:10:11,570 --> 01:10:13,190

 

1398
01:10:13,190 --> 01:10:17,260

 

1399
01:10:17,260 --> 01:10:20,630

 

1400
01:10:20,630 --> 01:10:23,030

 

1401
01:10:23,030 --> 01:10:25,640

 

1402
01:10:25,640 --> 01:10:27,770

 

1403
01:10:27,770 --> 01:10:30,200

 

1404
01:10:30,200 --> 01:10:32,420

 

1405
01:10:32,420 --> 01:10:35,150

 

1406
01:10:35,150 --> 01:10:37,640

 

1407
01:10:37,640 --> 01:10:39,770

 

1408
01:10:39,770 --> 01:10:41,810

 

1409
01:10:41,810 --> 01:10:43,280

 

1410
01:10:43,280 --> 01:10:45,320

 

1411
01:10:45,320 --> 01:10:47,780

 

1412
01:10:47,780 --> 01:10:49,760

 

1413
01:10:49,760 --> 01:10:52,760

 

1414
01:10:52,760 --> 01:10:55,310

 

1415
01:10:55,310 --> 01:10:57,230

 

1416
01:10:57,230 --> 01:10:59,660

 

1417
01:10:59,660 --> 01:11:02,330

 

1418
01:11:02,330 --> 01:11:05,420

 

1419
01:11:05,420 --> 01:11:07,700

 

1420
01:11:07,700 --> 01:11:09,580

 

1421
01:11:09,580 --> 01:11:13,460

 

1422
01:11:13,460 --> 01:11:16,430

 

1423
01:11:16,430 --> 01:11:19,790

 

1424
01:11:19,790 --> 01:11:22,310

 

1425
01:11:22,310 --> 01:11:23,600

 

1426
01:11:23,600 --> 01:11:25,690

 

1427
01:11:25,690 --> 01:11:30,160

 

1428
01:11:30,160 --> 01:11:32,500

 

1429
01:11:32,500 --> 01:11:35,520

 

1430
01:11:35,520 --> 01:11:38,610

 

1431
01:11:38,610 --> 01:11:41,050

 

1432
01:11:41,050 --> 01:11:43,270

 

1433
01:11:43,270 --> 01:11:45,790

 

1434
01:11:45,790 --> 01:11:48,010

 

1435
01:11:48,010 --> 01:11:51,280

 

1436
01:11:51,280 --> 01:11:52,750

 

1437
01:11:52,750 --> 01:11:55,600

 

1438
01:11:55,600 --> 01:11:57,250

 

1439
01:11:57,250 --> 01:11:59,670

 

1440
01:11:59,670 --> 01:12:03,730

 

1441
01:12:03,730 --> 01:12:05,530

 

1442
01:12:05,530 --> 01:12:10,840

 

1443
01:12:10,840 --> 01:12:14,650

 

1444
01:12:14,650 --> 01:12:16,330

 

1445
01:12:16,330 --> 01:12:18,700

 

1446
01:12:18,700 --> 01:12:20,950

 

1447
01:12:20,950 --> 01:12:22,000

 

1448
01:12:22,000 --> 01:12:25,000

 

1449
01:12:25,000 --> 01:12:27,730

 

1450
01:12:27,730 --> 01:12:30,040

 

1451
01:12:30,040 --> 01:12:31,390

 

1452
01:12:31,390 --> 01:12:34,240

 

1453
01:12:34,240 --> 01:12:36,100

 

1454
01:12:36,100 --> 01:12:38,880

 

1455
01:12:38,880 --> 01:12:38,890

 

1456
01:12:38,890 --> 01:12:42,250


1457
01:12:42,250 --> 01:12:45,590

 

1458
01:12:45,590 --> 01:12:49,760

 

1459
01:12:49,760 --> 01:12:49,770

 

1460
01:12:49,770 --> 01:12:50,870


1461
01:12:50,870 --> 01:12:54,170

 

1462
01:12:54,170 --> 01:12:58,670

 

1463
01:12:58,670 --> 01:13:01,010

 

1464
01:13:01,010 --> 01:13:03,260

 

1465
01:13:03,260 --> 01:13:07,070

 

1466
01:13:07,070 --> 01:13:09,140

 

1467
01:13:09,140 --> 01:13:10,760

 

1468
01:13:10,760 --> 01:13:12,650

 

1469
01:13:12,650 --> 01:13:14,270

 

1470
01:13:14,270 --> 01:13:17,030

 

1471
01:13:17,030 --> 01:13:20,660

 

1472
01:13:20,660 --> 01:13:24,430

 

1473
01:13:24,430 --> 01:13:26,810

 

1474
01:13:26,810 --> 01:13:29,720

 

1475
01:13:29,720 --> 01:13:32,410

 

1476
01:13:32,410 --> 01:13:39,940

 

1477
01:13:39,940 --> 01:13:42,650

 

1478
01:13:42,650 --> 01:13:45,020

 

1479
01:13:45,020 --> 01:13:46,460

 

1480
01:13:46,460 --> 01:13:49,490

 

1481
01:13:49,490 --> 01:13:51,350

 

1482
01:13:51,350 --> 01:13:56,440

 

1483
01:13:56,440 --> 01:14:02,030

 

1484
01:14:02,030 --> 01:14:06,020

 

1485
01:14:06,020 --> 01:14:08,300

 

1486
01:14:08,300 --> 01:14:10,250

 

1487
01:14:10,250 --> 01:14:12,410

 

1488
01:14:12,410 --> 01:14:15,800

 

1489
01:14:15,800 --> 01:14:19,310

 

1490
01:14:19,310 --> 01:14:21,920

 

1491
01:14:21,920 --> 01:14:24,050

 

1492
01:14:24,050 --> 01:14:26,240

 

1493
01:14:26,240 --> 01:14:30,770

 

1494
01:14:30,770 --> 01:14:37,700

 

1495
01:14:37,700 --> 01:14:41,480

 

1496
01:14:41,480 --> 01:14:43,100

 

1497
01:14:43,100 --> 01:14:44,980

 

1498
01:14:44,980 --> 01:14:47,330

 

1499
01:14:47,330 --> 01:14:51,620

 

1500
01:14:51,620 --> 01:14:53,960

 

1501
01:14:53,960 --> 01:14:53,970

 

1502
01:14:53,970 --> 01:14:54,229


1503
01:14:54,229 --> 01:14:56,839

 

1504
01:14:56,839 --> 01:14:58,729

 

1505
01:14:58,729 --> 01:15:02,109

 

1506
01:15:02,109 --> 01:15:04,520

 

1507
01:15:04,520 --> 01:15:07,759

 

1508
01:15:07,759 --> 01:15:10,939

 

1509
01:15:10,939 --> 01:15:12,799

 

1510
01:15:12,799 --> 01:15:15,549

 

1511
01:15:15,549 --> 01:15:17,479

 

1512
01:15:17,479 --> 01:15:18,919

 

1513
01:15:18,919 --> 01:15:21,890

 

1514
01:15:21,890 --> 01:15:23,720

 

1515
01:15:23,720 --> 01:15:25,489

 

1516
01:15:25,489 --> 01:15:29,000

 

1517
01:15:29,000 --> 01:15:31,549

 

1518
01:15:31,549 --> 01:15:34,100

 

1519
01:15:34,100 --> 01:15:35,839

 

1520
01:15:35,839 --> 01:15:39,350

 

1521
01:15:39,350 --> 01:15:41,600

 

1522
01:15:41,600 --> 01:15:44,180

 

1523
01:15:44,180 --> 01:15:46,069

 

1524
01:15:46,069 --> 01:15:48,770

 

1525
01:15:48,770 --> 01:15:50,390

 

1526
01:15:50,390 --> 01:15:53,089

 

1527
01:15:53,089 --> 01:15:54,950

 

1528
01:15:54,950 --> 01:15:57,529

 

1529
01:15:57,529 --> 01:15:59,419

 

1530
01:15:59,419 --> 01:16:01,640

 

1531
01:16:01,640 --> 01:16:03,739

 

1532
01:16:03,739 --> 01:16:06,109

 

1533
01:16:06,109 --> 01:16:10,669

 

1534
01:16:10,669 --> 01:16:12,229

 

1535
01:16:12,229 --> 01:16:15,950

 

1536
01:16:15,950 --> 01:16:19,520

 

1537
01:16:19,520 --> 01:16:21,859

 

1538
01:16:21,859 --> 01:16:23,239

 

1539
01:16:23,239 --> 01:16:24,649

 

1540
01:16:24,649 --> 01:16:27,109

 

1541
01:16:27,109 --> 01:16:28,459

 

1542
01:16:28,459 --> 01:16:31,339

 

1543
01:16:31,339 --> 01:16:32,839

 

1544
01:16:32,839 --> 01:16:38,450

 

1545
01:16:38,450 --> 01:16:39,830

 

1546
01:16:39,830 --> 01:16:43,609

 

1547
01:16:43,609 --> 01:16:45,350

 

1548
01:16:45,350 --> 01:16:48,410

 

1549
01:16:48,410 --> 01:16:50,419

 

1550
01:16:50,419 --> 01:16:52,759

 

1551
01:16:52,759 --> 01:16:56,660

 

1552
01:16:56,660 --> 01:16:58,879

 

1553
01:16:58,879 --> 01:17:01,459

 

1554
01:17:01,459 --> 01:17:03,709

 

1555
01:17:03,709 --> 01:17:05,530

 

1556
01:17:05,530 --> 01:17:08,560

 

1557
01:17:08,560 --> 01:17:10,150

 

1558
01:17:10,150 --> 01:17:13,330

 

1559
01:17:13,330 --> 01:17:15,580

 

1560
01:17:15,580 --> 01:17:18,610

 

1561
01:17:18,610 --> 01:17:20,860

 

1562
01:17:20,860 --> 01:17:23,530

 

1563
01:17:23,530 --> 01:17:27,670

 

1564
01:17:27,670 --> 01:17:30,310

 

1565
01:17:30,310 --> 01:17:32,020

 

1566
01:17:32,020 --> 01:17:33,670

 

1567
01:17:33,670 --> 01:17:36,670

 

1568
01:17:36,670 --> 01:17:39,580

 

1569
01:17:39,580 --> 01:17:41,560

 

1570
01:17:41,560 --> 01:17:43,240

 

1571
01:17:43,240 --> 01:17:44,470

 

1572
01:17:44,470 --> 01:17:47,950

 

1573
01:17:47,950 --> 01:17:52,660

 

1574
01:17:52,660 --> 01:17:54,580

 

1575
01:17:54,580 --> 01:17:57,430

 

1576
01:17:57,430 --> 01:17:58,840

 

1577
01:17:58,840 --> 01:18:00,400

 

1578
01:18:00,400 --> 01:18:02,770

 

1579
01:18:02,770 --> 01:18:04,990

 

1580
01:18:04,990 --> 01:18:08,350

 

1581
01:18:08,350 --> 01:18:10,150

 

1582
01:18:10,150 --> 01:18:12,940

 

1583
01:18:12,940 --> 01:18:16,540

 

1584
01:18:16,540 --> 01:18:18,850

 

1585
01:18:18,850 --> 01:18:21,100

 

1586
01:18:21,100 --> 01:18:24,670

 

1587
01:18:24,670 --> 01:18:27,550

 

1588
01:18:27,550 --> 01:18:29,710

 

1589
01:18:29,710 --> 01:18:32,110

 

1590
01:18:32,110 --> 01:18:33,670

 

1591
01:18:33,670 --> 01:18:36,880

 

1592
01:18:36,880 --> 01:18:39,100

 

1593
01:18:39,100 --> 01:18:41,170

 

1594
01:18:41,170 --> 01:18:45,430

 

1595
01:18:45,430 --> 01:18:46,150

 

1596
01:18:46,150 --> 01:18:48,160

 

1597
01:18:48,160 --> 01:18:51,400

 

1598
01:18:51,400 --> 01:18:54,480

 

1599
01:18:54,480 --> 01:19:01,350

 

1600
01:19:01,350 --> 01:19:05,920

 

1601
01:19:05,920 --> 01:19:08,860

 

1602
01:19:08,860 --> 01:19:11,140

 

1603
01:19:11,140 --> 01:19:13,810

 

1604
01:19:13,810 --> 01:19:18,400

 

1605
01:19:18,400 --> 01:19:19,250

 

1606
01:19:19,250 --> 01:19:23,810

 

1607
01:19:23,810 --> 01:19:25,130

 

1608
01:19:25,130 --> 01:19:27,260

 

1609
01:19:27,260 --> 01:19:28,970

 

1610
01:19:28,970 --> 01:19:31,160

 

1611
01:19:31,160 --> 01:19:33,890

 

1612
01:19:33,890 --> 01:19:36,229

 

1613
01:19:36,229 --> 01:19:37,640

 

1614
01:19:37,640 --> 01:19:41,500

 

1615
01:19:41,500 --> 01:19:44,330

 

1616
01:19:44,330 --> 01:19:48,080

 

1617
01:19:48,080 --> 01:19:50,090

 

1618
01:19:50,090 --> 01:19:51,920

 

1619
01:19:51,920 --> 01:19:54,110

 

1620
01:19:54,110 --> 01:19:57,470

 

1621
01:19:57,470 --> 01:20:00,910

 

1622
01:20:00,910 --> 01:20:02,870

 

1623
01:20:02,870 --> 01:20:04,550

 

1624
01:20:04,550 --> 01:20:07,760

 

1625
01:20:07,760 --> 01:20:10,130

 

1626
01:20:10,130 --> 01:20:12,620

 

1627
01:20:12,620 --> 01:20:14,270

 

1628
01:20:14,270 --> 01:20:15,920

 

1629
01:20:15,920 --> 01:20:18,979

 

1630
01:20:18,979 --> 01:20:22,310

 

1631
01:20:22,310 --> 01:20:25,280

 

1632
01:20:25,280 --> 01:20:27,860

 

1633
01:20:27,860 --> 01:20:29,360

 

1634
01:20:29,360 --> 01:20:30,979

 

1635
01:20:30,979 --> 01:20:33,200

 

1636
01:20:33,200 --> 01:20:35,300

 

1637
01:20:35,300 --> 01:20:37,970

 

1638
01:20:37,970 --> 01:20:39,950

 

1639
01:20:39,950 --> 01:20:45,050

 

1640
01:20:45,050 --> 01:20:47,720

 

1641
01:20:47,720 --> 01:20:49,250

 

1642
01:20:49,250 --> 01:20:53,590

 

1643
01:20:53,590 --> 01:20:57,590

 

1644
01:20:57,590 --> 01:20:59,900

 

1645
01:20:59,900 --> 01:21:02,060

 

1646
01:21:02,060 --> 01:21:04,790

 

1647
01:21:04,790 --> 01:21:07,550

 

1648
01:21:07,550 --> 01:21:09,140

 

1649
01:21:09,140 --> 01:21:09,150

 

1650
01:21:09,150 --> 01:21:09,890


1651
01:21:09,890 --> 01:21:12,760

 

1652
01:21:12,760 --> 01:21:16,190

 

1653
01:21:16,190 --> 01:21:20,060

 

1654
01:21:20,060 --> 01:21:22,160

 

1655
01:21:22,160 --> 01:21:25,010

 

1656
01:21:25,010 --> 01:21:27,979

 

1657
01:21:27,979 --> 01:21:32,089

 

1658
01:21:32,089 --> 01:21:35,510

 

1659
01:21:35,510 --> 01:21:39,560

 

1660
01:21:39,560 --> 01:21:41,660

 

1661
01:21:41,660 --> 01:21:44,330

 

1662
01:21:44,330 --> 01:21:49,160

 

1663
01:21:49,160 --> 01:21:50,750

 

1664
01:21:50,750 --> 01:21:53,649

 

1665
01:21:53,649 --> 01:21:56,810

 

1666
01:21:56,810 --> 01:21:58,280

 

1667
01:21:58,280 --> 01:22:00,589

 

1668
01:22:00,589 --> 01:22:03,020

 

1669
01:22:03,020 --> 01:22:05,120

 

1670
01:22:05,120 --> 01:22:07,700

 

1671
01:22:07,700 --> 01:22:12,560