1
00:00:01,069 --> 00:00:03,189

the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  okay let's get started on Thursday we  are really privileged to have John  Bentley who is one of the Masters of  performance engineering come and give us  a guest lecture in 1982 he wrote this  wonderful little book from which we  adapted the Bentley rules that you folks  have seen and he's also famous for  things like KD trees who's seen KD trees  yeah so he invented KD trees and who's  ever used the master method of the  current saw he invented the master  method of recurrence solving and so I  mean he's just a wonderful wonderful  fellow with lots and lots of insights  and just like a superior performance  engineer and he's going to talk give a  guest lecture on Thursday and so I would  encourage you to bring your friends  bring friends maybe who dropped the  class okay because this is like this is  the and and others who you may who you  know if any of you have urops other  people in your research group whether  there are graduate students or whatever  they're all welcome to come because this  is really you know one of the  opportunities you get to to actually  meet one of the you know legends of the  field if you will and you know have a  and you'll see he's very approachable  he's very approachable so um so anyway  my advertisement for John Bentley good  so let's start I want to talk about  speculative parallelism for a little bit  because that's kind of what you need to  make the code go fast so if you look at  code like alpha beta it turns out that  it's inherently serial okay that is if  you don't you know if you try to do  things in parallel then you miss the  cut-offs and if you miss the cut-offs  you know the beta cut-offs then  you start doing work that you wouldn't  necessarily need to do and so the only  way to sort of make this type of code  run fast is to use speculation which is  to say that you're gonna guess that you  can do stuff in parallel and it'll be  worthwhile and then occasionally you're  going to be wrong but the trick is well  how do you minimize the wasted effort  when things go wrong so let's just take  a look at a very simple example of of  speculative parallelism which is Thresh  holding a sum so here we have a little  program that is returning a boolean it  takes as input an array of size n so  threshold in sums you have an array a of  length n and a limit and these are all  unsigned integers say and I start out  the summit zero and what I'm gonna do is  add up all the elements of the array and  if it exceeds the limit then I'm gonna  return okay and so um how might I  optimize this code a little bit yeah  no let's say on one one processor on one  processor what might you do yeah once  you check so that once you would exceed  it  why keep adding the other numbers okay  so here's a code that does that quit  early if the partial product ever sees  the threshold this isn't necessarily an  optimization it was noticed that now in  the optimization not only do I have you  know a memory reference and in addition  I also now have a you know I also have a  unpredictable branch actually it's  predictable branch sorry predictable  branch but it's still one more one more  step because it's always going to  predict that it's not exceeding until it  actually does exceed so that'll be  pretty predictable and but still I've  added something into the loop so that  maybe so how might I mitigate that  problem so I don't want to add too much  into the inner loop yeah  [Music]  yeah so basically I can do what's called  strip mining I replace the loop of n  iterations with a loop of let's say n  over four iterations with an inner loop  of four iterations and every fourth  iteration I check to see whether or not  I've exceeded okay so that the cost of  the the check is going to be you know  relatively small at that point okay so  there are ways of of making this sort of  go fast okay so now we want to make this  operate in parallel okay and the problem  in when I operate in parallel is that is  that as I'm adding stuff up I want to do  it so I'm going to do this here with a  reducer and so basically I'm going to  add up the values in the reducer but now  I'd like to do the same thing of being  able to quit early and so the question  is well how can we paralyze a short  circuited loop okay how can we and so  the way I'm going to do this and this is  sort of by hand here but it's going to  give you so actually as you know  underneath the loop is really a divide  and conquer loop and so you know I could  I could write it as a as a parallel loop  and now it becomes I think a little bit  clearer how I could in this case what  I'm going to do is is return the value  of the sum and now the question is well  how can I quit early and save the work  if I exceed the threshold okay  understanding that you know when I'm  executing this in something like silk  you know it's going to tend to you know  go down one path and often it's going to  be a serial execution so I'd like if  it's if it's possible so here's here's  one way of doing it which is then I  add a and abort flag to the code and  which is going to say whether I should  keep going or whether I have actually  exceeded at that point and so I'm going  to use that recursively and now I take a  look at for each time through the loop  or through the divide and conquer I look  to see whether the sum is greater than a  limit and I haven't you know already  aborted the flag then I set the abort  flag to be true okay  why do I bother testing the abort flag  before I set it to true so notice that  that setting the abort flag is a race  isn't it a determinacy race okay because  I have great thank you it's because I  have the the stuff operating in parallel  is all trying to set the abort flag  whenever something abort so I can have  two guys who are in parallel setting the  abort flag but if they're setting it  they're setting it to the same value so  it's a benign race okay assuming your  memory model is such that you can  actually set the values so so what  happens here when why do i why do you  bother checking this  so what happens when several guys in  parallel want to write to the same  variable this is quiz one stuff yeah  yeah you can have it bouncing along the  cache it will serialize it right because  because if they're all trying to to  write then they have to get exclusive  access for it to write and modify it and  so that happens boom boom boom one at a  time  and so by checking it first it can be in  a shared state and then only one guy has  to one guy clobbers it and then it'll  update all the other ones so it makes it  so we don't continually have a true  sharing race and then in the checking to  see if it exceeds we basically just  check to see if you know we basically  call this function we set the abort flag  to false and then we return the sum of  all the values and if it aborts then it  just returns early okay so it doesn't  have to keep going through the  computation and of course once again you  can coerce in the leaves and so forth so  so this is non-deterministic code right  which we should never write unless we  have to okay because that's the only way  of getting performance and you have to  make sure you of course you reset the  abort flag after the use and then  actually do you need a memory fence here  on the abort flag do you need to set you  know where where would you put a an  abort flag to make sure that the value  was  yeah yes or what what would you have to  do  [Music]  okay so so indeed it turns out you don't  need a memory fence here and the reason  is because the code is correct whether  it's the initial value force or whether  it's become true so it doesn't matter  when it becomes true and so there's an  issue of if you put in the fence then  you know that it becomes true earlier  okay rather than waiting for the  convocation but that may actually slow  things down because memory fences are  expensive but because the transition is  always from false to true during the  execution you don't actually need a  memory fence there okay in order to make  sure that you've got the correct value  does that make sense so you don't need a  memory fence so so that's a classic  instance of speculative parallelism it  occurs when a program spawns some  parallel work that might not be  performed in a serial execution so  you're performing it anticipating that  you're probably going to need to do it  okay and then typically what you want to  do is have some way of backing out if  you discover that that you've made made  it you know that you didn't need to do  that way have some way of making sure  that you don't do any more okay so basic  rule of thumb of speculation is don't  spawn speculative work unless there's  little other opportunity for parallelism  and there's a good chance that will be  needed so one of the things I've seen in  research papers no less is people who  say oh I'm going to you know I have a  calculation where I'm trying to find say  the minimum of a bunch of values and so  I'm gonna you know spawn off you know  end things each of which is looking for  the minimum as soon as the minimum one  comes back I'm going to retract all the  other computations and I'm gonna get  super linear speed-up that way because  because in the serial execution I might  have done the longer one first and maybe  the minimum is not the not the first one  or whatever and so they they claim super  linear speed-up by doing speculative  parallelism but that's not really that's  not really a good example because there  was a better serial code if that was  really a good way of doing it they  should have been doing what's called  dovetailing which is they doing a little  bit of this computation a little bit of  this a little bit of this a little bit  of this etc and then going back that  would have been a better algorithm for  which you would then use it and the risk  they have is that they're spawning off  and things of which most of them may not  be needed and now they don't get any  speed up even though they've just used a  lot more work and that's often is is a  bad bad choice so usually you don't want  to spawn speculative work unless there's  little other opportunity and there's a  good chance it'll be needed okay  my experience is you know how much what  kind of chance do you need you need to  you know have certainly less than you  know for parallelism you've got to have  if if the chance that you're going to  need the work if you have P processors  if the chance that it could be not  needed is is less than actually have a  theorem at the end which I'll which I'll  I'll refer you to ok because a little  bit hard to say because I didn't put on  the slide but there's there's a final  slide for this which gives a nice little  theorem about when you should do this so  now let's talk about parallel alpha beta  search because that's kind of what  you're doing with your principle  variation search and so if you remember  the analysis done by Knuth and Morris  they basically showed that for gram tree  with branching factor B and depth D and  alpha beta search with move searched in  best first order examines that exactly B  to the  d over 2 plus B to the floor of D over 2  minus 1 nodes at ply D so that's  basically square routing the amount of  work that you would need if you did a  full depth fly that's with alpha beta ok  so naive algorithm looks at B to the D  nodes at depth D and so for the same  work the search depth is effectively  doubled or the work is Square rooted ok  so that we saw last time in Helens  lecture ok so the key optimization here  is pruning the game tree and as I  mentioned the beginning of this when you  prune the game tree you've got what's  effectively a serial thing and if you if  you let something go ahead you may be  working on something that would be  pruned so then how does the parallelism  really help you there you're just  wasting processors it could be better  spent perhaps elsewhere except where  else can you spend it because there's no  other parallelism in the in the code so  we want to find some solution so the  solution comes for a from an idea from a  very nice paper by Burkhardt manin and  some of his students in Germany and they  made the observation that in a best  ordered tree the degree of every node is  either either 1 or maximal this is this  is not their observation this is  actually the observation in the knuth  paper ok so when you have a best ordered  tree if you can get all the moves  ordered in their true order ok then it  turns out that either you're you're  exploring all the children or you are  refuting something and you get a cut off  from the very first one that you look at  ok and so in this case for example it  turns out on the principle variation  those are all full  with sorry they have you have to explore  all the children and then from there on  it alternates one level it's it you know  it you have just a single child that  needs to be explored and when you  explore it if it's best ordered you will  get a beta cutoff for the for the others  and then it alternates and then it's  full width' and so their idea was to say  well if the first child fails to  generate a beta cutoff speculate that in  fact you have the best one and the  remaining children can be searched in  parallel without wasting any work okay  so if it fails you're gonna say oh I'm  gonna speculate that this is in fact a  full-width thing now in practice you  don't necessarily get things best  ordered but they're a bunch of  heuristics in the code we've given you  and in in chess code to make it so that  you tend to do things in the in the  proper order the most important of those  is have you if you've seen the move  before okay and it's in the  transposition table you use the move  that you've seen before even if it's to  a shallower depth okay you guess that  that's going to be their best move still  even if you search deeper and that's  pretty good and then you so they call  that the young siblings weight algorithm  they actually call it the young brothers  weight but in the modern era we call it  young siblings weight okay  and we abort sub computations that prove  to be unnecessary so you're gonna start  out searching but then you want to you  know get rid of things that you discover  oops you know I did get a cutoff from  searching from one of these things I  don't need to do this let's not have it  keep spawning and generating you know  working some other thing let's let's  let's abort it and here's the idea for  the abort mechanism so what you do is  you you know the abort can occur at any  given node saying I don't need to you  get a beta cutoff you want to abort all  the children that wouldn't have been  searched anyway but they may have  already been spawned off so you have a  record in each node that tells you  whether or not you've aborted  and what you do is you just simply climb  up the tree okay to see whether or not  you know periodically whether or not you  have an ancestor who is aborted if you  have an ancestor that's aborted says I'm  aborting my sub you know the the side  computations then you say oh I'm done  and I just have to return okay and so  you check that on a regular basis okay  so does that do people follow what that  mechanism is and so that's basically  what you're going to be implementing for  you know for the parallelization okay is  this thing of climbing up you're gonna  spawn you know you're gonna guess after  you've searched one child that it's good  hey actually some people say why don't  you check two before you you know search  the others okay and so that you're even  more sure that you don't have a cut off  because neither the first two aborted  because in practice of course the game  tree is not best ordered and so you're  gonna waste some work okay so but the  idea is you're going to pull up the tree  to see whether or not and of course you  don't want to pull on every evaluation  that you do you want to just pull  frequently so if you may have a counter  in there that says okay every tenth time  I'm going to pull up the tree or every  you know you put a voodoo parameter  there that says how often it makes sense  to actually check because there's cost  you going up and checking ok versus  versus exploring more of the tree okay  and so I have an example here and I  think this is where I'm gonna cut this  short so let me there's an example which  I suggest you take a look at I want to  as I say spend time doing QA and talking  about the other parts of the program and  give you folks some good ideas for for  things so this is sort of the theory  thing and then I have some slides that  will explain this with examples and more  depth because I don't expect that  everybody got every detail of this okay  so this is the so I have some  examples and so forth in here that I'm  gonna let you guys look at on your own  okay now if you paralyze the spawning  off loop okay of of the the younger  siblings you will get a code with races  okay and the reason you get races is  because there are several there are  three data structures okay that the  search is employing that are that are  kind of global the first is the  transposition table looking things up to  see whether or not you've seen them  before that's a major optimization you  don't want to get rid of the  transposition table and so you have a  choice for the transposition table what  are you going to do with it are you  going to lock it so that the races  become at least your data race free and  updated atomically or you could  replicate it for example you could have  a worker local copy of the data  structure and each each worker that is  working on it only accesses their own  local copies okay you know or you know  you can use a you can make a copy when  things are stolen or you can make just  one where you just maintain it locally  or you can decide that well you know  even if there's a race there the odds  that it's going to affect how I play in  the game are not that high so I'll run  with race okay because any other  solution is going to be is going to be  more expensive okay and and then maybe  you get funny answers so one thing is  for for that kind of data structure let  me just recommend is that you have some  way of turning off the data structure so  that you can actually do the parallel  search and get deterministic repeatable  results  okay so even though it may be aborting  some things may not be aborting you want  to have some way of executing and so for  example you want to be able to turn off  even the speculation in your code so you  can test all the other things in your  code that don't depend on the  speculation okay because if you have  something that's not deterministic as I  say it's a nightmare to debug okay so we  valued you know your value you're gonna  do the evaluation function hey turn off  if you're testing it turn off the  speculation that you should be able to  find out whether you've got bugs in your  evaluation function there's no point in  discovering you of a bug and it's like  well where did it come from or whatever  okay so you want to have things  returning out and also you want to have  ways of turning off for example access  to the transposition table yes I'm going  to speculate but no I'm not going to  access the transposition table in  parallel because because I may get a  race on the entry in the transposition  table okay  I will the the hint that I will give you  for that is that in the in the code that  for our program that took second prize  in the world computer Chess Championship  many years ago I think it was in 1999  where we we actually almost won the  tournament and it's a and we got we lost  in the playoff we we would have won the  tournament except that we suggested our  programmers suggested a rule change at  the beginning of the tournament that  everybody agreed to and then we were on  the short end of that otherwise we would  have been world champions and that was  the last competition against computers  that deep-blue the IBM computer that  beat Kasparov the the human channa it's  just a few months later they performed  it and we actually they placed third in  the tournament we tied for first  our only loss was to deep blue okay and  we were running on an 1824 node soup  computer at Sandia National Labs a  computer that probably is today less  powerful than this okay so but it was  you know it was very big type of  computation so we you know and we were  you know it they basically said if  there's a tie they're gonna base base  who wins on strength of opponents so you  take a look at your opponent's records  and if you had stronger opponents you'll  win against somebody and we said you  know if there's a tie just between two  programs we should really have them you  know face off against each other as an  extra round and everything yeah that's a  good idea and then wouldn't you know at  the end we had the strongest you know  opponents and and we were tied with  another program called fritz an  excellent program and we were out  searching fritz in the in the playoff  game and then we saw a move that in  afterwards we when we were able to do  offline searches deeper we were out  searching them by like two ply okay and  then but then we there's a move that  looks like it's a good move for the  depth we were looking at it doesn't look  like a good move if you search much  deeper or frankly if you search much  shallower is one of these things where  just look like a good move for the ply  we happen to be searching because  there's no guarantee because you can't  see the end of the game there's no  guarantee that if you search deeper  you're actually going to beat somebody  else okay sometime because there's the  horizon effect of you can't just simply  can't see what's coming up in the future  you can only see sort of on average so  we made this bad move we almost  recovered we almost tied had we tied we  would have been world champions because  we had the stronger we won the  tiebreaker and we weren't able to  recover from the from the error and so  we took second prize as like  anyway in that program we decided that  we were just gonna let there be a race  and the Rhian the transposition table  and the reason was we calculated what  are the odds that the race actually  affects the value that you would  actually pick that value and if it  affected the value what are the odds  that affects the move that you're gonna  make and it affects the move you're  gonna make what are the odds that  affects the the game and if he affects  the game what are the odds that affects  your result in the tournament and when  you figured all these things out it was  like and we would slow the program down  more by putting in for example locking  okay because that would slow down every  access then we would if we just if we  just ran things because one of the  things that happens in alpha beta that  that thing is if you get an extreme  value usually those are locked off  because if it's such a good move for you  your opponent's not going to let you  play it so if you ended up with a score  that was extreme because it was based on  a value that you were racing on usually  it's gonna not even propagate up to the  root of the tree so anyway so we just  ran you know naked so to speak so there  are two other data structures you're  gonna have to worry about to make  decision about another one is the killer  [Music]  data structure so the killer data  structure is a heuristic that says you  know at a given depth of code you tend  to see the same moves that are good and  a move that is good for one for one  thing at a given depth tends to be good  for something else so for example it may  be that you know you play Bishop takes  Queen  okay so you've won the other players  Queen and then their response is thier  response is to do something irrelevant  okay well now you've you know you made  them okay well there's a lot of other  moves where you know you could make them  on that move okay for which they're  playing things on the board that are  irrelevant okay and so the same type of  move tends to be a killer always you're  you're able to you know at that depth in  that position you're you're able to make  the same move in it you know if they  don't address the issue that's always a  good move to check and so that ends up  being ordered near the front so the  killer table keeps track of that and I  think in our code we have two killers is  that right Helen I think we it's I think  it's set up to allow you to do for up to  four killers but we only do two in the  code that we gave you I mean you can  enable it and see whether four killers  me but that's a shared data structure  and so one of the questions is should  you be keeping a local copy of that or  should you be you know using a global  one that they all share okay and  updating it the third one is the best  move table which is used to order all  the low order things so the first thing  we most important is did you get a move  out of the transposition table that  tends to be quite accurate then second  is did you get a move out of did you get  a move out of the killer table and  finally it's sort of like statistically  how good are these moves have you seen  these a lot before and if you've seen  them a lot before that's how the other  moves get ordered and that's kept in the  best move table and once again that's a  shared table in the search you're gonna  have to figure out do you want to do a  thread-local you know worker local copy  or do you want to do you want to you  know synchronize it or how are you going  to manage that data structure okay and  the answer I would say is different for  the compared to what you do with a  transposition table okay transposition  table generally it's not worth locking  or or whatever and it is good to share  because if you have seen that move  before that saved you a huge amount of  computation  to be able to look it up and not not do  it okay so those are some of the tips  for parallelization which we'll get to  in the beta too but now I want to talk  about for most of the rest of time talk  about some of the other things that you  can do with the code you've currently  got and let me take these in some some  order and then we can ask questions so  open you who's contemplating doing an  opening book anybody for for beta one  they're gonna do an opening book for  beta 1 or beta 2 hoo-hoo for beta 1 let  me see okay beta 2 who wants to do an  opening book beta for the final okay  yeah okay so you know the idea here is  to pre-compute best moves at the  beginning of the game you know so that  you know it's like well we know what the  starting position is so what on I you  know spend $1,000 or whatever on Amazon  and and you know compute things to some  ungodly ply right and see what the best  moves are at the beginning of the game  and put that into a book so I don't have  to search those so that's the idea I  think to begin with there are lower  hanging fruit than then the opening book  you know if you look at it you're going  to be in opening book you know if you're  lucky for you know six or eight moves  and the games tend to last have you  looked to see how long games last ten to  last yeah what is it  no most games don't go 4096 we don't let  them go that long anyway so anybody's  take a look statistically to see how  long games are I think they tend to be  like 40 or 50 moves okay I mean this  year is different from other years  because we have different rules but I  think it's like 40 or 50 moves so you're  not spending you know so you're doing  something that will help you in 10% of  the game where's there other places you  could do it we're going to help you in  the whole game nevertheless we've had  teams that did a great job on opening  books and clobbered people by you know  by having a fantastic opening book okay  it's actually cheaper to keep separate  opening books for each side than to keep  one opening book for both sides okay and  in this game it's actually fairly easy  to keep a symmetry thing and and  basically have an one opening book that  works for the side on move okay and that  store that allows you to store you don't  have to store you know if if I make a  given move if I have a given position I  only need in principle to store one  answer for that position okay whereas  then my opponent may make any number of  moves for which then I only need to know  one move what's my best move in that  position so you can see that you're only  having two you know you have once again  this property that on my move I only  need to have one move stored my  opponent's moves I need to have all the  moves stored okay and then my move I  have one move story and that basically  means you're effectively going twice the  depth as if you kept all my moves and  then all of the other opponent's moves  and then all of those because when I do  that it's like well why am i you know  why do I need to know what the what the  the best move so it's better to keep  them separate when you build the opening  book you'll store a lot less you have a  lot smaller storage you'll be able to  store  more moves if you do that okay but I  would say not necessarily the best the  first thing I would go to optimize and  but it's also the case that this is a  place where you know one of the teams  you know built a fabulous opening book  one year and and one of the things about  the opening book is it allows you also  to sort of threshold and say you know in  a given move I don't just have to have  one move let me have three moves you  know and pick randomly among them so  that I'm not predictable what the team  did that used the opening book is they  observed that everybody was using words  just searching from the beginning so  they were all finding exactly the same  moves at the beginning of the game and  so they knew exactly what the path was  that all the other programs were gonna  follow and so they could make the best  move on their move but if you have  something that's got some randomness to  it and there is a jitter in there who  found the randomness the place we insert  randomness and the code right so there  was I think Zach had a had a comment on  Piazza about that so there's a place in  the code where we did there the amount  so that you will get unpredictable  results and that way you won't always  play the same move okay just by changing  things this is like you know a tent you  know hundredths of a pawn value is like  not very big and like who cares whether  or not you know we have you know it's  you know we're not actually these  heuristics aren't measuring things so  closely that we care about a tenth of  book a hundredth of a pawn so you can  dither things and have one move be  better than another and not really  affect the quality of the playing okay  so it is a good idea to do something to  not be so predictable okay but to build  a full opening book it that's a lot of  work for beta one okay for beta two in  the final yeah you know you'll get to  that point the next thing is iterative  deepening to people understand how  part of the search code works yeah no  let me go over it okay so you can  imagine saying I'm gonna search to depth  D and in fact instead of just searching  the epithelia depth one search if you  look at the code there if there's a  depth one search and we do a depth to  search then we do a depth research then  we do a depth for search and we keep  doing that until our time control  expires why is that a good strategy  generally well first of all the amount  of work with each depth is growing  exponentially so you're only spending in  principle a constant factor more work  than you would if you search to depth D  right off the bat okay but the important  thing is that you can keep move or  during information as you search deeper  and the move ordering information  remember we want this we want our  searches to use best first move ordering  so that we get all the cut-offs we can  possibly get okay for pruning and so by  searching it easier actually you end up  with better move ordering okay because  when you find the same position the in  the transposition table you say oh well  I didn't search it as deep as I need to  search it now so I can't use the value  that's in the transposition table for  the position because it may be say you  know I've searched something that's you  know apply 8 in the transmission table  but I've got a search to apply 9 not  good enough a search but the value the  the move that you've found at ply 8  that's probably a pretty good first move  you know pretty good best move guess it  best move okay so by doing iterative  deepening you actually get the move  ordering for all those intermediate  positions you know really really really  strongly because that transposition  table is the very best information  you've got for what the first move is  what your best move is sorry ok and also  as I mentioned it's a good mechanism for  time control is that clear  so that's why you bother is one of these  things that looks like it's redundant  why are you searching if you're gonna  search the depth dy search the depth D  minus one you need to do that as part of  depth e well no because you're gonna get  move ordering information this can be  really valuable when you search the  depth key minus one so when you search  the depth D you've got much better  pruning than if you just went straight  to depth D and had no information about  the move ordering does that make sense  in game database so here's the idea if  there's a few enough pieces on the board  you can pre-compute the outcomes and  store them in a database okay so so for  example the most common database to come  up with is the King versus King database  okay so if you do King versus King that  means that it's not that you expect the  game will end up as King versus King  it's that you're gonna encounter that in  the search it'd be nice to know who has  the win if your King versus King now the  code actually plays optimally for King  versus King okay the code we gave you  will we'll do we'll play the King versus  King you know perfectly well and in fact  there's just two heuristics that are  responsible for that one is the the  lives called K aggressive heuristic and  the other one is which basically makes  you move towards your opponent and the  other one is the the K face heuristic  which makes you point towards your  opponent and those two things turn out  to do a pretty good job of when there's  when there's two kings and playing they  go through this little any immed it'd  everybody do that by hand by the way do  you play King versus King so a really  good idea if you want to understand the  game is just is just go and do a king  versus King version of the of the game  get an eight by eight chessboard and  just put down two tokens that you call  our Kings that have orientations and  then just just the two of you and a  friend just try to  see if you you know one can make the  other and you'll see that it goes  through a very ritualistic type of  behavior which ends up with one of the  Kings being mated so it's nice to know  which one it's going to be so if you  encounter that but here's the question  is how many games actually make it to an  end game once again you have to look at  what the odds are there because if games  are ending in the middle game there may  not be any need to search all the way to  the end game okay so so that's something  that you need to weigh okay now if you  do an endgame database it doesn't  suffice to just or win loss or draw for  a position and the reason is because you  can get into a situation where you've  got a winning position and then you say  okay well let me move to another winning  position and another winning position  and another winning position oh and I'm  back it to my original winning position  and I just keep going around in a circle  always in a winning position never  moving towards the win okay and that of  course will end up in a draw so you need  to know which direction do you go for  the winning position so the typical  thing you do there is you keep the  distance to mate to avoid cycling so you  go from instead of keeping just a  boolean value you say here's my distance  to winning okay my distance to winning  is six now when I'm searching I don't  want to go a distance to winning that  seven or six I want to go to a distance  of winning that's five when I make my  move okay  so it's very important to make sure that  you maintain the distance in order to  avoid cycling you know quiescence search  we talked about quiescence which is you  know to make sure that when you're doing  the evaluation you're not in the middle  of an exchange you know so you zap their  pawn they're gonna zap your pawn or  maybe even zap your king okay and you  don't want to evaluate in the middle  there and so the idea of quiescence is  that when you're done with the regular  search you just play off moves that  involve captures and  you evaluate the position okay so that's  quieting the position okay makes sense  another heuristic there is no move  pruning in most positions there's always  something better to do than nothing so  sitting still is usually not as good as  actually making a move okay  and so the idea is suppose that I can  forfeit my move and search a shallower  depth and I still have the best position  okay then I still am winning well then  and it generates a cutoff then why  bother exploring any other moves I  probably am in a really good position  here okay and if if I don't manage to  get a cut off from the no move then I  you know then I do the full the full  depth search okay so so the place this  is dangerous is in zoogs wang in chess  it's called zoogs along situations so  slugs along a situation where everything  is fine but if you have to make a move  you lose so usually making and having  the advantage of a move that's called  the initiative in chess and that's a  good thing you want the initiative you  want to have the extra move over your  opponent but there are some situations  where if you move you lose okay and so  you want to make sure you don't have a  Zhuge Liang situation I don't actually  know if there's a Zhuge Liang situation  in lesser chess okay so it's if somebody  comes up with one I'd be interested to  see that you know where if you had to  move I lose but if I sits there so what  in chess what they do is in the end game  that's when zoogs lines come up they  turn off the null move pruning so is  this clear what null move does for you  what what's going on there so I see some  people who are quizzical anybody want to  ask a question I see some people hide  with sort of quizzical looks on their  faces  okay the idea is for no movies my  position is so good okay that even if I  do nothing I still win okay so I don't  want to search anymore in this part of  it because you're you know I get a beta  cutoff you're not gonna let me make this  move that the move that got me here yeah  question oh did you really  yeah okay that was it okay good yeah my  move is so good I could I can do nothing  and I still win okay and so why bother  you know let me verify that without  necessary doing as deep a search so no  move is fairly cheap to do and it you  know often results in you know for in a  lot of positions for cutoff and if I  don't get a cutoff then I just do the  normal search okay this is a pretty  complicated piece of code isn't it okay  you know pretty complicated okay  there are other situations we mentioned  killers okay  there's also move extensions so typical  move extensions that people look at is  you grant an extra ply in chess if the  king is in check or for certain captures  or if you have a forced move so suppose  you have a move and it's the only move  you can make then don't count that as  apply so you can search deeper along  lines where they're forced moves that's  what they do in chess lie search s not  quite so clear what you do okay which  ones you would do extensions for okay  because it it's rare that you have just  one move in in lesser chess compared to  in in Chester where you ever by force  move it's like if you don't do this  you're going to get captured or you know  made it or what have you  okay so that may come up in less your  chest but anyway it's it's something to  think about  so the transposition table so we talk  about this as a search tree but it's  really a dag because I can get to the  same position by transposing moves you  know this guy does a this guy does B  this guy does C this guy does D I may  get it's the same thing by doing you  know a d c b I transpose those two moves  and I get to exactly the same position  okay and so I'd like not to search that  position again if I've seen it and  that's what the transposition does  there's a quality score that is in the  transposition table that tells you how  good a move you've made and the quality  is essentially the depth that you had to  search in order to establish the value  that's stored in the transposition table  and so you don't want to use something  of too low quality when you're searching  if you have to search the depth D  something of quality D minus 1 is not  good enough ok because otherwise you'll  not be searching you know the full tree  but something typically that is deeper  you know certainly you know if you if if  I'm looking at depth D and I find  something in transposition table that's  d plus 1 or D plus 2 that's great to use  that just gave me an even deeper view of  what's behind that move okay and so if  you look at the logic there you'll see  that that's how they do it it's very  tricky one of the things that's tricky  is when you find a mate how you store  that in the transposition table and  you'll see there's special code for  handling mate positions and the reason  is because what you're interested in  doing when you find a mate is knowing  your distance to mate but not from that  position the distance our mate from the  root of your search okay so for example  [Music]  you know let's say this is the root of  my searched and I searched down to a  given point and now I do a lookup in the  transposition table and I discovered  that there's a mate in five here okay  and this is let's say let's I have  searched nine-ply or something okay well  if I store that this is a mate in five  and I store the value for mate and five  then if it makes it up to the top here  it thinks there's a mate and five but  there isn't a mate in five there's a  mate in 14 okay so when you look it up  and you discover there's a mate and five  in the table you have to do a little bit  of a calculation to translate that into  being a mate in in 14 as the value  that's going to be used here rather than  a mate in five does that make sense so  you'll see that's the logic that's in  there for dealing with mates so mate  basically takes a very large number way  larger than all the material on the  board and then it subtracts what your  number of ply is to get to get to me so  some big number  you know I don't know 32,000 or  something minus the depth to me is what  you is what is how you represent a mate  you know so some very very big number  and and then minus the minus the depth  and that way you can make sure that  you're going for a mate in 13 instead of  a mate and 14 for example you know  preferring that you take the shorter  path make sense  so that's one of the things to look out  in there and there's also a lot of stuff  in there the transposition table has we  talked about caching it's actually a K  associative cache K way associative  cache and so you can a very K to see  what's the best choice of you know how  many how many entries you should have  the more entries you have the longer it  will take you to search them  on the other hand the more likely it is  that good moves stay in the cache okay  so you can decide what's what's the  right trade-off there okay so there's a  good tuning optimization there okay  questions any questions about  transposition table transmission today  is a major optimization so Zobrist  hashing so most of these we taught Helen  talked about at some level but I wanted  to sort of give a chance to have people  ask questions nobody's asking questions  came here for Q&A all you're getting is  a okay so let me explain Zobrist hashing  again a little bit okay so so Zobrist  hashing very clever idea so we have our  board right with with a bunch of pieces  on it that's probably the wrong number  of squares right that's seven by six  okay who cares okay so so the idea is  that what I'm going to do is have a  table of random numbers that I'm going  to compute in advance and this is gonna  be indexed by you know my row my column  my piece type and my orientation okay  and every different combination of row  column type and orientation corresponds  to a different random number in the  table okay okay if some sort of random  number there okay so you know so and  what my hash function is is it's the XOR  of all of the values of all the pieces  that are on this table with their  orientations okay so if I have a king  here you know that's a White King  I guess I need to know who's who's not  just what the type is I also need to  know whether it's white or black right  so the the side okay I think actually  that gets encoded somehow but in any  case yeah I think maybe in the type we  actually keep whether it's a whiter pawn  black pawn yeah it's in the type I think  is the way we actually implement it that  okay so there's white king black king  white pawn black pawn or space so if I  have a king there that corresponds to a  certain value if I end up with a pawn  here I'll say a white pawn then that  will be another one and I XOR those  together and I take the black King okay  and I XOR his position is and the you  know black pawn XOR that in and so I XOR  all these four values  that's the hash function that I use to  do things like look up things in the  transposition table now if I had to  compute this every time I change the  position you know that's if I have a lot  of pieces how many pieces I've got I've  got seven eight sixteen pieces right  there's each side has seven pawns in a  king so sixteen pieces I have to do 16 X  Wars in order to compute my hash  function so Zobrist rationing is really  a lot clever it takes advantage of that  XOR trick that I taught you in the first  now it wasn't the first it was in the  the was the second lecture yeah you know  the bit tricks lecture okay and that is  that XOR is its own inverse so if I want  to remove a piece let's say I'm going to  move this pawn okay from here to here so  I remove this piece what do I do to my  hash function to remove a piece I look  up the value for that pawn in the hash  table and I XOR that into my hash for  the table and I now have  hache for those three pieces left that  make sense so I have my hash function  which is a hash of the four things I'm  not sure you guys can see very well over  there but and I basically simply XOR it  with the hash of you know the pawn that  I removed you know from the you know of  the pawn that I removed in this case and  now I moved it to here so what do I do I  look up that one and I XOR it that value  in here for the new position near the  new pawn position and that gives me now  the the the hash for the new table with  a pawn in that position so any move that  I make I can basically update my hash  function with only two X source and XOR  czar very fast there one instruction you  know there one and they can do lots of X  ORS and stuff so this is actually very  very cheap thing to do  okay very cheap thing to do so that's  Zobrist hashing that you can keep these  these things up to date and that's  something we implemented for you that's  an optimization that was a freebie we  could have given you the hash function  you know like this and had you implement  that but this is one of the ones we gave  you as a freebie for optimization you're  not all saying thank you very much  professor leiserson  okay now in some sense I took away an  opportunity for optimisation didn't I  there okay and so in the transposition  table there are records for the Zobrist  key the score the move the quality also  a bound type whether it's upper lower or  exact because when I return something  from alpha-beta I only know a bound on  it if it's if it's greater than alpha or  less than beta and in some sense the age  of how how old is this move because as  things get older I also want to age them  out of the table  there's several aging things therein  you'll see the best move table also has  an aging process whereas every time it  updates the values gives a new value it  ages all the other values so that they  gradually disappear and aren't relevant  one of the ones that people have get  confused about is the late move  reductions  okay so called LMR okay this is the  situation where I have I'm going to do  let's say my parallel search to you know  of all my moves and the question is you  know it's again you're trying to prune  everything you can okay and so this is  the idea is which are the moves that are  more important to search deeper the ones  near the beginning of your move list if  it's sorted in best first order or the  ones towards the end of your move list  so where where is it most important to  search deeply for things that you think  are possibly the best move or the ones  that you think are the worst moves yeah  yeah it makes sense to search the best  moves right so the idea is well if  something is way down on the move  ordering list why search it as deeply  it's probably not as good a move and so  let me search it more shallow I probably  don't lose much of an opportunity to  discover that that's actually a you know  is indeed a bad move okay there's a  reason that got ordered down there and  so that's late move reduction so with a  good move ordering a beta it kind of  will either occur right away or not also  you search the first few moves normally  and then you start reducing the depth  for four moves I believe in our code we  have two numbers where we reduce by  depth one after a certain number of  moves and reduced by depth two after a  certain number of other moves those are  things that you can tune I wouldn't tune  them very much I don't think just  because and once again I could probably  be wrong here and some of those discover  oh you know if you tune it like this  it's way better okay but those are those  are that's the idea of the late move  reductions probably one of the most  important things to think about is the  move is the representation of the board  right now we represent the board as it  is okay that's a terrible representation  it's very you know time-consuming  there's sort of two major ways you can  do it oh I didn't put the other one here  okay well anyway so one of them is bit  boards here you use a 64 bit to  represent for example where all the  pawns are on the 64 squares of the board  okay and then you can use pop count and  other bit tricks to do move generation  and to implement other chess concepts  okay so if you're looking to see what  are the possible places upon can move  and you want to say can they move right  and let's say it's stored in row major  order you can just do a right shift by  one and that tells you where are all the  places that pawns those pawns could move  and now you can just pick them off one  bit at a time to generate your move list  okay  and and then you can do it that way if  you're gonna move up a thing well then  you're actually doing a shift by or down  you're doing a shift by how much by 8  right to move up or down okay if you're  storing things in row major all right  that makes sense right  so if it's it's 8 by 8 and you're  keeping a bit for each thing then if I  want to generate you know where is this  one if I shift this whole thing stored  in row major order by 8 if I shift it  right it basically puts it there okay so  moving it by you know 7 8 9 that gives  you and then shifting it by 1 gives you  the you know or minus 1 gives you this  or left by 1 and then similarly you can  do by you know shifting by 7 8 or 9 that  way and I can generate all the possible  moves so that's one way of doing move  generation he's using bit for it and  there are a lot of things for example  that you can do with bit boards in  parallel because you can say did I make  a capture here okay or let me use a bit  board to represent where the laser goes  did I make a move that is that is going  to affect the path of laser okay because  because you know one of the major  optimizations you can do is in the in  the evaluations in dealing with a laser  okay because you're spending a lot of  time stroking out Laser positions what's  the point of doing that if you made a  move of something and it didn't affect  where the laser goes why bother  you know doing it out you can just cash  what the what the value of the laser is  and there are a lot of lot more good  stuff on the chest programming wiki okay  so yeah question  [Music]  yeah you got to be careful there right  because if I do a shift to the right for  example so what will I do  I'll just do a mask to eliminate all the  things that got wrapped around okay  right so two instructions or whatever  yeah  details yes details are good though good  question  good question who's timed their program  where are the where are the  opportunities that you see for  performance engineering for a first pass  what are those expensive things what do  you have as expensive thing  sorry the marking laser path yep good  what else is is time expensive  yeah laser coverage boy that is really  expensive okay that is really expensive  so where else is expensive  what else is expensive so I would  definitely go after you know L cover  that's a that's a huge one to go after  one of the things by the way if you are  making your your changes to the code and  you're gonna change the representation  leave the old representation there take  it out at the end add the new  representation and put in assertions  that say that things are equivalent or  whatever but don't get rid of the old  stuff because you'll end up with broken  code and definitely using use things  like perfectiy to tell you whether or  not you you know made any change you  know made any changes if you touch  anything with move ordering sorry not a  move generation not move ordered move  generation so where else is expensive  yeah  sure you want to get your how much  detail do you want how it actually works  okay so what it's supposed to do is  what's supposed to do is figure out how  safe what you're the idea is I want my  laser to get closer to your King and I  want your laser not to be close to my  king  right and if I can move into positions  where my laser is closer to your King  but your laser doesn't get closer to my  King that's a good sort of thing but  when we say get the laser close you know  what happens when I've got say let me do  it this way a position like this so  here's the okay suppose I look at this  position how close does the laser get  Mike let's say I'm black here and I look  at the laser well the path of the laser  is it bounces there and goes across  right okay so I didn't get very close to  the King there but I'm one move away  from getting it pretty close because if  I just move this guy out of the way now  I've got it really quite close okay so  if I compare that to let's say a  situation where instead of the the  plants are there let's say a pawn is  here okay  now the laser is actually closer to the  King that it was in the first position  but it's a much worse position right the  first one was much better when I had the  pawns here and here because I was simply  one move away from getting it really  close and so if you use just the direct  laser thing and we did some tests on  this this turns out to be not very it  doesn't guide the program very well on  getting into situations where my laser  is getting close to your king  that makes sense so then we said well  how can we you know how should we  measure on how close the laser get so  what we said is well let's take a look  at all the possible moves from here of  one move you know and then look to see  how close we get things and the way we  did that is we said let's you know this  took actually Helen and I worked on this  really hard okay because this is a new  heuristic that we have not used in  previous years and it works great it's  just slow as anything okay but it works  well so you have to evaluate is it worth  doing something if it's really slow it  may be that you do better to use a  simpler heuristic and get deeper search  than it is spending a lot of time  evaluating but anyway we gave you one  that if you can make it go fast should  be really good so what the idea is that  what we do is we look at all the  different paths from moving any piece  want you know one piece and look at the  paths of the laser and what we do is we  go we count one for every position that  we go away 2 3 4 5 6 7 etc we actually  add an extra value if we bounce off in  an opponent's how much do we add Helen  do we add one or two if we bounce off an  opponent I think - yeah so if this is an  opponent's pawn we would basically go  from 3 to 5 6 7 you know 8 9 and we  basically do that for all the moves and  the way we combine them is we take the  minimum number that I can get there ok  what's the what's the cheapest way I can  get to every every square that the laser  goes so we take all the different moves  we stroke them out and we take the  minimum so if I have another path let's  say by turning the king it will go this  way and there's a you know there's  another one here that's my pawn ok then  I may get there in 1 2 3 4 5 6 7 and so  then this would become a value of 7 here  okay so that's the first thing is we  basically get a map of you know how  close are things and the second thing we  do is say well how valuable is it to  have these these you know to be near the  particular King and then what we ended  up with is something where if I look at  the if I look at the distance that I am  away let's say this one for example is  one row in one column away this is zero  row zero columns we use I think it's one  over the row plus one times one over the  column plus one okay as a multiplier to  wait how good it is that we've been  close to the king and we invert these  values so that they're you know so the  closer you you know so that it's better  to have a smaller number there and we  add them up okay and that basically  gives we I don't quite add them up we  actually no I'm sorry what we do is we  look at these things as a fraction of my  shortest path distance sorry that's what  we did yes I'm sorry that is a previous  sourcing so so here for example let's  say this is the nine the shortest way of  getting there is one two three four five  six seven eight  so this would actually when we do the  conversion this would we give a value of  eight ninths for being in that square  okay so we didn't get it where's this  one the shortest path distance is 7  because of this path and you can get  there and seven this would have a value  of one so this is better you know and so  we do that for all the squares we get a  fraction of how much do we do and then  we wait it by something like this which  falls away quickly but it's important in  heuristics like this to have a smooth  path so that you can get things closer  if I made all these things be 0 it  wouldn't know that it could you know  move up and get a little bit better you  know in the second or third order  you know digit to get closer and so then  we add it all up and that ends up being  a number we discovered that sort of in  the range of  you know you know like what did we  figure the range was there there's like  you know up to like four or so right  something something like four on average  you know would be the maximum value  would be and then we said okay then we  have this magic constant multiplier that  you can play with that says okay let's  represent that as a fraction of a pawn  how much is that worth and we multiply  by that value okay so that's what we're  doing okay so that it makes it so that  we do tend to move the laser our laser  closer we subtract how close the  opponent can get from us and then we say  that's the evaluation now if you have a  better way of determining whether a  laser is getting close than this one  that's cheaper to compute that's good  okay for first pass I would just simply  try to make this go fast because for  many of the moves that you're going to  look at is going to be moving this pawn  you know to there nothing changed  there's no point in there stroking this  out and cakung are the minimum etc  because i didn't touch the laser path  things are gonna touch a laser path are  things that you move on or off the path  so why bother okay and so if there's a  clever way of hashing this you know so  that you're caching it so that you can  you know do things that's good too  okay any questions about other things  that could be done so another place I'll  tell you that's a good idea to look at  is especially once you get this  heuristic operate a little bit faster is  the sorting of moves okay is is actually  pretty expensive  okay what's the you know once you figure  out here all the moves there are a lot  of moves now you go through and you do a  sort and if you think about it you know  in a best move order tree sometimes you  only explore one of those so why'd you  bother sorting all of them okay on the  other hand sometimes you do need to sort  all of them so how you optimize that  sort so that you don't waste time  sorting stuff that you're not going to  actually ever explore  another opportunity for optimization  okay yeah question the moves are sorted  by these things like that there there's  a sort key in there that represents a  whole bunch of things like whether it  was a killer if it's a killer or it's a  transposition table value gets a very  big value if it's a statistical thing  that the history table says in  historically this has been a pretty good  move then you get another value and then  exchanges get ordered and so forth so  there's a bunch of things that are you  know where where you end up with  information and then it sorts those  things okay so that's actually kind of  complicated logic in terms of what the  actual values are that are used there  but the idea is you know here all the  moves now let's figure out what's the  best our best guess as to what they are  most of the good moves are coming out of  the transposition table right  but sometimes the transposition table  you know it's not it says it's not a  very good move and there may be a better  move okay so the quality of what you get  out of transposition table is good but  doesn't mean you know that that's always  the best move okay yeah any other  questions about what should be worked on  let me just make sure I okay go ahead  where does the sorting happen that  happens at the I think at the move  generation is that where it is is the  sort in the move generation or is it in  search I think it's in search I think  you're right  search calls move generation and then  sorts it okay I think that's right  yeah question  [Music]  very tiny board implementation yes well  as I say keep the old one around well I  you use the old one while you're still  getting the new one right okay it's also  good to be able to go from old  representation to new representation and  having conversion functions okay but  yeah making representation changes you  know painful painful and is it's  probably boy if there was one tool I  would love that would automate stuff  like that that would be it you know to  be able to change representations of  things and still have things go well I  should have mentioned by the way the  other representation besides bit board  another one is just to keep a list of  the pieces and their positions okay  because that's going to be smaller than  keeping this great big board okay then  you keep this list sorted or do you not  keep it sorted and do you but the  advantage of that is particularly as the  game goes on that list gets shorter and  shorter okay and so so you're you know  if you're doing less and manipulating  the the board position that's a that's  generally a good thing so for the board  reputation my recommendation is to  refactor all the board axes into a  function and then you would still use  the old representation in the function  then you can validate that you refactor  everything correctly and then you can  easily change the board reputation to  whatever you want and keep changing it  without having to do a lot of refactor  off that for that so find all the board  accesses and refactor that to a function  call before you modify it because the  you know the compiler will inline that  stuff so you know putting in a function  call is not necessarily a bad idea and  if it doesn't you can put it in a macro  and it'll be effectively the same thing  so great idea great idea okay yeah  question  testing testing testing you know if you  can test and know that when you make a  change it's correct you know and that  you're not that's probably the number  one hole that people go into is they  don't test adequately so having good  test infrastructure you know it's good  yeah  [Music]  as long as you're doing it  deterministically yes that's because  there's a randomness thing you turn off  the ran there's a little variable you  can set the variable to be not random  and then it will be yes it's run  serially and risk can give her the mic  [Music]  yeah juice  yeah I got out of the opening phone I  said the  [Music]  yeah it shouldn't be if you run serially  the only things that you know should be  doing things as if you're doing I mean  it should be deterministic if you turn  off the the random number generator so  as I say that's put in so that it will  behave you know not predictably but  that's exactly the kind of thing you  want to find right at the beginning so  as exactly things find out all the ways  of making it deterministic and that  would be really important for beta-2  okay so Thursday okay we have you know  John Bentley legend of the field  okay opportunity to meet a celebrity  okay bring friends okay he's going to  give a great lecture  you

2
00:00:03,189 --> 00:00:05,769
the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  okay let's get started on Thursday we  are really privileged to have John  Bentley who is one of the Masters of  performance engineering come and give us  a guest lecture in 1982 he wrote this  wonderful little book from which we  adapted the Bentley rules that you folks  have seen and he's also famous for  things like KD trees who's seen KD trees  yeah so he invented KD trees and who's  ever used the master method of the  current saw he invented the master  method of recurrence solving and so I  mean he's just a wonderful wonderful  fellow with lots and lots of insights  and just like a superior performance  engineer and he's going to talk give a  guest lecture on Thursday and so I would  encourage you to bring your friends  bring friends maybe who dropped the  class okay because this is like this is  the and and others who you may who you  know if any of you have urops other  people in your research group whether  there are graduate students or whatever  they're all welcome to come because this  is really you know one of the  opportunities you get to to actually  meet one of the you know legends of the  field if you will and you know have a  and you'll see he's very approachable  he's very approachable so um so anyway  my advertisement for John Bentley good  so let's start I want to talk about  speculative parallelism for a little bit  because that's kind of what you need to  make the code go fast so if you look at  code like alpha beta it turns out that  it's inherently serial okay that is if  you don't you know if you try to do  things in parallel then you miss the  cut-offs and if you miss the cut-offs  you know the beta cut-offs then  you start doing work that you wouldn't  necessarily need to do and so the only  way to sort of make this type of code  run fast is to use speculation which is  to say that you're gonna guess that you  can do stuff in parallel and it'll be  worthwhile and then occasionally you're  going to be wrong but the trick is well  how do you minimize the wasted effort  when things go wrong so let's just take  a look at a very simple example of of  speculative parallelism which is Thresh  holding a sum so here we have a little  program that is returning a boolean it  takes as input an array of size n so  threshold in sums you have an array a of  length n and a limit and these are all  unsigned integers say and I start out  the summit zero and what I'm gonna do is  add up all the elements of the array and  if it exceeds the limit then I'm gonna  return okay and so um how might I  optimize this code a little bit yeah  no let's say on one one processor on one  processor what might you do yeah once  you check so that once you would exceed  it  why keep adding the other numbers okay  so here's a code that does that quit  early if the partial product ever sees  the threshold this isn't necessarily an  optimization it was noticed that now in  the optimization not only do I have you  know a memory reference and in addition  I also now have a you know I also have a  unpredictable branch actually it's  predictable branch sorry predictable  branch but it's still one more one more  step because it's always going to  predict that it's not exceeding until it  actually does exceed so that'll be  pretty predictable and but still I've  added something into the loop so that  maybe so how might I mitigate that  problem so I don't want to add too much  into the inner loop yeah  [Music]  yeah so basically I can do what's called  strip mining I replace the loop of n  iterations with a loop of let's say n  over four iterations with an inner loop  of four iterations and every fourth  iteration I check to see whether or not  I've exceeded okay so that the cost of  the the check is going to be you know  relatively small at that point okay so  there are ways of of making this sort of  go fast okay so now we want to make this  operate in parallel okay and the problem  in when I operate in parallel is that is  that as I'm adding stuff up I want to do  it so I'm going to do this here with a  reducer and so basically I'm going to  add up the values in the reducer but now  I'd like to do the same thing of being  able to quit early and so the question  is well how can we paralyze a short  circuited loop okay how can we and so  the way I'm going to do this and this is  sort of by hand here but it's going to  give you so actually as you know  underneath the loop is really a divide  and conquer loop and so you know I could  I could write it as a as a parallel loop  and now it becomes I think a little bit  clearer how I could in this case what  I'm going to do is is return the value  of the sum and now the question is well  how can I quit early and save the work  if I exceed the threshold okay  understanding that you know when I'm  executing this in something like silk  you know it's going to tend to you know  go down one path and often it's going to  be a serial execution so I'd like if  it's if it's possible so here's here's  one way of doing it which is then I  add a and abort flag to the code and  which is going to say whether I should  keep going or whether I have actually  exceeded at that point and so I'm going  to use that recursively and now I take a  look at for each time through the loop  or through the divide and conquer I look  to see whether the sum is greater than a  limit and I haven't you know already  aborted the flag then I set the abort  flag to be true okay  why do I bother testing the abort flag  before I set it to true so notice that  that setting the abort flag is a race  isn't it a determinacy race okay because  I have great thank you it's because I  have the the stuff operating in parallel  is all trying to set the abort flag  whenever something abort so I can have  two guys who are in parallel setting the  abort flag but if they're setting it  they're setting it to the same value so  it's a benign race okay assuming your  memory model is such that you can  actually set the values so so what  happens here when why do i why do you  bother checking this  so what happens when several guys in  parallel want to write to the same  variable this is quiz one stuff yeah  yeah you can have it bouncing along the  cache it will serialize it right because  because if they're all trying to to  write then they have to get exclusive  access for it to write and modify it and  so that happens boom boom boom one at a  time  and so by checking it first it can be in  a shared state and then only one guy has  to one guy clobbers it and then it'll  update all the other ones so it makes it  so we don't continually have a true  sharing race and then in the checking to  see if it exceeds we basically just  check to see if you know we basically  call this function we set the abort flag  to false and then we return the sum of  all the values and if it aborts then it  just returns early okay so it doesn't  have to keep going through the  computation and of course once again you  can coerce in the leaves and so forth so  so this is non-deterministic code right  which we should never write unless we  have to okay because that's the only way  of getting performance and you have to  make sure you of course you reset the  abort flag after the use and then  actually do you need a memory fence here  on the abort flag do you need to set you  know where where would you put a an  abort flag to make sure that the value  was  yeah yes or what what would you have to  do  [Music]  okay so so indeed it turns out you don't  need a memory fence here and the reason  is because the code is correct whether  it's the initial value force or whether  it's become true so it doesn't matter  when it becomes true and so there's an  issue of if you put in the fence then  you know that it becomes true earlier  okay rather than waiting for the  convocation but that may actually slow  things down because memory fences are  expensive but because the transition is  always from false to true during the  execution you don't actually need a  memory fence there okay in order to make  sure that you've got the correct value  does that make sense so you don't need a  memory fence so so that's a classic  instance of speculative parallelism it  occurs when a program spawns some  parallel work that might not be  performed in a serial execution so  you're performing it anticipating that  you're probably going to need to do it  okay and then typically what you want to  do is have some way of backing out if  you discover that that you've made made  it you know that you didn't need to do  that way have some way of making sure  that you don't do any more okay so basic  rule of thumb of speculation is don't  spawn speculative work unless there's  little other opportunity for parallelism  and there's a good chance that will be  needed so one of the things I've seen in  research papers no less is people who  say oh I'm going to you know I have a  calculation where I'm trying to find say  the minimum of a bunch of values and so  I'm gonna you know spawn off you know  end things each of which is looking for  the minimum as soon as the minimum one  comes back I'm going to retract all the  other computations and I'm gonna get  super linear speed-up that way because  because in the serial execution I might  have done the longer one first and maybe  the minimum is not the not the first one  or whatever and so they they claim super  linear speed-up by doing speculative  parallelism but that's not really that's  not really a good example because there  was a better serial code if that was  really a good way of doing it they  should have been doing what's called  dovetailing which is they doing a little  bit of this computation a little bit of  this a little bit of this a little bit  of this etc and then going back that  would have been a better algorithm for  which you would then use it and the risk  they have is that they're spawning off  and things of which most of them may not  be needed and now they don't get any  speed up even though they've just used a  lot more work and that's often is is a  bad bad choice so usually you don't want  to spawn speculative work unless there's  little other opportunity and there's a  good chance it'll be needed okay  my experience is you know how much what  kind of chance do you need you need to  you know have certainly less than you  know for parallelism you've got to have  if if the chance that you're going to  need the work if you have P processors  if the chance that it could be not  needed is is less than actually have a  theorem at the end which I'll which I'll  I'll refer you to ok because a little  bit hard to say because I didn't put on  the slide but there's there's a final  slide for this which gives a nice little  theorem about when you should do this so  now let's talk about parallel alpha beta  search because that's kind of what  you're doing with your principle  variation search and so if you remember  the analysis done by Knuth and Morris  they basically showed that for gram tree  with branching factor B and depth D and  alpha beta search with move searched in  best first order examines that exactly B  to the  d over 2 plus B to the floor of D over 2  minus 1 nodes at ply D so that's  basically square routing the amount of  work that you would need if you did a  full depth fly that's with alpha beta ok  so naive algorithm looks at B to the D  nodes at depth D and so for the same  work the search depth is effectively  doubled or the work is Square rooted ok  so that we saw last time in Helens  lecture ok so the key optimization here  is pruning the game tree and as I  mentioned the beginning of this when you  prune the game tree you've got what's  effectively a serial thing and if you if  you let something go ahead you may be  working on something that would be  pruned so then how does the parallelism  really help you there you're just  wasting processors it could be better  spent perhaps elsewhere except where  else can you spend it because there's no  other parallelism in the in the code so  we want to find some solution so the  solution comes for a from an idea from a  very nice paper by Burkhardt manin and  some of his students in Germany and they  made the observation that in a best  ordered tree the degree of every node is  either either 1 or maximal this is this  is not their observation this is  actually the observation in the knuth  paper ok so when you have a best ordered  tree if you can get all the moves  ordered in their true order ok then it  turns out that either you're you're  exploring all the children or you are  refuting something and you get a cut off  from the very first one that you look at  ok and so in this case for example it  turns out on the principle variation  those are all full  with sorry they have you have to explore  all the children and then from there on  it alternates one level it's it you know  it you have just a single child that  needs to be explored and when you  explore it if it's best ordered you will  get a beta cutoff for the for the others  and then it alternates and then it's  full width' and so their idea was to say  well if the first child fails to  generate a beta cutoff speculate that in  fact you have the best one and the  remaining children can be searched in  parallel without wasting any work okay  so if it fails you're gonna say oh I'm  gonna speculate that this is in fact a  full-width thing now in practice you  don't necessarily get things best  ordered but they're a bunch of  heuristics in the code we've given you  and in in chess code to make it so that  you tend to do things in the in the  proper order the most important of those  is have you if you've seen the move  before okay and it's in the  transposition table you use the move  that you've seen before even if it's to  a shallower depth okay you guess that  that's going to be their best move still  even if you search deeper and that's  pretty good and then you so they call  that the young siblings weight algorithm  they actually call it the young brothers  weight but in the modern era we call it  young siblings weight okay  and we abort sub computations that prove  to be unnecessary so you're gonna start  out searching but then you want to you  know get rid of things that you discover  oops you know I did get a cutoff from  searching from one of these things I  don't need to do this let's not have it  keep spawning and generating you know  working some other thing let's let's  let's abort it and here's the idea for  the abort mechanism so what you do is  you you know the abort can occur at any  given node saying I don't need to you  get a beta cutoff you want to abort all  the children that wouldn't have been  searched anyway but they may have  already been spawned off so you have a  record in each node that tells you  whether or not you've aborted  and what you do is you just simply climb  up the tree okay to see whether or not  you know periodically whether or not you  have an ancestor who is aborted if you  have an ancestor that's aborted says I'm  aborting my sub you know the the side  computations then you say oh I'm done  and I just have to return okay and so  you check that on a regular basis okay  so does that do people follow what that  mechanism is and so that's basically  what you're going to be implementing for  you know for the parallelization okay is  this thing of climbing up you're gonna  spawn you know you're gonna guess after  you've searched one child that it's good  hey actually some people say why don't  you check two before you you know search  the others okay and so that you're even  more sure that you don't have a cut off  because neither the first two aborted  because in practice of course the game  tree is not best ordered and so you're  gonna waste some work okay so but the  idea is you're going to pull up the tree  to see whether or not and of course you  don't want to pull on every evaluation  that you do you want to just pull  frequently so if you may have a counter  in there that says okay every tenth time  I'm going to pull up the tree or every  you know you put a voodoo parameter  there that says how often it makes sense  to actually check because there's cost  you going up and checking ok versus  versus exploring more of the tree okay  and so I have an example here and I  think this is where I'm gonna cut this  short so let me there's an example which  I suggest you take a look at I want to  as I say spend time doing QA and talking  about the other parts of the program and  give you folks some good ideas for for  things so this is sort of the theory  thing and then I have some slides that  will explain this with examples and more  depth because I don't expect that  everybody got every detail of this okay  so this is the so I have some  examples and so forth in here that I'm  gonna let you guys look at on your own  okay now if you paralyze the spawning  off loop okay of of the the younger  siblings you will get a code with races  okay and the reason you get races is  because there are several there are  three data structures okay that the  search is employing that are that are  kind of global the first is the  transposition table looking things up to  see whether or not you've seen them  before that's a major optimization you  don't want to get rid of the  transposition table and so you have a  choice for the transposition table what  are you going to do with it are you  going to lock it so that the races  become at least your data race free and  updated atomically or you could  replicate it for example you could have  a worker local copy of the data  structure and each each worker that is  working on it only accesses their own  local copies okay you know or you know  you can use a you can make a copy when  things are stolen or you can make just  one where you just maintain it locally  or you can decide that well you know  even if there's a race there the odds  that it's going to affect how I play in  the game are not that high so I'll run  with race okay because any other  solution is going to be is going to be  more expensive okay and and then maybe  you get funny answers so one thing is  for for that kind of data structure let  me just recommend is that you have some  way of turning off the data structure so  that you can actually do the parallel  search and get deterministic repeatable  results  okay so even though it may be aborting  some things may not be aborting you want  to have some way of executing and so for  example you want to be able to turn off  even the speculation in your code so you  can test all the other things in your  code that don't depend on the  speculation okay because if you have  something that's not deterministic as I  say it's a nightmare to debug okay so we  valued you know your value you're gonna  do the evaluation function hey turn off  if you're testing it turn off the  speculation that you should be able to  find out whether you've got bugs in your  evaluation function there's no point in  discovering you of a bug and it's like  well where did it come from or whatever  okay so you want to have things  returning out and also you want to have  ways of turning off for example access  to the transposition table yes I'm going  to speculate but no I'm not going to  access the transposition table in  parallel because because I may get a  race on the entry in the transposition  table okay  I will the the hint that I will give you  for that is that in the in the code that  for our program that took second prize  in the world computer Chess Championship  many years ago I think it was in 1999  where we we actually almost won the  tournament and it's a and we got we lost  in the playoff we we would have won the  tournament except that we suggested our  programmers suggested a rule change at  the beginning of the tournament that  everybody agreed to and then we were on  the short end of that otherwise we would  have been world champions and that was  the last competition against computers  that deep-blue the IBM computer that  beat Kasparov the the human channa it's  just a few months later they performed  it and we actually they placed third in  the tournament we tied for first  our only loss was to deep blue okay and  we were running on an 1824 node soup  computer at Sandia National Labs a  computer that probably is today less  powerful than this okay so but it was  you know it was very big type of  computation so we you know and we were  you know it they basically said if  there's a tie they're gonna base base  who wins on strength of opponents so you  take a look at your opponent's records  and if you had stronger opponents you'll  win against somebody and we said you  know if there's a tie just between two  programs we should really have them you  know face off against each other as an  extra round and everything yeah that's a  good idea and then wouldn't you know at  the end we had the strongest you know  opponents and and we were tied with  another program called fritz an  excellent program and we were out  searching fritz in the in the playoff  game and then we saw a move that in  afterwards we when we were able to do  offline searches deeper we were out  searching them by like two ply okay and  then but then we there's a move that  looks like it's a good move for the  depth we were looking at it doesn't look  like a good move if you search much  deeper or frankly if you search much  shallower is one of these things where  just look like a good move for the ply  we happen to be searching because  there's no guarantee because you can't  see the end of the game there's no  guarantee that if you search deeper  you're actually going to beat somebody  else okay sometime because there's the  horizon effect of you can't just simply  can't see what's coming up in the future  you can only see sort of on average so  we made this bad move we almost  recovered we almost tied had we tied we  would have been world champions because  we had the stronger we won the  tiebreaker and we weren't able to  recover from the from the error and so  we took second prize as like  anyway in that program we decided that  we were just gonna let there be a race  and the Rhian the transposition table  and the reason was we calculated what  are the odds that the race actually  affects the value that you would  actually pick that value and if it  affected the value what are the odds  that affects the move that you're gonna  make and it affects the move you're  gonna make what are the odds that  affects the the game and if he affects  the game what are the odds that affects  your result in the tournament and when  you figured all these things out it was  like and we would slow the program down  more by putting in for example locking  okay because that would slow down every  access then we would if we just if we  just ran things because one of the  things that happens in alpha beta that  that thing is if you get an extreme  value usually those are locked off  because if it's such a good move for you  your opponent's not going to let you  play it so if you ended up with a score  that was extreme because it was based on  a value that you were racing on usually  it's gonna not even propagate up to the  root of the tree so anyway so we just  ran you know naked so to speak so there  are two other data structures you're  gonna have to worry about to make  decision about another one is the killer  [Music]  data structure so the killer data  structure is a heuristic that says you  know at a given depth of code you tend  to see the same moves that are good and  a move that is good for one for one  thing at a given depth tends to be good  for something else so for example it may  be that you know you play Bishop takes  Queen  okay so you've won the other players  Queen and then their response is thier  response is to do something irrelevant  okay well now you've you know you made  them okay well there's a lot of other  moves where you know you could make them  on that move okay for which they're  playing things on the board that are  irrelevant okay and so the same type of  move tends to be a killer always you're  you're able to you know at that depth in  that position you're you're able to make  the same move in it you know if they  don't address the issue that's always a  good move to check and so that ends up  being ordered near the front so the  killer table keeps track of that and I  think in our code we have two killers is  that right Helen I think we it's I think  it's set up to allow you to do for up to  four killers but we only do two in the  code that we gave you I mean you can  enable it and see whether four killers  me but that's a shared data structure  and so one of the questions is should  you be keeping a local copy of that or  should you be you know using a global  one that they all share okay and  updating it the third one is the best  move table which is used to order all  the low order things so the first thing  we most important is did you get a move  out of the transposition table that  tends to be quite accurate then second  is did you get a move out of did you get  a move out of the killer table and  finally it's sort of like statistically  how good are these moves have you seen  these a lot before and if you've seen  them a lot before that's how the other  moves get ordered and that's kept in the  best move table and once again that's a  shared table in the search you're gonna  have to figure out do you want to do a  thread-local you know worker local copy  or do you want to do you want to you  know synchronize it or how are you going  to manage that data structure okay and  the answer I would say is different for  the compared to what you do with a  transposition table okay transposition  table generally it's not worth locking  or or whatever and it is good to share  because if you have seen that move  before that saved you a huge amount of  computation  to be able to look it up and not not do  it okay so those are some of the tips  for parallelization which we'll get to  in the beta too but now I want to talk  about for most of the rest of time talk  about some of the other things that you  can do with the code you've currently  got and let me take these in some some  order and then we can ask questions so  open you who's contemplating doing an  opening book anybody for for beta one  they're gonna do an opening book for  beta 1 or beta 2 hoo-hoo for beta 1 let  me see okay beta 2 who wants to do an  opening book beta for the final okay  yeah okay so you know the idea here is  to pre-compute best moves at the  beginning of the game you know so that  you know it's like well we know what the  starting position is so what on I you  know spend $1,000 or whatever on Amazon  and and you know compute things to some  ungodly ply right and see what the best  moves are at the beginning of the game  and put that into a book so I don't have  to search those so that's the idea I  think to begin with there are lower  hanging fruit than then the opening book  you know if you look at it you're going  to be in opening book you know if you're  lucky for you know six or eight moves  and the games tend to last have you  looked to see how long games last ten to  last yeah what is it  no most games don't go 4096 we don't let  them go that long anyway so anybody's  take a look statistically to see how  long games are I think they tend to be  like 40 or 50 moves okay I mean this  year is different from other years  because we have different rules but I  think it's like 40 or 50 moves so you're  not spending you know so you're doing  something that will help you in 10% of  the game where's there other places you  could do it we're going to help you in  the whole game nevertheless we've had  teams that did a great job on opening  books and clobbered people by you know  by having a fantastic opening book okay  it's actually cheaper to keep separate  opening books for each side than to keep  one opening book for both sides okay and  in this game it's actually fairly easy  to keep a symmetry thing and and  basically have an one opening book that  works for the side on move okay and that  store that allows you to store you don't  have to store you know if if I make a  given move if I have a given position I  only need in principle to store one  answer for that position okay whereas  then my opponent may make any number of  moves for which then I only need to know  one move what's my best move in that  position so you can see that you're only  having two you know you have once again  this property that on my move I only  need to have one move stored my  opponent's moves I need to have all the  moves stored okay and then my move I  have one move story and that basically  means you're effectively going twice the  depth as if you kept all my moves and  then all of the other opponent's moves  and then all of those because when I do  that it's like well why am i you know  why do I need to know what the what the  the best move so it's better to keep  them separate when you build the opening  book you'll store a lot less you have a  lot smaller storage you'll be able to  store  more moves if you do that okay but I  would say not necessarily the best the  first thing I would go to optimize and  but it's also the case that this is a  place where you know one of the teams  you know built a fabulous opening book  one year and and one of the things about  the opening book is it allows you also  to sort of threshold and say you know in  a given move I don't just have to have  one move let me have three moves you  know and pick randomly among them so  that I'm not predictable what the team  did that used the opening book is they  observed that everybody was using words  just searching from the beginning so  they were all finding exactly the same  moves at the beginning of the game and  so they knew exactly what the path was  that all the other programs were gonna  follow and so they could make the best  move on their move but if you have  something that's got some randomness to  it and there is a jitter in there who  found the randomness the place we insert  randomness and the code right so there  was I think Zach had a had a comment on  Piazza about that so there's a place in  the code where we did there the amount  so that you will get unpredictable  results and that way you won't always  play the same move okay just by changing  things this is like you know a tent you  know hundredths of a pawn value is like  not very big and like who cares whether  or not you know we have you know it's  you know we're not actually these  heuristics aren't measuring things so  closely that we care about a tenth of  book a hundredth of a pawn so you can  dither things and have one move be  better than another and not really  affect the quality of the playing okay  so it is a good idea to do something to  not be so predictable okay but to build  a full opening book it that's a lot of  work for beta one okay for beta two in  the final yeah you know you'll get to  that point the next thing is iterative  deepening to people understand how  part of the search code works yeah no  let me go over it okay so you can  imagine saying I'm gonna search to depth  D and in fact instead of just searching  the epithelia depth one search if you  look at the code there if there's a  depth one search and we do a depth to  search then we do a depth research then  we do a depth for search and we keep  doing that until our time control  expires why is that a good strategy  generally well first of all the amount  of work with each depth is growing  exponentially so you're only spending in  principle a constant factor more work  than you would if you search to depth D  right off the bat okay but the important  thing is that you can keep move or  during information as you search deeper  and the move ordering information  remember we want this we want our  searches to use best first move ordering  so that we get all the cut-offs we can  possibly get okay for pruning and so by  searching it easier actually you end up  with better move ordering okay because  when you find the same position the in  the transposition table you say oh well  I didn't search it as deep as I need to  search it now so I can't use the value  that's in the transposition table for  the position because it may be say you  know I've searched something that's you  know apply 8 in the transmission table  but I've got a search to apply 9 not  good enough a search but the value the  the move that you've found at ply 8  that's probably a pretty good first move  you know pretty good best move guess it  best move okay so by doing iterative  deepening you actually get the move  ordering for all those intermediate  positions you know really really really  strongly because that transposition  table is the very best information  you've got for what the first move is  what your best move is sorry ok and also  as I mentioned it's a good mechanism for  time control is that clear  so that's why you bother is one of these  things that looks like it's redundant  why are you searching if you're gonna  search the depth dy search the depth D  minus one you need to do that as part of  depth e well no because you're gonna get  move ordering information this can be  really valuable when you search the  depth key minus one so when you search  the depth D you've got much better  pruning than if you just went straight  to depth D and had no information about  the move ordering does that make sense  in game database so here's the idea if  there's a few enough pieces on the board  you can pre-compute the outcomes and  store them in a database okay so so for  example the most common database to come  up with is the King versus King database  okay so if you do King versus King that  means that it's not that you expect the  game will end up as King versus King  it's that you're gonna encounter that in  the search it'd be nice to know who has  the win if your King versus King now the  code actually plays optimally for King  versus King okay the code we gave you  will we'll do we'll play the King versus  King you know perfectly well and in fact  there's just two heuristics that are  responsible for that one is the the  lives called K aggressive heuristic and  the other one is which basically makes  you move towards your opponent and the  other one is the the K face heuristic  which makes you point towards your  opponent and those two things turn out  to do a pretty good job of when there's  when there's two kings and playing they  go through this little any immed it'd  everybody do that by hand by the way do  you play King versus King so a really  good idea if you want to understand the  game is just is just go and do a king  versus King version of the of the game  get an eight by eight chessboard and  just put down two tokens that you call  our Kings that have orientations and  then just just the two of you and a  friend just try to  see if you you know one can make the  other and you'll see that it goes  through a very ritualistic type of  behavior which ends up with one of the  Kings being mated so it's nice to know  which one it's going to be so if you  encounter that but here's the question  is how many games actually make it to an  end game once again you have to look at  what the odds are there because if games  are ending in the middle game there may  not be any need to search all the way to  the end game okay so so that's something  that you need to weigh okay now if you  do an endgame database it doesn't  suffice to just or win loss or draw for  a position and the reason is because you  can get into a situation where you've  got a winning position and then you say  okay well let me move to another winning  position and another winning position  and another winning position oh and I'm  back it to my original winning position  and I just keep going around in a circle  always in a winning position never  moving towards the win okay and that of  course will end up in a draw so you need  to know which direction do you go for  the winning position so the typical  thing you do there is you keep the  distance to mate to avoid cycling so you  go from instead of keeping just a  boolean value you say here's my distance  to winning okay my distance to winning  is six now when I'm searching I don't  want to go a distance to winning that  seven or six I want to go to a distance  of winning that's five when I make my  move okay  so it's very important to make sure that  you maintain the distance in order to  avoid cycling you know quiescence search  we talked about quiescence which is you  know to make sure that when you're doing  the evaluation you're not in the middle  of an exchange you know so you zap their  pawn they're gonna zap your pawn or  maybe even zap your king okay and you  don't want to evaluate in the middle  there and so the idea of quiescence is  that when you're done with the regular  search you just play off moves that  involve captures and  you evaluate the position okay so that's  quieting the position okay makes sense  another heuristic there is no move  pruning in most positions there's always  something better to do than nothing so  sitting still is usually not as good as  actually making a move okay  and so the idea is suppose that I can  forfeit my move and search a shallower  depth and I still have the best position  okay then I still am winning well then  and it generates a cutoff then why  bother exploring any other moves I  probably am in a really good position  here okay and if if I don't manage to  get a cut off from the no move then I  you know then I do the full the full  depth search okay so so the place this  is dangerous is in zoogs wang in chess  it's called zoogs along situations so  slugs along a situation where everything  is fine but if you have to make a move  you lose so usually making and having  the advantage of a move that's called  the initiative in chess and that's a  good thing you want the initiative you  want to have the extra move over your  opponent but there are some situations  where if you move you lose okay and so  you want to make sure you don't have a  Zhuge Liang situation I don't actually  know if there's a Zhuge Liang situation  in lesser chess okay so it's if somebody  comes up with one I'd be interested to  see that you know where if you had to  move I lose but if I sits there so what  in chess what they do is in the end game  that's when zoogs lines come up they  turn off the null move pruning so is  this clear what null move does for you  what what's going on there so I see some  people who are quizzical anybody want to  ask a question I see some people hide  with sort of quizzical looks on their  faces  okay the idea is for no movies my  position is so good okay that even if I  do nothing I still win okay so I don't  want to search anymore in this part of  it because you're you know I get a beta  cutoff you're not gonna let me make this  move that the move that got me here yeah  question oh did you really  yeah okay that was it okay good yeah my  move is so good I could I can do nothing  and I still win okay and so why bother  you know let me verify that without  necessary doing as deep a search so no  move is fairly cheap to do and it you  know often results in you know for in a  lot of positions for cutoff and if I  don't get a cutoff then I just do the  normal search okay this is a pretty  complicated piece of code isn't it okay  you know pretty complicated okay  there are other situations we mentioned  killers okay  there's also move extensions so typical  move extensions that people look at is  you grant an extra ply in chess if the  king is in check or for certain captures  or if you have a forced move so suppose  you have a move and it's the only move  you can make then don't count that as  apply so you can search deeper along  lines where they're forced moves that's  what they do in chess lie search s not  quite so clear what you do okay which  ones you would do extensions for okay  because it it's rare that you have just  one move in in lesser chess compared to  in in Chester where you ever by force  move it's like if you don't do this  you're going to get captured or you know  made it or what have you  okay so that may come up in less your  chest but anyway it's it's something to  think about  so the transposition table so we talk  about this as a search tree but it's  really a dag because I can get to the  same position by transposing moves you  know this guy does a this guy does B  this guy does C this guy does D I may  get it's the same thing by doing you  know a d c b I transpose those two moves  and I get to exactly the same position  okay and so I'd like not to search that  position again if I've seen it and  that's what the transposition does  there's a quality score that is in the  transposition table that tells you how  good a move you've made and the quality  is essentially the depth that you had to  search in order to establish the value  that's stored in the transposition table  and so you don't want to use something  of too low quality when you're searching  if you have to search the depth D  something of quality D minus 1 is not  good enough ok because otherwise you'll  not be searching you know the full tree  but something typically that is deeper  you know certainly you know if you if if  I'm looking at depth D and I find  something in transposition table that's  d plus 1 or D plus 2 that's great to use  that just gave me an even deeper view of  what's behind that move okay and so if  you look at the logic there you'll see  that that's how they do it it's very  tricky one of the things that's tricky  is when you find a mate how you store  that in the transposition table and  you'll see there's special code for  handling mate positions and the reason  is because what you're interested in  doing when you find a mate is knowing  your distance to mate but not from that  position the distance our mate from the  root of your search okay so for example  [Music]  you know let's say this is the root of  my searched and I searched down to a  given point and now I do a lookup in the  transposition table and I discovered  that there's a mate in five here okay  and this is let's say let's I have  searched nine-ply or something okay well  if I store that this is a mate in five  and I store the value for mate and five  then if it makes it up to the top here  it thinks there's a mate and five but  there isn't a mate in five there's a  mate in 14 okay so when you look it up  and you discover there's a mate and five  in the table you have to do a little bit  of a calculation to translate that into  being a mate in in 14 as the value  that's going to be used here rather than  a mate in five does that make sense so  you'll see that's the logic that's in  there for dealing with mates so mate  basically takes a very large number way  larger than all the material on the  board and then it subtracts what your  number of ply is to get to get to me so  some big number  you know I don't know 32,000 or  something minus the depth to me is what  you is what is how you represent a mate  you know so some very very big number  and and then minus the minus the depth  and that way you can make sure that  you're going for a mate in 13 instead of  a mate and 14 for example you know  preferring that you take the shorter  path make sense  so that's one of the things to look out  in there and there's also a lot of stuff  in there the transposition table has we  talked about caching it's actually a K  associative cache K way associative  cache and so you can a very K to see  what's the best choice of you know how  many how many entries you should have  the more entries you have the longer it  will take you to search them  on the other hand the more likely it is  that good moves stay in the cache okay  so you can decide what's what's the  right trade-off there okay so there's a  good tuning optimization there okay  questions any questions about  transposition table transmission today  is a major optimization so Zobrist  hashing so most of these we taught Helen  talked about at some level but I wanted  to sort of give a chance to have people  ask questions nobody's asking questions  came here for Q&A all you're getting is  a okay so let me explain Zobrist hashing  again a little bit okay so so Zobrist  hashing very clever idea so we have our  board right with with a bunch of pieces  on it that's probably the wrong number  of squares right that's seven by six  okay who cares okay so so the idea is  that what I'm going to do is have a  table of random numbers that I'm going  to compute in advance and this is gonna  be indexed by you know my row my column  my piece type and my orientation okay  and every different combination of row  column type and orientation corresponds  to a different random number in the  table okay okay if some sort of random  number there okay so you know so and  what my hash function is is it's the XOR  of all of the values of all the pieces  that are on this table with their  orientations okay so if I have a king  here you know that's a White King  I guess I need to know who's who's not  just what the type is I also need to  know whether it's white or black right  so the the side okay I think actually  that gets encoded somehow but in any  case yeah I think maybe in the type we  actually keep whether it's a whiter pawn  black pawn yeah it's in the type I think  is the way we actually implement it that  okay so there's white king black king  white pawn black pawn or space so if I  have a king there that corresponds to a  certain value if I end up with a pawn  here I'll say a white pawn then that  will be another one and I XOR those  together and I take the black King okay  and I XOR his position is and the you  know black pawn XOR that in and so I XOR  all these four values  that's the hash function that I use to  do things like look up things in the  transposition table now if I had to  compute this every time I change the  position you know that's if I have a lot  of pieces how many pieces I've got I've  got seven eight sixteen pieces right  there's each side has seven pawns in a  king so sixteen pieces I have to do 16 X  Wars in order to compute my hash  function so Zobrist rationing is really  a lot clever it takes advantage of that  XOR trick that I taught you in the first  now it wasn't the first it was in the  the was the second lecture yeah you know  the bit tricks lecture okay and that is  that XOR is its own inverse so if I want  to remove a piece let's say I'm going to  move this pawn okay from here to here so  I remove this piece what do I do to my  hash function to remove a piece I look  up the value for that pawn in the hash  table and I XOR that into my hash for  the table and I now have  hache for those three pieces left that  make sense so I have my hash function  which is a hash of the four things I'm  not sure you guys can see very well over  there but and I basically simply XOR it  with the hash of you know the pawn that  I removed you know from the you know of  the pawn that I removed in this case and  now I moved it to here so what do I do I  look up that one and I XOR it that value  in here for the new position near the  new pawn position and that gives me now  the the the hash for the new table with  a pawn in that position so any move that  I make I can basically update my hash  function with only two X source and XOR  czar very fast there one instruction you  know there one and they can do lots of X  ORS and stuff so this is actually very  very cheap thing to do  okay very cheap thing to do so that's  Zobrist hashing that you can keep these  these things up to date and that's  something we implemented for you that's  an optimization that was a freebie we  could have given you the hash function  you know like this and had you implement  that but this is one of the ones we gave  you as a freebie for optimization you're  not all saying thank you very much  professor leiserson  okay now in some sense I took away an  opportunity for optimisation didn't I  there okay and so in the transposition  table there are records for the Zobrist  key the score the move the quality also  a bound type whether it's upper lower or  exact because when I return something  from alpha-beta I only know a bound on  it if it's if it's greater than alpha or  less than beta and in some sense the age  of how how old is this move because as  things get older I also want to age them  out of the table  there's several aging things therein  you'll see the best move table also has  an aging process whereas every time it  updates the values gives a new value it  ages all the other values so that they  gradually disappear and aren't relevant  one of the ones that people have get  confused about is the late move  reductions  okay so called LMR okay this is the  situation where I have I'm going to do  let's say my parallel search to you know  of all my moves and the question is you  know it's again you're trying to prune  everything you can okay and so this is  the idea is which are the moves that are  more important to search deeper the ones  near the beginning of your move list if  it's sorted in best first order or the  ones towards the end of your move list  so where where is it most important to  search deeply for things that you think  are possibly the best move or the ones  that you think are the worst moves yeah  yeah it makes sense to search the best  moves right so the idea is well if  something is way down on the move  ordering list why search it as deeply  it's probably not as good a move and so  let me search it more shallow I probably  don't lose much of an opportunity to  discover that that's actually a you know  is indeed a bad move okay there's a  reason that got ordered down there and  so that's late move reduction so with a  good move ordering a beta it kind of  will either occur right away or not also  you search the first few moves normally  and then you start reducing the depth  for four moves I believe in our code we  have two numbers where we reduce by  depth one after a certain number of  moves and reduced by depth two after a  certain number of other moves those are  things that you can tune I wouldn't tune  them very much I don't think just  because and once again I could probably  be wrong here and some of those discover  oh you know if you tune it like this  it's way better okay but those are those  are that's the idea of the late move  reductions probably one of the most  important things to think about is the  move is the representation of the board  right now we represent the board as it  is okay that's a terrible representation  it's very you know time-consuming  there's sort of two major ways you can  do it oh I didn't put the other one here  okay well anyway so one of them is bit  boards here you use a 64 bit to  represent for example where all the  pawns are on the 64 squares of the board  okay and then you can use pop count and  other bit tricks to do move generation  and to implement other chess concepts  okay so if you're looking to see what  are the possible places upon can move  and you want to say can they move right  and let's say it's stored in row major  order you can just do a right shift by  one and that tells you where are all the  places that pawns those pawns could move  and now you can just pick them off one  bit at a time to generate your move list  okay  and and then you can do it that way if  you're gonna move up a thing well then  you're actually doing a shift by or down  you're doing a shift by how much by 8  right to move up or down okay if you're  storing things in row major all right  that makes sense right  so if it's it's 8 by 8 and you're  keeping a bit for each thing then if I  want to generate you know where is this  one if I shift this whole thing stored  in row major order by 8 if I shift it  right it basically puts it there okay so  moving it by you know 7 8 9 that gives  you and then shifting it by 1 gives you  the you know or minus 1 gives you this  or left by 1 and then similarly you can  do by you know shifting by 7 8 or 9 that  way and I can generate all the possible  moves so that's one way of doing move  generation he's using bit for it and  there are a lot of things for example  that you can do with bit boards in  parallel because you can say did I make  a capture here okay or let me use a bit  board to represent where the laser goes  did I make a move that is that is going  to affect the path of laser okay because  because you know one of the major  optimizations you can do is in the in  the evaluations in dealing with a laser  okay because you're spending a lot of  time stroking out Laser positions what's  the point of doing that if you made a  move of something and it didn't affect  where the laser goes why bother  you know doing it out you can just cash  what the what the value of the laser is  and there are a lot of lot more good  stuff on the chest programming wiki okay  so yeah question  [Music]  yeah you got to be careful there right  because if I do a shift to the right for  example so what will I do  I'll just do a mask to eliminate all the  things that got wrapped around okay  right so two instructions or whatever  yeah  details yes details are good though good  question  good question who's timed their program  where are the where are the  opportunities that you see for  performance engineering for a first pass  what are those expensive things what do  you have as expensive thing  sorry the marking laser path yep good  what else is is time expensive  yeah laser coverage boy that is really  expensive okay that is really expensive  so where else is expensive  what else is expensive so I would  definitely go after you know L cover  that's a that's a huge one to go after  one of the things by the way if you are  making your your changes to the code and  you're gonna change the representation  leave the old representation there take  it out at the end add the new  representation and put in assertions  that say that things are equivalent or  whatever but don't get rid of the old  stuff because you'll end up with broken  code and definitely using use things  like perfectiy to tell you whether or  not you you know made any change you  know made any changes if you touch  anything with move ordering sorry not a  move generation not move ordered move  generation so where else is expensive  yeah  sure you want to get your how much  detail do you want how it actually works  okay so what it's supposed to do is  what's supposed to do is figure out how  safe what you're the idea is I want my  laser to get closer to your King and I  want your laser not to be close to my  king  right and if I can move into positions  where my laser is closer to your King  but your laser doesn't get closer to my  King that's a good sort of thing but  when we say get the laser close you know  what happens when I've got say let me do  it this way a position like this so  here's the okay suppose I look at this  position how close does the laser get  Mike let's say I'm black here and I look  at the laser well the path of the laser  is it bounces there and goes across  right okay so I didn't get very close to  the King there but I'm one move away  from getting it pretty close because if  I just move this guy out of the way now  I've got it really quite close okay so  if I compare that to let's say a  situation where instead of the the  plants are there let's say a pawn is  here okay  now the laser is actually closer to the  King that it was in the first position  but it's a much worse position right the  first one was much better when I had the  pawns here and here because I was simply  one move away from getting it really  close and so if you use just the direct  laser thing and we did some tests on  this this turns out to be not very it  doesn't guide the program very well on  getting into situations where my laser  is getting close to your king  that makes sense so then we said well  how can we you know how should we  measure on how close the laser get so  what we said is well let's take a look  at all the possible moves from here of  one move you know and then look to see  how close we get things and the way we  did that is we said let's you know this  took actually Helen and I worked on this  really hard okay because this is a new  heuristic that we have not used in  previous years and it works great it's  just slow as anything okay but it works  well so you have to evaluate is it worth  doing something if it's really slow it  may be that you do better to use a  simpler heuristic and get deeper search  than it is spending a lot of time  evaluating but anyway we gave you one  that if you can make it go fast should  be really good so what the idea is that  what we do is we look at all the  different paths from moving any piece  want you know one piece and look at the  paths of the laser and what we do is we  go we count one for every position that  we go away 2 3 4 5 6 7 etc we actually  add an extra value if we bounce off in  an opponent's how much do we add Helen  do we add one or two if we bounce off an  opponent I think - yeah so if this is an  opponent's pawn we would basically go  from 3 to 5 6 7 you know 8 9 and we  basically do that for all the moves and  the way we combine them is we take the  minimum number that I can get there ok  what's the what's the cheapest way I can  get to every every square that the laser  goes so we take all the different moves  we stroke them out and we take the  minimum so if I have another path let's  say by turning the king it will go this  way and there's a you know there's  another one here that's my pawn ok then  I may get there in 1 2 3 4 5 6 7 and so  then this would become a value of 7 here  okay so that's the first thing is we  basically get a map of you know how  close are things and the second thing we  do is say well how valuable is it to  have these these you know to be near the  particular King and then what we ended  up with is something where if I look at  the if I look at the distance that I am  away let's say this one for example is  one row in one column away this is zero  row zero columns we use I think it's one  over the row plus one times one over the  column plus one okay as a multiplier to  wait how good it is that we've been  close to the king and we invert these  values so that they're you know so the  closer you you know so that it's better  to have a smaller number there and we  add them up okay and that basically  gives we I don't quite add them up we  actually no I'm sorry what we do is we  look at these things as a fraction of my  shortest path distance sorry that's what  we did yes I'm sorry that is a previous  sourcing so so here for example let's  say this is the nine the shortest way of  getting there is one two three four five  six seven eight  so this would actually when we do the  conversion this would we give a value of  eight ninths for being in that square  okay so we didn't get it where's this  one the shortest path distance is 7  because of this path and you can get  there and seven this would have a value  of one so this is better you know and so  we do that for all the squares we get a  fraction of how much do we do and then  we wait it by something like this which  falls away quickly but it's important in  heuristics like this to have a smooth  path so that you can get things closer  if I made all these things be 0 it  wouldn't know that it could you know  move up and get a little bit better you  know in the second or third order  you know digit to get closer and so then  we add it all up and that ends up being  a number we discovered that sort of in  the range of  you know you know like what did we  figure the range was there there's like  you know up to like four or so right  something something like four on average  you know would be the maximum value  would be and then we said okay then we  have this magic constant multiplier that  you can play with that says okay let's  represent that as a fraction of a pawn  how much is that worth and we multiply  by that value okay so that's what we're  doing okay so that it makes it so that  we do tend to move the laser our laser  closer we subtract how close the  opponent can get from us and then we say  that's the evaluation now if you have a  better way of determining whether a  laser is getting close than this one  that's cheaper to compute that's good  okay for first pass I would just simply  try to make this go fast because for  many of the moves that you're going to  look at is going to be moving this pawn  you know to there nothing changed  there's no point in there stroking this  out and cakung are the minimum etc  because i didn't touch the laser path  things are gonna touch a laser path are  things that you move on or off the path  so why bother okay and so if there's a  clever way of hashing this you know so  that you're caching it so that you can  you know do things that's good too  okay any questions about other things  that could be done so another place I'll  tell you that's a good idea to look at  is especially once you get this  heuristic operate a little bit faster is  the sorting of moves okay is is actually  pretty expensive  okay what's the you know once you figure  out here all the moves there are a lot  of moves now you go through and you do a  sort and if you think about it you know  in a best move order tree sometimes you  only explore one of those so why'd you  bother sorting all of them okay on the  other hand sometimes you do need to sort  all of them so how you optimize that  sort so that you don't waste time  sorting stuff that you're not going to  actually ever explore  another opportunity for optimization  okay yeah question the moves are sorted  by these things like that there there's  a sort key in there that represents a  whole bunch of things like whether it  was a killer if it's a killer or it's a  transposition table value gets a very  big value if it's a statistical thing  that the history table says in  historically this has been a pretty good  move then you get another value and then  exchanges get ordered and so forth so  there's a bunch of things that are you  know where where you end up with  information and then it sorts those  things okay so that's actually kind of  complicated logic in terms of what the  actual values are that are used there  but the idea is you know here all the  moves now let's figure out what's the  best our best guess as to what they are  most of the good moves are coming out of  the transposition table right  but sometimes the transposition table  you know it's not it says it's not a  very good move and there may be a better  move okay so the quality of what you get  out of transposition table is good but  doesn't mean you know that that's always  the best move okay yeah any other  questions about what should be worked on  let me just make sure I okay go ahead  where does the sorting happen that  happens at the I think at the move  generation is that where it is is the  sort in the move generation or is it in  search I think it's in search I think  you're right  search calls move generation and then  sorts it okay I think that's right  yeah question  [Music]  very tiny board implementation yes well  as I say keep the old one around well I  you use the old one while you're still  getting the new one right okay it's also  good to be able to go from old  representation to new representation and  having conversion functions okay but  yeah making representation changes you  know painful painful and is it's  probably boy if there was one tool I  would love that would automate stuff  like that that would be it you know to  be able to change representations of  things and still have things go well I  should have mentioned by the way the  other representation besides bit board  another one is just to keep a list of  the pieces and their positions okay  because that's going to be smaller than  keeping this great big board okay then  you keep this list sorted or do you not  keep it sorted and do you but the  advantage of that is particularly as the  game goes on that list gets shorter and  shorter okay and so so you're you know  if you're doing less and manipulating  the the board position that's a that's  generally a good thing so for the board  reputation my recommendation is to  refactor all the board axes into a  function and then you would still use  the old representation in the function  then you can validate that you refactor  everything correctly and then you can  easily change the board reputation to  whatever you want and keep changing it  without having to do a lot of refactor  off that for that so find all the board  accesses and refactor that to a function  call before you modify it because the  you know the compiler will inline that  stuff so you know putting in a function  call is not necessarily a bad idea and  if it doesn't you can put it in a macro  and it'll be effectively the same thing  so great idea great idea okay yeah  question  testing testing testing you know if you  can test and know that when you make a  change it's correct you know and that  you're not that's probably the number  one hole that people go into is they  don't test adequately so having good  test infrastructure you know it's good  yeah  [Music]  as long as you're doing it  deterministically yes that's because  there's a randomness thing you turn off  the ran there's a little variable you  can set the variable to be not random  and then it will be yes it's run  serially and risk can give her the mic  [Music]  yeah juice  yeah I got out of the opening phone I  said the  [Music]  yeah it shouldn't be if you run serially  the only things that you know should be  doing things as if you're doing I mean  it should be deterministic if you turn  off the the random number generator so  as I say that's put in so that it will  behave you know not predictably but  that's exactly the kind of thing you  want to find right at the beginning so  as exactly things find out all the ways  of making it deterministic and that  would be really important for beta-2  okay so Thursday okay we have you know  John Bentley legend of the field  okay opportunity to meet a celebrity  okay bring friends okay he's going to  give a great lecture  you
 

3
00:00:05,769 --> 00:00:08,019

 

4
00:00:08,019 --> 00:00:09,850

 

5
00:00:09,850 --> 00:00:10,930

 

6
00:00:10,930 --> 00:00:13,120

 

7
00:00:13,120 --> 00:00:15,160

 

8
00:00:15,160 --> 00:00:24,390

 

9
00:00:24,390 --> 00:00:30,130

 

10
00:00:30,130 --> 00:00:33,220

 

11
00:00:33,220 --> 00:00:38,670

 

12
00:00:38,670 --> 00:00:42,130

 

13
00:00:42,130 --> 00:00:45,790

 

14
00:00:45,790 --> 00:00:48,010

 

15
00:00:48,010 --> 00:00:49,990

 

16
00:00:49,990 --> 00:00:52,540

 

17
00:00:52,540 --> 00:00:55,229

 

18
00:00:55,229 --> 00:00:58,990

 

19
00:00:58,990 --> 00:01:01,090

 

20
00:01:01,090 --> 00:01:02,710

 

21
00:01:02,710 --> 00:01:05,109

 

22
00:01:05,109 --> 00:01:08,040

 

23
00:01:08,040 --> 00:01:12,040

 

24
00:01:12,040 --> 00:01:16,210

 

25
00:01:16,210 --> 00:01:19,510

 

26
00:01:19,510 --> 00:01:21,550

 

27
00:01:21,550 --> 00:01:25,330

 

28
00:01:25,330 --> 00:01:27,280

 

29
00:01:27,280 --> 00:01:29,649

 

30
00:01:29,649 --> 00:01:33,100

 

31
00:01:33,100 --> 00:01:34,750

 

32
00:01:34,750 --> 00:01:36,070

 

33
00:01:36,070 --> 00:01:37,480

 

34
00:01:37,480 --> 00:01:40,300

 

35
00:01:40,300 --> 00:01:41,530

 

36
00:01:41,530 --> 00:01:44,410

 

37
00:01:44,410 --> 00:01:47,050

 

38
00:01:47,050 --> 00:01:50,260

 

39
00:01:50,260 --> 00:01:53,249

 

40
00:01:53,249 --> 00:01:56,800

 

41
00:01:56,800 --> 00:02:00,370

 

42
00:02:00,370 --> 00:02:02,230

 

43
00:02:02,230 --> 00:02:04,270

 

44
00:02:04,270 --> 00:02:07,780

 

45
00:02:07,780 --> 00:02:09,969

 

46
00:02:09,969 --> 00:02:12,970

 

47
00:02:12,970 --> 00:02:17,199

 

48
00:02:17,199 --> 00:02:21,430

 

49
00:02:21,430 --> 00:02:23,229

 

50
00:02:23,229 --> 00:02:26,590

 

51
00:02:26,590 --> 00:02:29,360

 

52
00:02:29,360 --> 00:02:31,250

 

53
00:02:31,250 --> 00:02:34,520

 

54
00:02:34,520 --> 00:02:39,199

 

55
00:02:39,199 --> 00:02:42,760

 

56
00:02:42,760 --> 00:02:46,699

 

57
00:02:46,699 --> 00:02:48,470

 

58
00:02:48,470 --> 00:02:51,380

 

59
00:02:51,380 --> 00:02:54,140

 

60
00:02:54,140 --> 00:02:56,930

 

61
00:02:56,930 --> 00:02:59,750

 

62
00:02:59,750 --> 00:03:05,860

 

63
00:03:05,860 --> 00:03:08,540

 

64
00:03:08,540 --> 00:03:12,170

 

65
00:03:12,170 --> 00:03:16,670

 

66
00:03:16,670 --> 00:03:19,449

 

67
00:03:19,449 --> 00:03:22,220

 

68
00:03:22,220 --> 00:03:24,620

 

69
00:03:24,620 --> 00:03:28,309

 

70
00:03:28,309 --> 00:03:31,009

 

71
00:03:31,009 --> 00:03:34,729

 

72
00:03:34,729 --> 00:03:38,690

 

73
00:03:38,690 --> 00:03:44,740

 

74
00:03:44,740 --> 00:04:04,089

 

75
00:04:04,089 --> 00:04:07,869

 

76
00:04:07,869 --> 00:04:14,080

 

77
00:04:14,080 --> 00:04:15,610

 

78
00:04:15,610 --> 00:04:15,620

 

79
00:04:15,620 --> 00:04:16,029


80
00:04:16,029 --> 00:04:19,870

 

81
00:04:19,870 --> 00:04:22,600

 

82
00:04:22,600 --> 00:04:24,340

 

83
00:04:24,340 --> 00:04:26,590

 

84
00:04:26,590 --> 00:04:28,270

 

85
00:04:28,270 --> 00:04:31,510

 

86
00:04:31,510 --> 00:04:34,450

 

87
00:04:34,450 --> 00:04:42,600

 

88
00:04:42,600 --> 00:04:44,830

 

89
00:04:44,830 --> 00:04:46,749

 

90
00:04:46,749 --> 00:04:49,570

 

91
00:04:49,570 --> 00:04:50,860

 

92
00:04:50,860 --> 00:04:52,749

 

93
00:04:52,749 --> 00:04:54,909

 

94
00:04:54,909 --> 00:04:58,990

 

95
00:04:58,990 --> 00:05:00,909

 

96
00:05:00,909 --> 00:05:02,950

 

97
00:05:02,950 --> 00:05:10,240

 

98
00:05:10,240 --> 00:05:16,830

 

99
00:05:16,830 --> 00:05:16,840

 

100
00:05:16,840 --> 00:05:20,659


101
00:05:20,659 --> 00:05:23,239

 

102
00:05:23,239 --> 00:05:27,170

 

103
00:05:27,170 --> 00:05:30,589

 

104
00:05:30,589 --> 00:05:33,679

 

105
00:05:33,679 --> 00:05:37,040

 

106
00:05:37,040 --> 00:05:40,159

 

107
00:05:40,159 --> 00:05:43,429

 

108
00:05:43,429 --> 00:05:46,010

 

109
00:05:46,010 --> 00:05:49,070

 

110
00:05:49,070 --> 00:05:51,439

 

111
00:05:51,439 --> 00:05:59,439

 

112
00:05:59,439 --> 00:06:04,700

 

113
00:06:04,700 --> 00:06:09,830

 

114
00:06:09,830 --> 00:06:13,399

 

115
00:06:13,399 --> 00:06:15,730

 

116
00:06:15,730 --> 00:06:19,939

 

117
00:06:19,939 --> 00:06:24,230

 

118
00:06:24,230 --> 00:06:25,939

 

119
00:06:25,939 --> 00:06:28,700

 

120
00:06:28,700 --> 00:06:32,360

 

121
00:06:32,360 --> 00:06:36,230

 

122
00:06:36,230 --> 00:06:38,089

 

123
00:06:38,089 --> 00:06:40,760

 

124
00:06:40,760 --> 00:06:42,110

 

125
00:06:42,110 --> 00:06:43,999

 

126
00:06:43,999 --> 00:06:47,809

 

127
00:06:47,809 --> 00:06:51,110

 

128
00:06:51,110 --> 00:06:54,559

 

129
00:06:54,559 --> 00:06:56,899

 

130
00:06:56,899 --> 00:07:00,829

 

131
00:07:00,829 --> 00:07:03,350

 

132
00:07:03,350 --> 00:07:05,990

 

133
00:07:05,990 --> 00:07:10,480

 

134
00:07:10,480 --> 00:07:12,170

 

135
00:07:12,170 --> 00:07:13,969

 

136
00:07:13,969 --> 00:07:17,300

 

137
00:07:17,300 --> 00:07:19,339

 

138
00:07:19,339 --> 00:07:21,709

 

139
00:07:21,709 --> 00:07:25,429

 

140
00:07:25,429 --> 00:07:28,730

 

141
00:07:28,730 --> 00:07:33,490

 

142
00:07:33,490 --> 00:07:35,689

 

143
00:07:35,689 --> 00:07:37,580

 

144
00:07:37,580 --> 00:07:41,120

 

145
00:07:41,120 --> 00:07:45,469

 

146
00:07:45,469 --> 00:07:48,790

 

147
00:07:48,790 --> 00:07:51,589

 

148
00:07:51,589 --> 00:07:53,870

 

149
00:07:53,870 --> 00:07:58,010

 

150
00:07:58,010 --> 00:08:00,170

 

151
00:08:00,170 --> 00:08:03,050

 

152
00:08:03,050 --> 00:08:05,480

 

153
00:08:05,480 --> 00:08:11,390

 

154
00:08:11,390 --> 00:08:13,850

 

155
00:08:13,850 --> 00:08:19,490

 

156
00:08:19,490 --> 00:08:22,909

 

157
00:08:22,909 --> 00:08:25,850

 

158
00:08:25,850 --> 00:08:28,700

 

159
00:08:28,700 --> 00:08:30,170

 

160
00:08:30,170 --> 00:08:32,089

 

161
00:08:32,089 --> 00:08:33,380

 

162
00:08:33,380 --> 00:08:36,170

 

163
00:08:36,170 --> 00:08:39,500

 

164
00:08:39,500 --> 00:08:41,750

 

165
00:08:41,750 --> 00:08:45,470

 

166
00:08:45,470 --> 00:08:49,460

 

167
00:08:49,460 --> 00:08:57,429

 

168
00:08:57,429 --> 00:08:59,589

 

169
00:08:59,589 --> 00:09:01,809

 

170
00:09:01,809 --> 00:09:10,169

 

171
00:09:10,169 --> 00:09:12,669

 

172
00:09:12,669 --> 00:09:15,969

 

173
00:09:15,969 --> 00:09:18,339

 

174
00:09:18,339 --> 00:09:21,549

 

175
00:09:21,549 --> 00:09:24,719

 

176
00:09:24,719 --> 00:09:27,729

 

177
00:09:27,729 --> 00:09:27,739

 

178
00:09:27,739 --> 00:09:28,210


179
00:09:28,210 --> 00:09:30,639

 

180
00:09:30,639 --> 00:09:33,549

 

181
00:09:33,549 --> 00:09:35,769

 

182
00:09:35,769 --> 00:09:37,869

 

183
00:09:37,869 --> 00:09:39,669

 

184
00:09:39,669 --> 00:09:45,249

 

185
00:09:45,249 --> 00:09:48,279

 

186
00:09:48,279 --> 00:09:55,419

 

187
00:09:55,419 --> 00:09:57,309

 

188
00:09:57,309 --> 00:10:00,549

 

189
00:10:00,549 --> 00:10:04,779

 

190
00:10:04,779 --> 00:10:08,529

 

191
00:10:08,529 --> 00:10:10,469

 

192
00:10:10,469 --> 00:10:12,519

 

193
00:10:12,519 --> 00:10:15,599

 

194
00:10:15,599 --> 00:10:18,729

 

195
00:10:18,729 --> 00:10:21,369

 

196
00:10:21,369 --> 00:10:24,279

 

197
00:10:24,279 --> 00:10:28,269

 

198
00:10:28,269 --> 00:10:29,559

 

199
00:10:29,559 --> 00:10:34,629

 

200
00:10:34,629 --> 00:10:37,569

 

201
00:10:37,569 --> 00:10:42,849

 

202
00:10:42,849 --> 00:10:45,549

 

203
00:10:45,549 --> 00:10:47,739

 

204
00:10:47,739 --> 00:10:47,749

 

205
00:10:47,749 --> 00:10:52,910


206
00:10:52,910 --> 00:10:59,460

 

207
00:10:59,460 --> 00:10:59,470

 

208
00:10:59,470 --> 00:11:02,030


209
00:11:02,030 --> 00:11:02,040

 

210
00:11:02,040 --> 00:11:08,680


211
00:11:08,680 --> 00:11:14,530

 

212
00:11:14,530 --> 00:11:18,430

 

213
00:11:18,430 --> 00:11:20,920

 

214
00:11:20,920 --> 00:11:23,110

 

215
00:11:23,110 --> 00:11:25,869

 

216
00:11:25,869 --> 00:11:29,139

 

217
00:11:29,139 --> 00:11:31,780

 

218
00:11:31,780 --> 00:11:35,309

 

219
00:11:35,309 --> 00:11:37,949

 

220
00:11:37,949 --> 00:11:39,879

 

221
00:11:39,879 --> 00:11:41,379

 

222
00:11:41,379 --> 00:11:43,840

 

223
00:11:43,840 --> 00:11:46,269

 

224
00:11:46,269 --> 00:11:48,579

 

225
00:11:48,579 --> 00:11:52,389

 

226
00:11:52,389 --> 00:11:54,660

 

227
00:11:54,660 --> 00:11:58,949

 

228
00:11:58,949 --> 00:12:03,639

 

229
00:12:03,639 --> 00:12:06,460

 

230
00:12:06,460 --> 00:12:08,319

 

231
00:12:08,319 --> 00:12:10,090

 

232
00:12:10,090 --> 00:12:13,120

 

233
00:12:13,120 --> 00:12:15,550

 

234
00:12:15,550 --> 00:12:17,639

 

235
00:12:17,639 --> 00:12:20,290

 

236
00:12:20,290 --> 00:12:23,860

 

237
00:12:23,860 --> 00:12:27,009

 

238
00:12:27,009 --> 00:12:28,480

 

239
00:12:28,480 --> 00:12:30,340

 

240
00:12:30,340 --> 00:12:36,639

 

241
00:12:36,639 --> 00:12:39,189

 

242
00:12:39,189 --> 00:12:41,019

 

243
00:12:41,019 --> 00:12:42,879

 

244
00:12:42,879 --> 00:12:45,249

 

245
00:12:45,249 --> 00:12:48,429

 

246
00:12:48,429 --> 00:12:51,730

 

247
00:12:51,730 --> 00:12:58,170

 

248
00:12:58,170 --> 00:13:00,249

 

249
00:13:00,249 --> 00:13:03,549

 

250
00:13:03,549 --> 00:13:08,079

 

251
00:13:08,079 --> 00:13:10,689

 

252
00:13:10,689 --> 00:13:12,189

 

253
00:13:12,189 --> 00:13:14,949

 

254
00:13:14,949 --> 00:13:18,460

 

255
00:13:18,460 --> 00:13:22,030

 

256
00:13:22,030 --> 00:13:24,370

 

257
00:13:24,370 --> 00:13:27,250

 

258
00:13:27,250 --> 00:13:29,980

 

259
00:13:29,980 --> 00:13:33,040

 

260
00:13:33,040 --> 00:13:35,020

 

261
00:13:35,020 --> 00:13:40,480

 

262
00:13:40,480 --> 00:13:41,800

 

263
00:13:41,800 --> 00:13:44,350

 

264
00:13:44,350 --> 00:13:46,360

 

265
00:13:46,360 --> 00:13:47,980

 

266
00:13:47,980 --> 00:13:50,170

 

267
00:13:50,170 --> 00:13:51,820

 

268
00:13:51,820 --> 00:13:53,560

 

269
00:13:53,560 --> 00:13:55,960

 

270
00:13:55,960 --> 00:13:58,420

 

271
00:13:58,420 --> 00:14:00,520

 

272
00:14:00,520 --> 00:14:02,950

 

273
00:14:02,950 --> 00:14:04,810

 

274
00:14:04,810 --> 00:14:07,870

 

275
00:14:07,870 --> 00:14:10,660

 

276
00:14:10,660 --> 00:14:13,930

 

277
00:14:13,930 --> 00:14:17,110

 

278
00:14:17,110 --> 00:14:18,670

 

279
00:14:18,670 --> 00:14:20,980

 

280
00:14:20,980 --> 00:14:24,550

 

281
00:14:24,550 --> 00:14:27,970

 

282
00:14:27,970 --> 00:14:30,670

 

283
00:14:30,670 --> 00:14:35,740

 

284
00:14:35,740 --> 00:14:37,930

 

285
00:14:37,930 --> 00:14:40,150

 

286
00:14:40,150 --> 00:14:42,190

 

287
00:14:42,190 --> 00:14:45,010

 

288
00:14:45,010 --> 00:14:50,380

 

289
00:14:50,380 --> 00:14:52,090

 

290
00:14:52,090 --> 00:14:54,460

 

291
00:14:54,460 --> 00:14:56,080

 

292
00:14:56,080 --> 00:14:58,480

 

293
00:14:58,480 --> 00:15:00,160

 

294
00:15:00,160 --> 00:15:03,790

 

295
00:15:03,790 --> 00:15:05,590

 

296
00:15:05,590 --> 00:15:07,060

 

297
00:15:07,060 --> 00:15:08,890

 

298
00:15:08,890 --> 00:15:14,890

 

299
00:15:14,890 --> 00:15:20,850

 

300
00:15:20,850 --> 00:15:23,920

 

301
00:15:23,920 --> 00:15:27,130

 

302
00:15:27,130 --> 00:15:29,500

 

303
00:15:29,500 --> 00:15:33,340

 

304
00:15:33,340 --> 00:15:34,350

 

305
00:15:34,350 --> 00:15:37,410

 

306
00:15:37,410 --> 00:15:40,740

 

307
00:15:40,740 --> 00:15:48,389

 

308
00:15:48,389 --> 00:15:49,949

 

309
00:15:49,949 --> 00:15:53,850

 

310
00:15:53,850 --> 00:15:57,509

 

311
00:15:57,509 --> 00:16:00,810

 

312
00:16:00,810 --> 00:16:02,790

 

313
00:16:02,790 --> 00:16:05,670

 

314
00:16:05,670 --> 00:16:08,759

 

315
00:16:08,759 --> 00:16:14,490

 

316
00:16:14,490 --> 00:16:19,250

 

317
00:16:19,250 --> 00:16:21,300

 

318
00:16:21,300 --> 00:16:23,490

 

319
00:16:23,490 --> 00:16:25,829

 

320
00:16:25,829 --> 00:16:27,360

 

321
00:16:27,360 --> 00:16:28,620

 

322
00:16:28,620 --> 00:16:31,470

 

323
00:16:31,470 --> 00:16:32,610

 

324
00:16:32,610 --> 00:16:34,500

 

325
00:16:34,500 --> 00:16:36,960

 

326
00:16:36,960 --> 00:16:38,310

 

327
00:16:38,310 --> 00:16:41,189

 

328
00:16:41,189 --> 00:16:45,630

 

329
00:16:45,630 --> 00:16:50,670

 

330
00:16:50,670 --> 00:16:54,960

 

331
00:16:54,960 --> 00:16:58,920

 

332
00:16:58,920 --> 00:17:00,780

 

333
00:17:00,780 --> 00:17:04,439

 

334
00:17:04,439 --> 00:17:09,240

 

335
00:17:09,240 --> 00:17:11,370

 

336
00:17:11,370 --> 00:17:13,549

 

337
00:17:13,549 --> 00:17:17,909

 

338
00:17:17,909 --> 00:17:19,590

 

339
00:17:19,590 --> 00:17:23,340

 

340
00:17:23,340 --> 00:17:26,120

 

341
00:17:26,120 --> 00:17:31,490

 

342
00:17:31,490 --> 00:17:35,130

 

343
00:17:35,130 --> 00:17:38,000

 

344
00:17:38,000 --> 00:17:42,060

 

345
00:17:42,060 --> 00:17:43,980

 

346
00:17:43,980 --> 00:17:47,390

 

347
00:17:47,390 --> 00:17:49,670

 

348
00:17:49,670 --> 00:17:52,730

 

349
00:17:52,730 --> 00:17:56,450

 

350
00:17:56,450 --> 00:17:57,950

 

351
00:17:57,950 --> 00:17:59,240

 

352
00:17:59,240 --> 00:18:01,250

 

353
00:18:01,250 --> 00:18:03,320

 

354
00:18:03,320 --> 00:18:05,810

 

355
00:18:05,810 --> 00:18:09,890

 

356
00:18:09,890 --> 00:18:12,350

 

357
00:18:12,350 --> 00:18:15,950

 

358
00:18:15,950 --> 00:18:18,110

 

359
00:18:18,110 --> 00:18:19,940

 

360
00:18:19,940 --> 00:18:23,660

 

361
00:18:23,660 --> 00:18:25,490

 

362
00:18:25,490 --> 00:18:27,350

 

363
00:18:27,350 --> 00:18:29,960

 

364
00:18:29,960 --> 00:18:31,460

 

365
00:18:31,460 --> 00:18:33,320

 

366
00:18:33,320 --> 00:18:35,030

 

367
00:18:35,030 --> 00:18:37,610

 

368
00:18:37,610 --> 00:18:40,340

 

369
00:18:40,340 --> 00:18:42,980

 

370
00:18:42,980 --> 00:18:44,930

 

371
00:18:44,930 --> 00:18:47,540

 

372
00:18:47,540 --> 00:18:49,880

 

373
00:18:49,880 --> 00:18:51,560

 

374
00:18:51,560 --> 00:18:53,960

 

375
00:18:53,960 --> 00:18:55,700

 

376
00:18:55,700 --> 00:18:57,680

 

377
00:18:57,680 --> 00:19:00,500

 

378
00:19:00,500 --> 00:19:03,560

 

379
00:19:03,560 --> 00:19:04,910

 

380
00:19:04,910 --> 00:19:07,700

 

381
00:19:07,700 --> 00:19:10,370

 

382
00:19:10,370 --> 00:19:16,280

 

383
00:19:16,280 --> 00:19:18,380

 

384
00:19:18,380 --> 00:19:20,630

 

385
00:19:20,630 --> 00:19:22,730

 

386
00:19:22,730 --> 00:19:25,790

 

387
00:19:25,790 --> 00:19:27,530

 

388
00:19:27,530 --> 00:19:29,660

 

389
00:19:29,660 --> 00:19:32,960

 

390
00:19:32,960 --> 00:19:34,640

 

391
00:19:34,640 --> 00:19:37,100

 

392
00:19:37,100 --> 00:19:39,830

 

393
00:19:39,830 --> 00:19:42,410

 

394
00:19:42,410 --> 00:19:45,049

 

395
00:19:45,049 --> 00:19:47,210

 

396
00:19:47,210 --> 00:19:49,160

 

397
00:19:49,160 --> 00:19:51,140

 

398
00:19:51,140 --> 00:19:53,090

 

399
00:19:53,090 --> 00:19:56,919

 

400
00:19:56,919 --> 00:19:59,710

 

401
00:19:59,710 --> 00:20:03,039

 

402
00:20:03,039 --> 00:20:06,940

 

403
00:20:06,940 --> 00:20:09,399

 

404
00:20:09,399 --> 00:20:12,190

 

405
00:20:12,190 --> 00:20:14,139

 

406
00:20:14,139 --> 00:20:18,340

 

407
00:20:18,340 --> 00:20:20,919

 

408
00:20:20,919 --> 00:20:24,430

 

409
00:20:24,430 --> 00:20:28,619

 

410
00:20:28,619 --> 00:20:31,360

 

411
00:20:31,360 --> 00:20:33,399

 

412
00:20:33,399 --> 00:20:36,210

 

413
00:20:36,210 --> 00:20:39,549

 

414
00:20:39,549 --> 00:20:41,320

 

415
00:20:41,320 --> 00:20:43,539

 

416
00:20:43,539 --> 00:20:45,789

 

417
00:20:45,789 --> 00:20:47,950

 

418
00:20:47,950 --> 00:20:53,740

 

419
00:20:53,740 --> 00:20:56,470

 

420
00:20:56,470 --> 00:20:58,480

 

421
00:20:58,480 --> 00:21:00,480

 

422
00:21:00,480 --> 00:21:04,509

 

423
00:21:04,509 --> 00:21:07,299

 

424
00:21:07,299 --> 00:21:11,409

 

425
00:21:11,409 --> 00:21:13,269

 

426
00:21:13,269 --> 00:21:14,889

 

427
00:21:14,889 --> 00:21:16,810

 

428
00:21:16,810 --> 00:21:19,080

 

429
00:21:19,080 --> 00:21:21,490

 

430
00:21:21,490 --> 00:21:23,919

 

431
00:21:23,919 --> 00:21:25,570

 

432
00:21:25,570 --> 00:21:27,700

 

433
00:21:27,700 --> 00:21:29,950

 

434
00:21:29,950 --> 00:21:31,720

 

435
00:21:31,720 --> 00:21:34,360

 

436
00:21:34,360 --> 00:21:38,310

 

437
00:21:38,310 --> 00:21:42,700

 

438
00:21:42,700 --> 00:21:44,980

 

439
00:21:44,980 --> 00:21:47,499

 

440
00:21:47,499 --> 00:21:50,499

 

441
00:21:50,499 --> 00:21:53,560

 

442
00:21:53,560 --> 00:21:55,299

 

443
00:21:55,299 --> 00:21:57,999

 

444
00:21:57,999 --> 00:22:00,730

 

445
00:22:00,730 --> 00:22:02,590

 

446
00:22:02,590 --> 00:22:07,119

 

447
00:22:07,119 --> 00:22:08,080

 

448
00:22:08,080 --> 00:22:11,680

 

449
00:22:11,680 --> 00:22:13,510

 

450
00:22:13,510 --> 00:22:16,090

 

451
00:22:16,090 --> 00:22:18,070

 

452
00:22:18,070 --> 00:22:22,660

 

453
00:22:22,660 --> 00:22:27,640

 

454
00:22:27,640 --> 00:22:31,270

 

455
00:22:31,270 --> 00:22:34,180

 

456
00:22:34,180 --> 00:22:35,590

 

457
00:22:35,590 --> 00:22:39,220

 

458
00:22:39,220 --> 00:22:41,980

 

459
00:22:41,980 --> 00:22:43,930

 

460
00:22:43,930 --> 00:22:47,230

 

461
00:22:47,230 --> 00:22:48,490

 

462
00:22:48,490 --> 00:22:51,310

 

463
00:22:51,310 --> 00:22:52,090

 

464
00:22:52,090 --> 00:22:54,370

 

465
00:22:54,370 --> 00:22:56,350

 

466
00:22:56,350 --> 00:22:57,730

 

467
00:22:57,730 --> 00:23:01,120

 

468
00:23:01,120 --> 00:23:03,910

 

469
00:23:03,910 --> 00:23:06,340

 

470
00:23:06,340 --> 00:23:07,960

 

471
00:23:07,960 --> 00:23:12,730

 

472
00:23:12,730 --> 00:23:18,250

 

473
00:23:18,250 --> 00:23:20,799

 

474
00:23:20,799 --> 00:23:30,580

 

475
00:23:30,580 --> 00:23:36,430

 

476
00:23:36,430 --> 00:23:38,320

 

477
00:23:38,320 --> 00:23:40,540

 

478
00:23:40,540 --> 00:23:43,270

 

479
00:23:43,270 --> 00:23:45,610

 

480
00:23:45,610 --> 00:23:47,320

 

481
00:23:47,320 --> 00:23:50,890

 

482
00:23:50,890 --> 00:23:54,130

 

483
00:23:54,130 --> 00:23:57,270

 

484
00:23:57,270 --> 00:24:02,500

 

485
00:24:02,500 --> 00:24:05,530

 

486
00:24:05,530 --> 00:24:11,230

 

487
00:24:11,230 --> 00:24:15,340

 

488
00:24:15,340 --> 00:24:17,380

 

489
00:24:17,380 --> 00:24:18,760

 

490
00:24:18,760 --> 00:24:22,720

 

491
00:24:22,720 --> 00:24:22,730

 

492
00:24:22,730 --> 00:24:24,080


493
00:24:24,080 --> 00:24:26,900

 

494
00:24:26,900 --> 00:24:29,000

 

495
00:24:29,000 --> 00:24:31,910

 

496
00:24:31,910 --> 00:24:33,350

 

497
00:24:33,350 --> 00:24:36,470

 

498
00:24:36,470 --> 00:24:38,360

 

499
00:24:38,360 --> 00:24:40,400

 

500
00:24:40,400 --> 00:24:43,820

 

501
00:24:43,820 --> 00:24:45,290

 

502
00:24:45,290 --> 00:24:49,070

 

503
00:24:49,070 --> 00:24:50,630

 

504
00:24:50,630 --> 00:24:53,510

 

505
00:24:53,510 --> 00:24:55,810

 

506
00:24:55,810 --> 00:24:58,540

 

507
00:24:58,540 --> 00:25:00,770

 

508
00:25:00,770 --> 00:25:02,660

 

509
00:25:02,660 --> 00:25:04,700

 

510
00:25:04,700 --> 00:25:06,080

 

511
00:25:06,080 --> 00:25:08,270

 

512
00:25:08,270 --> 00:25:09,560

 

513
00:25:09,560 --> 00:25:11,540

 

514
00:25:11,540 --> 00:25:13,400

 

515
00:25:13,400 --> 00:25:15,350

 

516
00:25:15,350 --> 00:25:17,570

 

517
00:25:17,570 --> 00:25:20,120

 

518
00:25:20,120 --> 00:25:21,890

 

519
00:25:21,890 --> 00:25:24,500

 

520
00:25:24,500 --> 00:25:27,380

 

521
00:25:27,380 --> 00:25:33,440

 

522
00:25:33,440 --> 00:25:37,640

 

523
00:25:37,640 --> 00:25:40,840

 

524
00:25:40,840 --> 00:25:47,200

 

525
00:25:47,200 --> 00:25:50,990

 

526
00:25:50,990 --> 00:25:58,310

 

527
00:25:58,310 --> 00:26:00,350

 

528
00:26:00,350 --> 00:26:02,690

 

529
00:26:02,690 --> 00:26:04,490

 

530
00:26:04,490 --> 00:26:05,690

 

531
00:26:05,690 --> 00:26:08,090

 

532
00:26:08,090 --> 00:26:09,920

 

533
00:26:09,920 --> 00:26:11,750

 

534
00:26:11,750 --> 00:26:14,450

 

535
00:26:14,450 --> 00:26:17,780

 

536
00:26:17,780 --> 00:26:21,560

 

537
00:26:21,560 --> 00:26:23,840

 

538
00:26:23,840 --> 00:26:26,300

 

539
00:26:26,300 --> 00:26:28,100

 

540
00:26:28,100 --> 00:26:32,510

 

541
00:26:32,510 --> 00:26:36,140

 

542
00:26:36,140 --> 00:26:38,890

 

543
00:26:38,890 --> 00:26:41,720

 

544
00:26:41,720 --> 00:26:47,330

 

545
00:26:47,330 --> 00:26:50,090

 

546
00:26:50,090 --> 00:26:52,760

 

547
00:26:52,760 --> 00:26:55,340

 

548
00:26:55,340 --> 00:26:57,770

 

549
00:26:57,770 --> 00:27:01,340

 

550
00:27:01,340 --> 00:27:03,620

 

551
00:27:03,620 --> 00:27:05,120

 

552
00:27:05,120 --> 00:27:06,950

 

553
00:27:06,950 --> 00:27:09,190

 

554
00:27:09,190 --> 00:27:12,320

 

555
00:27:12,320 --> 00:27:15,890

 

556
00:27:15,890 --> 00:27:18,080

 

557
00:27:18,080 --> 00:27:19,730

 

558
00:27:19,730 --> 00:27:22,670

 

559
00:27:22,670 --> 00:27:25,610

 

560
00:27:25,610 --> 00:27:27,110

 

561
00:27:27,110 --> 00:27:30,980

 

562
00:27:30,980 --> 00:27:33,920

 

563
00:27:33,920 --> 00:27:38,120

 

564
00:27:38,120 --> 00:27:40,850

 

565
00:27:40,850 --> 00:27:43,790

 

566
00:27:43,790 --> 00:27:46,880

 

567
00:27:46,880 --> 00:27:49,400

 

568
00:27:49,400 --> 00:27:51,230

 

569
00:27:51,230 --> 00:27:53,600

 

570
00:27:53,600 --> 00:27:55,400

 

571
00:27:55,400 --> 00:27:57,380

 

572
00:27:57,380 --> 00:27:59,720

 

573
00:27:59,720 --> 00:28:01,520

 

574
00:28:01,520 --> 00:28:03,050

 

575
00:28:03,050 --> 00:28:04,850

 

576
00:28:04,850 --> 00:28:06,500

 

577
00:28:06,500 --> 00:28:07,970

 

578
00:28:07,970 --> 00:28:08,960

 

579
00:28:08,960 --> 00:28:11,840

 

580
00:28:11,840 --> 00:28:14,120

 

581
00:28:14,120 --> 00:28:15,830

 

582
00:28:15,830 --> 00:28:18,200

 

583
00:28:18,200 --> 00:28:20,390

 

584
00:28:20,390 --> 00:28:23,540

 

585
00:28:23,540 --> 00:28:24,740

 

586
00:28:24,740 --> 00:28:26,420

 

587
00:28:26,420 --> 00:28:29,870

 

588
00:28:29,870 --> 00:28:32,450

 

589
00:28:32,450 --> 00:28:36,150

 

590
00:28:36,150 --> 00:28:40,030

 

591
00:28:40,030 --> 00:28:42,210

 

592
00:28:42,210 --> 00:28:45,550

 

593
00:28:45,550 --> 00:28:47,380

 

594
00:28:47,380 --> 00:28:52,530

 

595
00:28:52,530 --> 00:28:56,260

 

596
00:28:56,260 --> 00:28:57,880

 

597
00:28:57,880 --> 00:28:59,050

 

598
00:28:59,050 --> 00:29:00,970

 

599
00:29:00,970 --> 00:29:02,650

 

600
00:29:02,650 --> 00:29:03,850

 

601
00:29:03,850 --> 00:29:08,020

 

602
00:29:08,020 --> 00:29:09,280

 

603
00:29:09,280 --> 00:29:11,710

 

604
00:29:11,710 --> 00:29:13,780

 

605
00:29:13,780 --> 00:29:16,750

 

606
00:29:16,750 --> 00:29:20,880

 

607
00:29:20,880 --> 00:29:24,370

 

608
00:29:24,370 --> 00:29:26,710

 

609
00:29:26,710 --> 00:29:28,930

 

610
00:29:28,930 --> 00:29:31,120

 

611
00:29:31,120 --> 00:29:33,070

 

612
00:29:33,070 --> 00:29:36,210

 

613
00:29:36,210 --> 00:29:38,830

 

614
00:29:38,830 --> 00:29:40,060

 

615
00:29:40,060 --> 00:29:43,600

 

616
00:29:43,600 --> 00:29:46,750

 

617
00:29:46,750 --> 00:29:51,100

 

618
00:29:51,100 --> 00:29:53,080

 

619
00:29:53,080 --> 00:29:55,630

 

620
00:29:55,630 --> 00:30:03,160

 

621
00:30:03,160 --> 00:30:04,480

 

622
00:30:04,480 --> 00:30:06,130

 

623
00:30:06,130 --> 00:30:10,870

 

624
00:30:10,870 --> 00:30:10,880

 

625
00:30:10,880 --> 00:30:12,180


626
00:30:12,180 --> 00:30:14,530

 

627
00:30:14,530 --> 00:30:17,680

 

628
00:30:17,680 --> 00:30:22,990

 

629
00:30:22,990 --> 00:30:25,050

 

630
00:30:25,050 --> 00:30:28,450

 

631
00:30:28,450 --> 00:30:30,340

 

632
00:30:30,340 --> 00:30:31,960

 

633
00:30:31,960 --> 00:30:35,800

 

634
00:30:35,800 --> 00:30:35,810

 

635
00:30:35,810 --> 00:30:36,190


636
00:30:36,190 --> 00:30:38,500

 

637
00:30:38,500 --> 00:30:44,830

 

638
00:30:44,830 --> 00:30:47,950

 

639
00:30:47,950 --> 00:30:51,129

 

640
00:30:51,129 --> 00:30:54,879

 

641
00:30:54,879 --> 00:30:57,759

 

642
00:30:57,759 --> 00:31:00,999

 

643
00:31:00,999 --> 00:31:02,769

 

644
00:31:02,769 --> 00:31:05,830

 

645
00:31:05,830 --> 00:31:08,919

 

646
00:31:08,919 --> 00:31:13,779

 

647
00:31:13,779 --> 00:31:16,330

 

648
00:31:16,330 --> 00:31:18,609

 

649
00:31:18,609 --> 00:31:20,619

 

650
00:31:20,619 --> 00:31:23,619

 

651
00:31:23,619 --> 00:31:25,060

 

652
00:31:25,060 --> 00:31:27,879

 

653
00:31:27,879 --> 00:31:29,769

 

654
00:31:29,769 --> 00:31:32,379

 

655
00:31:32,379 --> 00:31:33,970

 

656
00:31:33,970 --> 00:31:36,249

 

657
00:31:36,249 --> 00:31:37,899

 

658
00:31:37,899 --> 00:31:39,820

 

659
00:31:39,820 --> 00:31:41,649

 

660
00:31:41,649 --> 00:31:43,930

 

661
00:31:43,930 --> 00:31:46,509

 

662
00:31:46,509 --> 00:31:50,019

 

663
00:31:50,019 --> 00:31:52,539

 

664
00:31:52,539 --> 00:31:57,310

 

665
00:31:57,310 --> 00:32:01,239

 

666
00:32:01,239 --> 00:32:04,060

 

667
00:32:04,060 --> 00:32:06,639

 

668
00:32:06,639 --> 00:32:08,619

 

669
00:32:08,619 --> 00:32:10,330

 

670
00:32:10,330 --> 00:32:14,139

 

671
00:32:14,139 --> 00:32:16,450

 

672
00:32:16,450 --> 00:32:19,539

 

673
00:32:19,539 --> 00:32:22,119

 

674
00:32:22,119 --> 00:32:23,649

 

675
00:32:23,649 --> 00:32:25,269

 

676
00:32:25,269 --> 00:32:27,340

 

677
00:32:27,340 --> 00:32:29,649

 

678
00:32:29,649 --> 00:32:31,629

 

679
00:32:31,629 --> 00:32:32,889

 

680
00:32:32,889 --> 00:32:35,080

 

681
00:32:35,080 --> 00:32:39,249

 

682
00:32:39,249 --> 00:32:41,200

 

683
00:32:41,200 --> 00:32:44,440

 

684
00:32:44,440 --> 00:32:46,779

 

685
00:32:46,779 --> 00:32:48,310

 

686
00:32:48,310 --> 00:32:50,859

 

687
00:32:50,859 --> 00:32:53,739

 

688
00:32:53,739 --> 00:32:56,470

 

689
00:32:56,470 --> 00:32:58,539

 

690
00:32:58,539 --> 00:33:00,369

 

691
00:33:00,369 --> 00:33:00,379

 

692
00:33:00,379 --> 00:33:01,280


693
00:33:01,280 --> 00:33:03,980

 

694
00:33:03,980 --> 00:33:06,260

 

695
00:33:06,260 --> 00:33:08,510

 

696
00:33:08,510 --> 00:33:11,840

 

697
00:33:11,840 --> 00:33:14,750

 

698
00:33:14,750 --> 00:33:16,100

 

699
00:33:16,100 --> 00:33:17,299

 

700
00:33:17,299 --> 00:33:26,480

 

701
00:33:26,480 --> 00:33:28,549

 

702
00:33:28,549 --> 00:33:30,350

 

703
00:33:30,350 --> 00:33:33,620

 

704
00:33:33,620 --> 00:33:35,150

 

705
00:33:35,150 --> 00:33:38,330

 

706
00:33:38,330 --> 00:33:42,290

 

707
00:33:42,290 --> 00:33:45,350

 

708
00:33:45,350 --> 00:33:49,549

 

709
00:33:49,549 --> 00:33:51,110

 

710
00:33:51,110 --> 00:33:53,960

 

711
00:33:53,960 --> 00:33:55,580

 

712
00:33:55,580 --> 00:33:58,820

 

713
00:33:58,820 --> 00:34:01,760

 

714
00:34:01,760 --> 00:34:05,690

 

715
00:34:05,690 --> 00:34:08,720

 

716
00:34:08,720 --> 00:34:10,430

 

717
00:34:10,430 --> 00:34:12,619

 

718
00:34:12,619 --> 00:34:20,200

 

719
00:34:20,200 --> 00:34:22,820

 

720
00:34:22,820 --> 00:34:26,540

 

721
00:34:26,540 --> 00:34:29,599

 

722
00:34:29,599 --> 00:34:32,300

 

723
00:34:32,300 --> 00:34:35,450

 

724
00:34:35,450 --> 00:34:39,320

 

725
00:34:39,320 --> 00:34:41,659

 

726
00:34:41,659 --> 00:34:46,770

 

727
00:34:46,770 --> 00:34:49,960

 

728
00:34:49,960 --> 00:34:53,470

 

729
00:34:53,470 --> 00:34:55,450

 

730
00:34:55,450 --> 00:34:59,200

 

731
00:34:59,200 --> 00:35:04,780

 

732
00:35:04,780 --> 00:35:06,099

 

733
00:35:06,099 --> 00:35:07,960

 

734
00:35:07,960 --> 00:35:10,870

 

735
00:35:10,870 --> 00:35:14,079

 

736
00:35:14,079 --> 00:35:16,240

 

737
00:35:16,240 --> 00:35:18,790

 

738
00:35:18,790 --> 00:35:19,839

 

739
00:35:19,839 --> 00:35:24,280

 

740
00:35:24,280 --> 00:35:26,319

 

741
00:35:26,319 --> 00:35:29,260

 

742
00:35:29,260 --> 00:35:33,569

 

743
00:35:33,569 --> 00:35:36,849

 

744
00:35:36,849 --> 00:35:39,520

 

745
00:35:39,520 --> 00:35:43,329

 

746
00:35:43,329 --> 00:35:44,650

 

747
00:35:44,650 --> 00:35:47,589

 

748
00:35:47,589 --> 00:35:50,770

 

749
00:35:50,770 --> 00:35:55,240

 

750
00:35:55,240 --> 00:35:57,400

 

751
00:35:57,400 --> 00:36:00,730

 

752
00:36:00,730 --> 00:36:03,700

 

753
00:36:03,700 --> 00:36:05,349

 

754
00:36:05,349 --> 00:36:09,970

 

755
00:36:09,970 --> 00:36:13,150

 

756
00:36:13,150 --> 00:36:15,160

 

757
00:36:15,160 --> 00:36:17,230

 

758
00:36:17,230 --> 00:36:19,270

 

759
00:36:19,270 --> 00:36:21,579

 

760
00:36:21,579 --> 00:36:24,069

 

761
00:36:24,069 --> 00:36:27,069

 

762
00:36:27,069 --> 00:36:29,050

 

763
00:36:29,050 --> 00:36:32,020

 

764
00:36:32,020 --> 00:36:33,490

 

765
00:36:33,490 --> 00:36:36,099

 

766
00:36:36,099 --> 00:36:38,950

 

767
00:36:38,950 --> 00:36:41,500

 

768
00:36:41,500 --> 00:36:43,510

 

769
00:36:43,510 --> 00:36:45,730

 

770
00:36:45,730 --> 00:36:48,390

 

771
00:36:48,390 --> 00:36:50,650

 

772
00:36:50,650 --> 00:36:53,260

 

773
00:36:53,260 --> 00:36:56,410

 

774
00:36:56,410 --> 00:36:57,940

 

775
00:36:57,940 --> 00:36:57,950

 

776
00:36:57,950 --> 00:36:58,530


777
00:36:58,530 --> 00:37:01,890

 

778
00:37:01,890 --> 00:37:05,130

 

779
00:37:05,130 --> 00:37:08,390

 

780
00:37:08,390 --> 00:37:12,660

 

781
00:37:12,660 --> 00:37:14,400

 

782
00:37:14,400 --> 00:37:16,830

 

783
00:37:16,830 --> 00:37:20,460

 

784
00:37:20,460 --> 00:37:22,800

 

785
00:37:22,800 --> 00:37:25,530

 

786
00:37:25,530 --> 00:37:26,850

 

787
00:37:26,850 --> 00:37:29,640

 

788
00:37:29,640 --> 00:37:31,500

 

789
00:37:31,500 --> 00:37:34,770

 

790
00:37:34,770 --> 00:37:36,840

 

791
00:37:36,840 --> 00:37:39,120

 

792
00:37:39,120 --> 00:37:40,500

 

793
00:37:40,500 --> 00:37:42,450

 

794
00:37:42,450 --> 00:37:45,060

 

795
00:37:45,060 --> 00:37:47,760

 

796
00:37:47,760 --> 00:37:49,860

 

797
00:37:49,860 --> 00:37:52,440

 

798
00:37:52,440 --> 00:37:55,410

 

799
00:37:55,410 --> 00:37:57,120

 

800
00:37:57,120 --> 00:37:59,460

 

801
00:37:59,460 --> 00:38:03,000

 

802
00:38:03,000 --> 00:38:06,360

 

803
00:38:06,360 --> 00:38:09,750

 

804
00:38:09,750 --> 00:38:11,370

 

805
00:38:11,370 --> 00:38:13,260

 

806
00:38:13,260 --> 00:38:14,970

 

807
00:38:14,970 --> 00:38:17,070

 

808
00:38:17,070 --> 00:38:20,610

 

809
00:38:20,610 --> 00:38:22,920

 

810
00:38:22,920 --> 00:38:24,720

 

811
00:38:24,720 --> 00:38:27,240

 

812
00:38:27,240 --> 00:38:31,640

 

813
00:38:31,640 --> 00:38:33,900

 

814
00:38:33,900 --> 00:38:35,400

 

815
00:38:35,400 --> 00:38:37,290

 

816
00:38:37,290 --> 00:38:39,600

 

817
00:38:39,600 --> 00:38:41,490

 

818
00:38:41,490 --> 00:38:43,680

 

819
00:38:43,680 --> 00:38:46,130

 

820
00:38:46,130 --> 00:38:49,680

 

821
00:38:49,680 --> 00:38:54,420

 

822
00:38:54,420 --> 00:38:56,550

 

823
00:38:56,550 --> 00:39:00,360

 

824
00:39:00,360 --> 00:39:02,760

 

825
00:39:02,760 --> 00:39:08,370

 

826
00:39:08,370 --> 00:39:12,370

 

827
00:39:12,370 --> 00:39:16,089

 

828
00:39:16,089 --> 00:39:20,499

 

829
00:39:20,499 --> 00:39:22,150

 

830
00:39:22,150 --> 00:39:25,539

 

831
00:39:25,539 --> 00:39:27,970

 

832
00:39:27,970 --> 00:39:30,069

 

833
00:39:30,069 --> 00:39:31,660

 

834
00:39:31,660 --> 00:39:34,210

 

835
00:39:34,210 --> 00:39:35,559

 

836
00:39:35,559 --> 00:39:37,480

 

837
00:39:37,480 --> 00:39:43,440

 

838
00:39:43,440 --> 00:39:46,660

 

839
00:39:46,660 --> 00:39:48,279

 

840
00:39:48,279 --> 00:39:51,640

 

841
00:39:51,640 --> 00:39:54,849

 

842
00:39:54,849 --> 00:39:57,849

 

843
00:39:57,849 --> 00:40:03,069

 

844
00:40:03,069 --> 00:40:06,880

 

845
00:40:06,880 --> 00:40:09,009

 

846
00:40:09,009 --> 00:40:12,099

 

847
00:40:12,099 --> 00:40:14,859

 

848
00:40:14,859 --> 00:40:17,289

 

849
00:40:17,289 --> 00:40:19,029

 

850
00:40:19,029 --> 00:40:23,140

 

851
00:40:23,140 --> 00:40:25,749

 

852
00:40:25,749 --> 00:40:29,140

 

853
00:40:29,140 --> 00:40:31,329

 

854
00:40:31,329 --> 00:40:33,069

 

855
00:40:33,069 --> 00:40:35,079

 

856
00:40:35,079 --> 00:40:37,359

 

857
00:40:37,359 --> 00:40:39,549

 

858
00:40:39,549 --> 00:40:42,880

 

859
00:40:42,880 --> 00:40:44,019

 

860
00:40:44,019 --> 00:40:47,170

 

861
00:40:47,170 --> 00:40:49,210

 

862
00:40:49,210 --> 00:40:51,849

 

863
00:40:51,849 --> 00:40:53,920

 

864
00:40:53,920 --> 00:40:56,069

 

865
00:40:56,069 --> 00:40:58,660

 

866
00:40:58,660 --> 00:41:03,099

 

867
00:41:03,099 --> 00:41:05,019

 

868
00:41:05,019 --> 00:41:06,519

 

869
00:41:06,519 --> 00:41:09,700

 

870
00:41:09,700 --> 00:41:12,490

 

871
00:41:12,490 --> 00:41:14,289

 

872
00:41:14,289 --> 00:41:17,410

 

873
00:41:17,410 --> 00:41:22,569

 

874
00:41:22,569 --> 00:41:23,890

 

875
00:41:23,890 --> 00:41:25,870

 

876
00:41:25,870 --> 00:41:28,450

 

877
00:41:28,450 --> 00:41:30,100

 

878
00:41:30,100 --> 00:41:31,090

 

879
00:41:31,090 --> 00:41:33,100

 

880
00:41:33,100 --> 00:41:35,620

 

881
00:41:35,620 --> 00:41:38,530

 

882
00:41:38,530 --> 00:41:40,180

 

883
00:41:40,180 --> 00:41:41,500

 

884
00:41:41,500 --> 00:41:43,450

 

885
00:41:43,450 --> 00:41:45,340

 

886
00:41:45,340 --> 00:41:47,650

 

887
00:41:47,650 --> 00:41:50,260

 

888
00:41:50,260 --> 00:41:56,610

 

889
00:41:56,610 --> 00:42:01,120

 

890
00:42:01,120 --> 00:42:02,830

 

891
00:42:02,830 --> 00:42:04,660

 

892
00:42:04,660 --> 00:42:10,030

 

893
00:42:10,030 --> 00:42:12,040

 

894
00:42:12,040 --> 00:42:15,780

 

895
00:42:15,780 --> 00:42:19,600

 

896
00:42:19,600 --> 00:42:21,910

 

897
00:42:21,910 --> 00:42:24,160

 

898
00:42:24,160 --> 00:42:25,750

 

899
00:42:25,750 --> 00:42:27,700

 

900
00:42:27,700 --> 00:42:31,750

 

901
00:42:31,750 --> 00:42:33,730

 

902
00:42:33,730 --> 00:42:36,670

 

903
00:42:36,670 --> 00:42:42,430

 

904
00:42:42,430 --> 00:42:45,190

 

905
00:42:45,190 --> 00:42:46,830

 

906
00:42:46,830 --> 00:42:52,710

 

907
00:42:52,710 --> 00:42:55,510

 

908
00:42:55,510 --> 00:42:58,570

 

909
00:42:58,570 --> 00:43:01,000

 

910
00:43:01,000 --> 00:43:05,110

 

911
00:43:05,110 --> 00:43:06,430

 

912
00:43:06,430 --> 00:43:08,650

 

913
00:43:08,650 --> 00:43:12,130

 

914
00:43:12,130 --> 00:43:14,110

 

915
00:43:14,110 --> 00:43:15,310

 

916
00:43:15,310 --> 00:43:17,320

 

917
00:43:17,320 --> 00:43:20,770

 

918
00:43:20,770 --> 00:43:22,330

 

919
00:43:22,330 --> 00:43:25,240

 

920
00:43:25,240 --> 00:43:27,940

 

921
00:43:27,940 --> 00:43:29,890

 

922
00:43:29,890 --> 00:43:32,530

 

923
00:43:32,530 --> 00:43:34,480

 

924
00:43:34,480 --> 00:43:38,050

 

925
00:43:38,050 --> 00:43:39,510

 

926
00:43:39,510 --> 00:43:41,250

 

927
00:43:41,250 --> 00:43:42,990

 

928
00:43:42,990 --> 00:43:44,780

 

929
00:43:44,780 --> 00:43:47,100

 

930
00:43:47,100 --> 00:43:49,650

 

931
00:43:49,650 --> 00:43:50,940

 

932
00:43:50,940 --> 00:43:52,980

 

933
00:43:52,980 --> 00:43:55,440

 

934
00:43:55,440 --> 00:43:58,080

 

935
00:43:58,080 --> 00:44:00,390

 

936
00:44:00,390 --> 00:44:03,060

 

937
00:44:03,060 --> 00:44:05,010

 

938
00:44:05,010 --> 00:44:09,300

 

939
00:44:09,300 --> 00:44:12,360

 

940
00:44:12,360 --> 00:44:14,190

 

941
00:44:14,190 --> 00:44:17,190

 

942
00:44:17,190 --> 00:44:21,420

 

943
00:44:21,420 --> 00:44:23,130

 

944
00:44:23,130 --> 00:44:25,410

 

945
00:44:25,410 --> 00:44:27,090

 

946
00:44:27,090 --> 00:44:29,310

 

947
00:44:29,310 --> 00:44:31,380

 

948
00:44:31,380 --> 00:44:33,980

 

949
00:44:33,980 --> 00:44:36,510

 

950
00:44:36,510 --> 00:44:39,450

 

951
00:44:39,450 --> 00:44:43,050

 

952
00:44:43,050 --> 00:44:46,350

 

953
00:44:46,350 --> 00:44:48,510

 

954
00:44:48,510 --> 00:44:50,820

 

955
00:44:50,820 --> 00:44:52,140

 

956
00:44:52,140 --> 00:44:55,320

 

957
00:44:55,320 --> 00:44:58,020

 

958
00:44:58,020 --> 00:45:00,840

 

959
00:45:00,840 --> 00:45:04,290

 

960
00:45:04,290 --> 00:45:06,630

 

961
00:45:06,630 --> 00:45:07,950

 

962
00:45:07,950 --> 00:45:10,980

 

963
00:45:10,980 --> 00:45:12,840

 

964
00:45:12,840 --> 00:45:14,340

 

965
00:45:14,340 --> 00:45:17,700

 

966
00:45:17,700 --> 00:45:20,460

 

967
00:45:20,460 --> 00:45:26,130

 

968
00:45:26,130 --> 00:45:29,190

 

969
00:45:29,190 --> 00:45:30,570

 

970
00:45:30,570 --> 00:45:32,670

 

971
00:45:32,670 --> 00:45:36,960

 

972
00:45:36,960 --> 00:45:39,660

 

973
00:45:39,660 --> 00:45:43,050

 

974
00:45:43,050 --> 00:45:44,460

 

975
00:45:44,460 --> 00:45:46,560

 

976
00:45:46,560 --> 00:45:48,240

 

977
00:45:48,240 --> 00:45:50,040

 

978
00:45:50,040 --> 00:45:52,310

 

979
00:45:52,310 --> 00:45:57,700

 

980
00:45:57,700 --> 00:46:02,980

 

981
00:46:02,980 --> 00:46:05,780

 

982
00:46:05,780 --> 00:46:09,080

 

983
00:46:09,080 --> 00:46:11,950

 

984
00:46:11,950 --> 00:46:14,930

 

985
00:46:14,930 --> 00:46:16,730

 

986
00:46:16,730 --> 00:46:20,450

 

987
00:46:20,450 --> 00:46:26,210

 

988
00:46:26,210 --> 00:46:31,060

 

989
00:46:31,060 --> 00:46:34,670

 

990
00:46:34,670 --> 00:46:37,190

 

991
00:46:37,190 --> 00:46:39,320

 

992
00:46:39,320 --> 00:46:41,000

 

993
00:46:41,000 --> 00:46:45,170

 

994
00:46:45,170 --> 00:46:48,970

 

995
00:46:48,970 --> 00:46:52,490

 

996
00:46:52,490 --> 00:46:58,040

 

997
00:46:58,040 --> 00:47:00,950

 

998
00:47:00,950 --> 00:47:03,560

 

999
00:47:03,560 --> 00:47:06,290

 

1000
00:47:06,290 --> 00:47:08,690

 

1001
00:47:08,690 --> 00:47:12,530

 

1002
00:47:12,530 --> 00:47:13,940

 

1003
00:47:13,940 --> 00:47:15,830

 

1004
00:47:15,830 --> 00:47:18,200

 

1005
00:47:18,200 --> 00:47:20,990

 

1006
00:47:20,990 --> 00:47:22,460

 

1007
00:47:22,460 --> 00:47:25,700

 

1008
00:47:25,700 --> 00:47:27,020

 

1009
00:47:27,020 --> 00:47:28,640

 

1010
00:47:28,640 --> 00:47:30,530

 

1011
00:47:30,530 --> 00:47:33,980

 

1012
00:47:33,980 --> 00:47:35,510

 

1013
00:47:35,510 --> 00:47:37,130

 

1014
00:47:37,130 --> 00:47:39,650

 

1015
00:47:39,650 --> 00:47:41,690

 

1016
00:47:41,690 --> 00:47:43,550

 

1017
00:47:43,550 --> 00:47:49,190

 

1018
00:47:49,190 --> 00:47:51,050

 

1019
00:47:51,050 --> 00:47:52,910

 

1020
00:47:52,910 --> 00:47:54,500

 

1021
00:47:54,500 --> 00:47:57,680

 

1022
00:47:57,680 --> 00:47:59,450

 

1023
00:47:59,450 --> 00:47:59,460

 

1024
00:47:59,460 --> 00:48:00,840


1025
00:48:00,840 --> 00:48:04,230

 

1026
00:48:04,230 --> 00:48:08,550

 

1027
00:48:08,550 --> 00:48:11,820

 

1028
00:48:11,820 --> 00:48:13,230

 

1029
00:48:13,230 --> 00:48:15,270

 

1030
00:48:15,270 --> 00:48:16,770

 

1031
00:48:16,770 --> 00:48:19,440

 

1032
00:48:19,440 --> 00:48:22,140

 

1033
00:48:22,140 --> 00:48:25,560

 

1034
00:48:25,560 --> 00:48:28,110

 

1035
00:48:28,110 --> 00:48:31,160

 

1036
00:48:31,160 --> 00:48:33,960

 

1037
00:48:33,960 --> 00:48:36,210

 

1038
00:48:36,210 --> 00:48:40,860

 

1039
00:48:40,860 --> 00:48:43,260

 

1040
00:48:43,260 --> 00:48:44,760

 

1041
00:48:44,760 --> 00:48:46,020

 

1042
00:48:46,020 --> 00:48:50,340

 

1043
00:48:50,340 --> 00:48:53,520

 

1044
00:48:53,520 --> 00:48:57,780

 

1045
00:48:57,780 --> 00:49:00,300

 

1046
00:49:00,300 --> 00:49:02,460

 

1047
00:49:02,460 --> 00:49:06,780

 

1048
00:49:06,780 --> 00:49:08,430

 

1049
00:49:08,430 --> 00:49:11,070

 

1050
00:49:11,070 --> 00:49:13,500

 

1051
00:49:13,500 --> 00:49:16,110

 

1052
00:49:16,110 --> 00:49:17,460

 

1053
00:49:17,460 --> 00:49:22,500

 

1054
00:49:22,500 --> 00:49:25,800

 

1055
00:49:25,800 --> 00:49:27,840

 

1056
00:49:27,840 --> 00:49:30,180

 

1057
00:49:30,180 --> 00:49:33,090

 

1058
00:49:33,090 --> 00:49:40,900

 

1059
00:49:40,900 --> 00:49:42,940

 

1060
00:49:42,940 --> 00:49:45,610

 

1061
00:49:45,610 --> 00:49:48,400

 

1062
00:49:48,400 --> 00:49:49,750

 

1063
00:49:49,750 --> 00:49:51,670

 

1064
00:49:51,670 --> 00:49:52,870

 

1065
00:49:52,870 --> 00:49:56,290

 

1066
00:49:56,290 --> 00:49:58,600

 

1067
00:49:58,600 --> 00:50:00,910

 

1068
00:50:00,910 --> 00:50:03,970

 

1069
00:50:03,970 --> 00:50:05,380

 

1070
00:50:05,380 --> 00:50:07,000

 

1071
00:50:07,000 --> 00:50:10,420

 

1072
00:50:10,420 --> 00:50:13,000

 

1073
00:50:13,000 --> 00:50:16,780

 

1074
00:50:16,780 --> 00:50:19,030

 

1075
00:50:19,030 --> 00:50:25,300

 

1076
00:50:25,300 --> 00:50:27,660

 

1077
00:50:27,660 --> 00:50:30,310

 

1078
00:50:30,310 --> 00:50:32,800

 

1079
00:50:32,800 --> 00:50:35,520

 

1080
00:50:35,520 --> 00:50:38,470

 

1081
00:50:38,470 --> 00:50:40,450

 

1082
00:50:40,450 --> 00:50:42,250

 

1083
00:50:42,250 --> 00:50:43,810

 

1084
00:50:43,810 --> 00:50:46,630

 

1085
00:50:46,630 --> 00:50:48,520

 

1086
00:50:48,520 --> 00:50:50,140

 

1087
00:50:50,140 --> 00:50:52,810

 

1088
00:50:52,810 --> 00:50:54,250

 

1089
00:50:54,250 --> 00:50:56,590

 

1090
00:50:56,590 --> 00:50:59,350

 

1091
00:50:59,350 --> 00:51:01,720

 

1092
00:51:01,720 --> 00:51:05,160

 

1093
00:51:05,160 --> 00:51:07,990

 

1094
00:51:07,990 --> 00:51:09,610

 

1095
00:51:09,610 --> 00:51:11,140

 

1096
00:51:11,140 --> 00:51:14,310

 

1097
00:51:14,310 --> 00:51:17,500

 

1098
00:51:17,500 --> 00:51:21,430

 

1099
00:51:21,430 --> 00:51:22,930

 

1100
00:51:22,930 --> 00:51:24,550

 

1101
00:51:24,550 --> 00:51:27,460

 

1102
00:51:27,460 --> 00:51:29,380

 

1103
00:51:29,380 --> 00:51:31,090

 

1104
00:51:31,090 --> 00:51:32,680

 

1105
00:51:32,680 --> 00:51:35,200

 

1106
00:51:35,200 --> 00:51:37,720

 

1107
00:51:37,720 --> 00:51:39,340

 

1108
00:51:39,340 --> 00:51:41,770

 

1109
00:51:41,770 --> 00:51:43,570

 

1110
00:51:43,570 --> 00:51:49,550

 

1111
00:51:49,550 --> 00:51:49,560

 

1112
00:51:49,560 --> 00:51:51,570


1113
00:51:51,570 --> 00:51:53,829

 

1114
00:51:53,829 --> 00:51:55,780

 

1115
00:51:55,780 --> 00:51:59,620

 

1116
00:51:59,620 --> 00:52:01,329

 

1117
00:52:01,329 --> 00:52:05,250

 

1118
00:52:05,250 --> 00:52:08,170

 

1119
00:52:08,170 --> 00:52:12,220

 

1120
00:52:12,220 --> 00:52:14,560

 

1121
00:52:14,560 --> 00:52:17,099

 

1122
00:52:17,099 --> 00:52:20,710

 

1123
00:52:20,710 --> 00:52:23,380

 

1124
00:52:23,380 --> 00:52:24,760

 

1125
00:52:24,760 --> 00:52:29,200

 

1126
00:52:29,200 --> 00:52:30,760

 

1127
00:52:30,760 --> 00:52:32,770

 

1128
00:52:32,770 --> 00:52:34,960

 

1129
00:52:34,960 --> 00:52:40,060

 

1130
00:52:40,060 --> 00:52:41,680

 

1131
00:52:41,680 --> 00:52:44,140

 

1132
00:52:44,140 --> 00:52:46,300

 

1133
00:52:46,300 --> 00:52:49,030

 

1134
00:52:49,030 --> 00:52:51,940

 

1135
00:52:51,940 --> 00:52:53,589

 

1136
00:52:53,589 --> 00:52:56,680

 

1137
00:52:56,680 --> 00:53:01,720

 

1138
00:53:01,720 --> 00:53:02,740

 

1139
00:53:02,740 --> 00:53:06,310

 

1140
00:53:06,310 --> 00:53:10,660

 

1141
00:53:10,660 --> 00:53:13,180

 

1142
00:53:13,180 --> 00:53:16,300

 

1143
00:53:16,300 --> 00:53:20,589

 

1144
00:53:20,589 --> 00:53:22,180

 

1145
00:53:22,180 --> 00:53:25,060

 

1146
00:53:25,060 --> 00:53:27,849

 

1147
00:53:27,849 --> 00:53:29,410

 

1148
00:53:29,410 --> 00:53:31,599

 

1149
00:53:31,599 --> 00:53:33,910

 

1150
00:53:33,910 --> 00:53:38,349

 

1151
00:53:38,349 --> 00:53:42,910

 

1152
00:53:42,910 --> 00:53:45,910

 

1153
00:53:45,910 --> 00:53:48,370

 

1154
00:53:48,370 --> 00:53:52,510

 

1155
00:53:52,510 --> 00:53:56,260

 

1156
00:53:56,260 --> 00:53:58,630

 

1157
00:53:58,630 --> 00:54:01,480

 

1158
00:54:01,480 --> 00:54:03,089

 

1159
00:54:03,089 --> 00:54:05,459

 

1160
00:54:05,459 --> 00:54:08,880

 

1161
00:54:08,880 --> 00:54:11,819

 

1162
00:54:11,819 --> 00:54:14,009

 

1163
00:54:14,009 --> 00:54:18,539

 

1164
00:54:18,539 --> 00:54:20,789

 

1165
00:54:20,789 --> 00:54:23,279

 

1166
00:54:23,279 --> 00:54:30,839

 

1167
00:54:30,839 --> 00:54:32,519

 

1168
00:54:32,519 --> 00:54:34,049

 

1169
00:54:34,049 --> 00:54:36,029

 

1170
00:54:36,029 --> 00:54:37,769

 

1171
00:54:37,769 --> 00:54:40,680

 

1172
00:54:40,680 --> 00:54:47,969

 

1173
00:54:47,969 --> 00:54:53,279

 

1174
00:54:53,279 --> 00:54:56,459

 

1175
00:54:56,459 --> 00:55:05,670

 

1176
00:55:05,670 --> 00:55:10,890

 

1177
00:55:10,890 --> 00:55:13,620

 

1178
00:55:13,620 --> 00:55:18,959

 

1179
00:55:18,959 --> 00:55:20,640

 

1180
00:55:20,640 --> 00:55:23,849

 

1181
00:55:23,849 --> 00:55:27,029

 

1182
00:55:27,029 --> 00:55:31,709

 

1183
00:55:31,709 --> 00:55:40,999

 

1184
00:55:40,999 --> 00:55:44,279

 

1185
00:55:44,279 --> 00:55:47,699

 

1186
00:55:47,699 --> 00:55:49,109

 

1187
00:55:49,109 --> 00:55:52,349

 

1188
00:55:52,349 --> 00:55:59,089

 

1189
00:55:59,089 --> 00:56:03,059

 

1190
00:56:03,059 --> 00:56:07,890

 

1191
00:56:07,890 --> 00:56:09,539

 

1192
00:56:09,539 --> 00:56:12,599

 

1193
00:56:12,599 --> 00:56:15,510

 

1194
00:56:15,510 --> 00:56:19,770

 

1195
00:56:19,770 --> 00:56:21,150

 

1196
00:56:21,150 --> 00:56:22,860

 

1197
00:56:22,860 --> 00:56:28,050

 

1198
00:56:28,050 --> 00:56:29,820

 

1199
00:56:29,820 --> 00:56:32,580

 

1200
00:56:32,580 --> 00:56:34,650

 

1201
00:56:34,650 --> 00:56:36,360

 

1202
00:56:36,360 --> 00:56:39,080

 

1203
00:56:39,080 --> 00:56:42,360

 

1204
00:56:42,360 --> 00:56:48,090

 

1205
00:56:48,090 --> 00:56:50,070

 

1206
00:56:50,070 --> 00:56:52,830

 

1207
00:56:52,830 --> 00:56:56,130

 

1208
00:56:56,130 --> 00:56:58,020

 

1209
00:56:58,020 --> 00:57:01,590

 

1210
00:57:01,590 --> 00:57:05,400

 

1211
00:57:05,400 --> 00:57:09,780

 

1212
00:57:09,780 --> 00:57:11,310

 

1213
00:57:11,310 --> 00:57:13,710

 

1214
00:57:13,710 --> 00:57:15,360

 

1215
00:57:15,360 --> 00:57:18,990

 

1216
00:57:18,990 --> 00:57:22,590

 

1217
00:57:22,590 --> 00:57:25,560

 

1218
00:57:25,560 --> 00:57:27,300

 

1219
00:57:27,300 --> 00:57:34,140

 

1220
00:57:34,140 --> 00:57:36,060

 

1221
00:57:36,060 --> 00:57:39,090

 

1222
00:57:39,090 --> 00:57:41,460

 

1223
00:57:41,460 --> 00:57:44,130

 

1224
00:57:44,130 --> 00:57:46,830

 

1225
00:57:46,830 --> 00:57:49,260

 

1226
00:57:49,260 --> 00:57:51,290

 

1227
00:57:51,290 --> 00:57:57,060

 

1228
00:57:57,060 --> 00:58:00,330

 

1229
00:58:00,330 --> 00:58:04,590

 

1230
00:58:04,590 --> 00:58:06,360

 

1231
00:58:06,360 --> 00:58:10,520

 

1232
00:58:10,520 --> 00:58:14,940

 

1233
00:58:14,940 --> 00:58:19,560

 

1234
00:58:19,560 --> 00:58:22,740

 

1235
00:58:22,740 --> 00:58:26,910

 

1236
00:58:26,910 --> 00:58:28,320

 

1237
00:58:28,320 --> 00:58:32,160

 

1238
00:58:32,160 --> 00:58:35,700

 

1239
00:58:35,700 --> 00:58:38,220

 

1240
00:58:38,220 --> 00:58:39,720

 

1241
00:58:39,720 --> 00:58:42,690

 

1242
00:58:42,690 --> 00:58:45,720

 

1243
00:58:45,720 --> 00:58:48,600

 

1244
00:58:48,600 --> 00:58:50,970

 

1245
00:58:50,970 --> 00:58:53,820

 

1246
00:58:53,820 --> 00:58:59,370

 

1247
00:58:59,370 --> 00:59:05,310

 

1248
00:59:05,310 --> 00:59:08,370

 

1249
00:59:08,370 --> 00:59:13,890

 

1250
00:59:13,890 --> 00:59:16,590

 

1251
00:59:16,590 --> 00:59:18,810

 

1252
00:59:18,810 --> 00:59:21,450

 

1253
00:59:21,450 --> 00:59:23,490

 

1254
00:59:23,490 --> 00:59:25,890

 

1255
00:59:25,890 --> 00:59:27,540

 

1256
00:59:27,540 --> 00:59:28,980

 

1257
00:59:28,980 --> 00:59:31,080

 

1258
00:59:31,080 --> 00:59:34,550

 

1259
00:59:34,550 --> 00:59:36,660

 

1260
00:59:36,660 --> 00:59:38,220

 

1261
00:59:38,220 --> 00:59:41,280

 

1262
00:59:41,280 --> 00:59:43,820

 

1263
00:59:43,820 --> 00:59:46,860

 

1264
00:59:46,860 --> 00:59:48,240

 

1265
00:59:48,240 --> 00:59:52,230

 

1266
00:59:52,230 --> 00:59:53,490

 

1267
00:59:53,490 --> 00:59:57,820

 

1268
00:59:57,820 --> 01:00:00,500

 

1269
01:00:00,500 --> 01:00:02,300

 

1270
01:00:02,300 --> 01:00:07,550

 

1271
01:00:07,550 --> 01:00:09,710

 

1272
01:00:09,710 --> 01:00:13,310

 

1273
01:00:13,310 --> 01:00:15,260

 

1274
01:00:15,260 --> 01:00:18,590

 

1275
01:00:18,590 --> 01:00:20,600

 

1276
01:00:20,600 --> 01:00:23,840

 

1277
01:00:23,840 --> 01:00:26,330

 

1278
01:00:26,330 --> 01:00:29,120

 

1279
01:00:29,120 --> 01:00:31,370

 

1280
01:00:31,370 --> 01:00:32,300

 

1281
01:00:32,300 --> 01:00:34,160

 

1282
01:00:34,160 --> 01:00:36,440

 

1283
01:00:36,440 --> 01:00:40,970

 

1284
01:00:40,970 --> 01:00:43,100

 

1285
01:00:43,100 --> 01:00:45,680

 

1286
01:00:45,680 --> 01:00:49,570

 

1287
01:00:49,570 --> 01:00:53,840

 

1288
01:00:53,840 --> 01:00:55,520

 

1289
01:00:55,520 --> 01:00:55,530

 

1290
01:00:55,530 --> 01:00:56,210


1291
01:00:56,210 --> 01:01:00,110

 

1292
01:01:00,110 --> 01:01:03,050

 

1293
01:01:03,050 --> 01:01:07,700

 

1294
01:01:07,700 --> 01:01:10,880

 

1295
01:01:10,880 --> 01:01:12,680

 

1296
01:01:12,680 --> 01:01:15,110

 

1297
01:01:15,110 --> 01:01:17,930

 

1298
01:01:17,930 --> 01:01:19,940

 

1299
01:01:19,940 --> 01:01:21,890

 

1300
01:01:21,890 --> 01:01:24,590

 

1301
01:01:24,590 --> 01:01:29,980

 

1302
01:01:29,980 --> 01:01:32,690

 

1303
01:01:32,690 --> 01:01:36,170

 

1304
01:01:36,170 --> 01:01:38,240

 

1305
01:01:38,240 --> 01:01:43,690

 

1306
01:01:43,690 --> 01:01:45,700

 

1307
01:01:45,700 --> 01:01:50,349

 

1308
01:01:50,349 --> 01:01:52,240

 

1309
01:01:52,240 --> 01:01:55,960

 

1310
01:01:55,960 --> 01:02:00,069

 

1311
01:02:00,069 --> 01:02:02,440

 

1312
01:02:02,440 --> 01:02:05,140

 

1313
01:02:05,140 --> 01:02:09,280

 

1314
01:02:09,280 --> 01:02:11,829

 

1315
01:02:11,829 --> 01:02:13,510

 

1316
01:02:13,510 --> 01:02:19,210

 

1317
01:02:19,210 --> 01:02:20,500

 

1318
01:02:20,500 --> 01:02:22,299

 

1319
01:02:22,299 --> 01:02:24,309

 

1320
01:02:24,309 --> 01:02:26,140

 

1321
01:02:26,140 --> 01:02:28,630

 

1322
01:02:28,630 --> 01:02:30,549

 

1323
01:02:30,549 --> 01:02:33,130

 

1324
01:02:33,130 --> 01:02:35,049

 

1325
01:02:35,049 --> 01:02:37,120

 

1326
01:02:37,120 --> 01:02:38,890

 

1327
01:02:38,890 --> 01:02:41,049

 

1328
01:02:41,049 --> 01:02:44,410

 

1329
01:02:44,410 --> 01:02:46,120

 

1330
01:02:46,120 --> 01:02:47,880

 

1331
01:02:47,880 --> 01:02:51,700

 

1332
01:02:51,700 --> 01:02:53,260

 

1333
01:02:53,260 --> 01:02:57,819

 

1334
01:02:57,819 --> 01:03:00,069

 

1335
01:03:00,069 --> 01:03:03,010

 

1336
01:03:03,010 --> 01:03:05,049

 

1337
01:03:05,049 --> 01:03:08,349

 

1338
01:03:08,349 --> 01:03:10,720

 

1339
01:03:10,720 --> 01:03:14,500

 

1340
01:03:14,500 --> 01:03:16,930

 

1341
01:03:16,930 --> 01:03:18,940

 

1342
01:03:18,940 --> 01:03:21,730

 

1343
01:03:21,730 --> 01:03:23,349

 

1344
01:03:23,349 --> 01:03:27,210

 

1345
01:03:27,210 --> 01:03:30,280

 

1346
01:03:30,280 --> 01:03:32,140

 

1347
01:03:32,140 --> 01:03:34,049

 

1348
01:03:34,049 --> 01:03:36,609

 

1349
01:03:36,609 --> 01:03:39,220

 

1350
01:03:39,220 --> 01:03:41,950

 

1351
01:03:41,950 --> 01:03:43,390

 

1352
01:03:43,390 --> 01:03:45,700

 

1353
01:03:45,700 --> 01:03:48,160

 

1354
01:03:48,160 --> 01:03:50,620

 

1355
01:03:50,620 --> 01:03:52,960

 

1356
01:03:52,960 --> 01:03:55,690

 

1357
01:03:55,690 --> 01:03:55,700

 

1358
01:03:55,700 --> 01:03:57,600


1359
01:03:57,600 --> 01:04:00,180

 

1360
01:04:00,180 --> 01:04:01,830

 

1361
01:04:01,830 --> 01:04:03,690

 

1362
01:04:03,690 --> 01:04:09,440

 

1363
01:04:09,440 --> 01:04:13,170

 

1364
01:04:13,170 --> 01:04:15,390

 

1365
01:04:15,390 --> 01:04:16,440

 

1366
01:04:16,440 --> 01:04:22,860

 

1367
01:04:22,860 --> 01:04:24,930

 

1368
01:04:24,930 --> 01:04:27,240

 

1369
01:04:27,240 --> 01:04:29,580

 

1370
01:04:29,580 --> 01:04:32,940

 

1371
01:04:32,940 --> 01:04:36,480

 

1372
01:04:36,480 --> 01:04:40,830

 

1373
01:04:40,830 --> 01:04:43,650

 

1374
01:04:43,650 --> 01:04:45,540

 

1375
01:04:45,540 --> 01:04:48,240

 

1376
01:04:48,240 --> 01:04:51,660

 

1377
01:04:51,660 --> 01:04:53,870

 

1378
01:04:53,870 --> 01:04:56,400

 

1379
01:04:56,400 --> 01:04:58,140

 

1380
01:04:58,140 --> 01:04:59,310

 

1381
01:04:59,310 --> 01:05:01,110

 

1382
01:05:01,110 --> 01:05:02,940

 

1383
01:05:02,940 --> 01:05:06,090

 

1384
01:05:06,090 --> 01:05:08,120

 

1385
01:05:08,120 --> 01:05:12,750

 

1386
01:05:12,750 --> 01:05:21,320

 

1387
01:05:21,320 --> 01:05:23,720

 

1388
01:05:23,720 --> 01:05:26,910

 

1389
01:05:26,910 --> 01:05:29,250

 

1390
01:05:29,250 --> 01:05:31,110

 

1391
01:05:31,110 --> 01:05:33,720

 

1392
01:05:33,720 --> 01:05:36,060

 

1393
01:05:36,060 --> 01:05:37,740

 

1394
01:05:37,740 --> 01:05:41,120

 

1395
01:05:41,120 --> 01:05:43,920

 

1396
01:05:43,920 --> 01:05:45,750

 

1397
01:05:45,750 --> 01:05:48,810

 

1398
01:05:48,810 --> 01:05:52,290

 

1399
01:05:52,290 --> 01:05:58,800

 

1400
01:05:58,800 --> 01:05:58,810

 

1401
01:05:58,810 --> 01:06:01,140


1402
01:06:01,140 --> 01:06:03,190

 

1403
01:06:03,190 --> 01:06:05,950

 

1404
01:06:05,950 --> 01:06:07,720

 

1405
01:06:07,720 --> 01:06:11,680

 

1406
01:06:11,680 --> 01:06:14,140

 

1407
01:06:14,140 --> 01:06:16,150

 

1408
01:06:16,150 --> 01:06:16,160

 

1409
01:06:16,160 --> 01:06:16,690


1410
01:06:16,690 --> 01:06:19,870

 

1411
01:06:19,870 --> 01:06:19,880

 

1412
01:06:19,880 --> 01:06:20,470


1413
01:06:20,470 --> 01:06:25,180

 

1414
01:06:25,180 --> 01:06:26,950

 

1415
01:06:26,950 --> 01:06:29,580

 

1416
01:06:29,580 --> 01:06:32,140

 

1417
01:06:32,140 --> 01:06:33,970

 

1418
01:06:33,970 --> 01:06:38,600

 

1419
01:06:38,600 --> 01:06:44,300

 

1420
01:06:44,300 --> 01:06:47,680

 

1421
01:06:47,680 --> 01:06:50,960

 

1422
01:06:50,960 --> 01:06:54,050

 

1423
01:06:54,050 --> 01:06:59,200

 

1424
01:06:59,200 --> 01:07:01,340

 

1425
01:07:01,340 --> 01:07:03,310

 

1426
01:07:03,310 --> 01:07:06,820

 

1427
01:07:06,820 --> 01:07:09,670

 

1428
01:07:09,670 --> 01:07:14,900

 

1429
01:07:14,900 --> 01:07:16,910

 

1430
01:07:16,910 --> 01:07:20,960

 

1431
01:07:20,960 --> 01:07:23,210

 

1432
01:07:23,210 --> 01:07:25,160

 

1433
01:07:25,160 --> 01:07:26,780

 

1434
01:07:26,780 --> 01:07:29,720

 

1435
01:07:29,720 --> 01:07:31,430

 

1436
01:07:31,430 --> 01:07:34,100

 

1437
01:07:34,100 --> 01:07:35,600

 

1438
01:07:35,600 --> 01:07:39,020

 

1439
01:07:39,020 --> 01:07:40,970

 

1440
01:07:40,970 --> 01:07:43,190

 

1441
01:07:43,190 --> 01:07:45,410

 

1442
01:07:45,410 --> 01:07:48,760

 

1443
01:07:48,760 --> 01:07:48,770

 

1444
01:07:48,770 --> 01:07:52,100


1445
01:07:52,100 --> 01:07:54,140

 

1446
01:07:54,140 --> 01:07:58,840

 

1447
01:07:58,840 --> 01:08:04,390

 

1448
01:08:04,390 --> 01:08:08,420

 

1449
01:08:08,420 --> 01:08:12,080

 

1450
01:08:12,080 --> 01:08:16,610

 

1451
01:08:16,610 --> 01:08:19,250

 

1452
01:08:19,250 --> 01:08:19,260

 

1453
01:08:19,260 --> 01:08:20,200


1454
01:08:20,200 --> 01:08:24,560

 

1455
01:08:24,560 --> 01:08:26,450

 

1456
01:08:26,450 --> 01:08:28,730

 

1457
01:08:28,730 --> 01:08:30,650

 

1458
01:08:30,650 --> 01:08:32,900

 

1459
01:08:32,900 --> 01:08:37,160

 

1460
01:08:37,160 --> 01:08:44,900

 

1461
01:08:44,900 --> 01:08:53,300

 

1462
01:08:53,300 --> 01:08:56,180

 

1463
01:08:56,180 --> 01:09:00,080

 

1464
01:09:00,080 --> 01:09:01,340

 

1465
01:09:01,340 --> 01:09:02,930

 

1466
01:09:02,930 --> 01:09:07,580

 

1467
01:09:07,580 --> 01:09:10,070

 

1468
01:09:10,070 --> 01:09:13,400

 

1469
01:09:13,400 --> 01:09:15,740

 

1470
01:09:15,740 --> 01:09:19,010

 

1471
01:09:19,010 --> 01:09:21,440

 

1472
01:09:21,440 --> 01:09:24,980

 

1473
01:09:24,980 --> 01:09:27,440

 

1474
01:09:27,440 --> 01:09:31,700

 

1475
01:09:31,700 --> 01:09:36,230

 

1476
01:09:36,230 --> 01:09:37,910

 

1477
01:09:37,910 --> 01:09:41,690

 

1478
01:09:41,690 --> 01:09:43,370

 

1479
01:09:43,370 --> 01:09:46,160

 

1480
01:09:46,160 --> 01:09:48,860

 

1481
01:09:48,860 --> 01:09:52,790

 

1482
01:09:52,790 --> 01:09:54,560

 

1483
01:09:54,560 --> 01:09:58,220

 

1484
01:09:58,220 --> 01:10:00,380

 

1485
01:10:00,380 --> 01:10:03,230

 

1486
01:10:03,230 --> 01:10:05,140

 

1487
01:10:05,140 --> 01:10:08,200

 

1488
01:10:08,200 --> 01:10:10,600

 

1489
01:10:10,600 --> 01:10:12,760

 

1490
01:10:12,760 --> 01:10:14,620

 

1491
01:10:14,620 --> 01:10:17,070

 

1492
01:10:17,070 --> 01:10:21,280

 

1493
01:10:21,280 --> 01:10:23,530

 

1494
01:10:23,530 --> 01:10:26,260

 

1495
01:10:26,260 --> 01:10:28,390

 

1496
01:10:28,390 --> 01:10:31,750

 

1497
01:10:31,750 --> 01:10:33,070

 

1498
01:10:33,070 --> 01:10:35,470

 

1499
01:10:35,470 --> 01:10:38,530

 

1500
01:10:38,530 --> 01:10:41,230

 

1501
01:10:41,230 --> 01:10:43,690

 

1502
01:10:43,690 --> 01:10:45,730

 

1503
01:10:45,730 --> 01:10:48,640

 

1504
01:10:48,640 --> 01:10:50,590

 

1505
01:10:50,590 --> 01:10:53,110

 

1506
01:10:53,110 --> 01:10:54,700

 

1507
01:10:54,700 --> 01:10:57,220

 

1508
01:10:57,220 --> 01:10:59,470

 

1509
01:10:59,470 --> 01:11:02,580

 

1510
01:11:02,580 --> 01:11:05,650

 

1511
01:11:05,650 --> 01:11:08,140

 

1512
01:11:08,140 --> 01:11:10,900

 

1513
01:11:10,900 --> 01:11:17,020

 

1514
01:11:17,020 --> 01:11:19,480

 

1515
01:11:19,480 --> 01:11:22,000

 

1516
01:11:22,000 --> 01:11:23,650

 

1517
01:11:23,650 --> 01:11:27,970

 

1518
01:11:27,970 --> 01:11:30,280

 

1519
01:11:30,280 --> 01:11:38,290

 

1520
01:11:38,290 --> 01:11:40,630

 

1521
01:11:40,630 --> 01:11:43,300

 

1522
01:11:43,300 --> 01:11:47,140

 

1523
01:11:47,140 --> 01:11:49,210

 

1524
01:11:49,210 --> 01:11:51,100

 

1525
01:11:51,100 --> 01:11:54,100

 

1526
01:11:54,100 --> 01:11:55,990

 

1527
01:11:55,990 --> 01:11:59,290

 

1528
01:11:59,290 --> 01:12:02,170

 

1529
01:12:02,170 --> 01:12:04,150

 

1530
01:12:04,150 --> 01:12:10,630

 

1531
01:12:10,630 --> 01:12:14,440

 

1532
01:12:14,440 --> 01:12:16,929

 

1533
01:12:16,929 --> 01:12:19,479

 

1534
01:12:19,479 --> 01:12:21,790

 

1535
01:12:21,790 --> 01:12:23,500

 

1536
01:12:23,500 --> 01:12:25,779

 

1537
01:12:25,779 --> 01:12:31,239

 

1538
01:12:31,239 --> 01:12:32,739

 

1539
01:12:32,739 --> 01:12:35,529

 

1540
01:12:35,529 --> 01:12:41,979

 

1541
01:12:41,979 --> 01:12:46,299

 

1542
01:12:46,299 --> 01:12:48,939

 

1543
01:12:48,939 --> 01:12:52,029

 

1544
01:12:52,029 --> 01:12:56,469

 

1545
01:12:56,469 --> 01:13:01,540

 

1546
01:13:01,540 --> 01:13:03,549

 

1547
01:13:03,549 --> 01:13:07,060

 

1548
01:13:07,060 --> 01:13:11,049

 

1549
01:13:11,049 --> 01:13:13,000

 

1550
01:13:13,000 --> 01:13:15,339

 

1551
01:13:15,339 --> 01:13:19,359

 

1552
01:13:19,359 --> 01:13:21,159

 

1553
01:13:21,159 --> 01:13:23,290

 

1554
01:13:23,290 --> 01:13:25,149

 

1555
01:13:25,149 --> 01:13:28,359

 

1556
01:13:28,359 --> 01:13:30,520

 

1557
01:13:30,520 --> 01:13:33,520

 

1558
01:13:33,520 --> 01:13:36,189

 

1559
01:13:36,189 --> 01:13:37,929

 

1560
01:13:37,929 --> 01:13:39,250

 

1561
01:13:39,250 --> 01:13:41,799

 

1562
01:13:41,799 --> 01:13:43,629

 

1563
01:13:43,629 --> 01:13:45,929

 

1564
01:13:45,929 --> 01:13:47,979

 

1565
01:13:47,979 --> 01:13:50,409

 

1566
01:13:50,409 --> 01:13:52,600

 

1567
01:13:52,600 --> 01:13:53,770

 

1568
01:13:53,770 --> 01:13:57,879

 

1569
01:13:57,879 --> 01:13:59,319

 

1570
01:13:59,319 --> 01:14:01,750

 

1571
01:14:01,750 --> 01:14:03,640

 

1572
01:14:03,640 --> 01:14:07,379

 

1573
01:14:07,379 --> 01:14:09,669

 

1574
01:14:09,669 --> 01:14:12,009

 

1575
01:14:12,009 --> 01:14:14,620

 

1576
01:14:14,620 --> 01:14:16,270

 

1577
01:14:16,270 --> 01:14:18,879

 

1578
01:14:18,879 --> 01:14:21,250

 

1579
01:14:21,250 --> 01:14:24,939

 

1580
01:14:24,939 --> 01:14:27,009

 

1581
01:14:27,009 --> 01:14:28,839

 

1582
01:14:28,839 --> 01:14:30,270

 

1583
01:14:30,270 --> 01:14:35,040

 

1584
01:14:35,040 --> 01:14:38,640

 

1585
01:14:38,640 --> 01:14:40,640

 

1586
01:14:40,640 --> 01:14:42,839

 

1587
01:14:42,839 --> 01:14:44,520

 

1588
01:14:44,520 --> 01:14:46,649

 

1589
01:14:46,649 --> 01:14:48,660

 

1590
01:14:48,660 --> 01:14:51,270

 

1591
01:14:51,270 --> 01:14:53,219

 

1592
01:14:53,219 --> 01:14:54,779

 

1593
01:14:54,779 --> 01:14:58,290

 

1594
01:14:58,290 --> 01:15:01,080

 

1595
01:15:01,080 --> 01:15:03,569

 

1596
01:15:03,569 --> 01:15:06,180

 

1597
01:15:06,180 --> 01:15:07,830

 

1598
01:15:07,830 --> 01:15:10,049

 

1599
01:15:10,049 --> 01:15:12,810

 

1600
01:15:12,810 --> 01:15:15,600

 

1601
01:15:15,600 --> 01:15:19,109

 

1602
01:15:19,109 --> 01:15:22,259

 

1603
01:15:22,259 --> 01:15:25,290

 

1604
01:15:25,290 --> 01:15:26,640

 

1605
01:15:26,640 --> 01:15:29,250

 

1606
01:15:29,250 --> 01:15:31,830

 

1607
01:15:31,830 --> 01:15:33,239

 

1608
01:15:33,239 --> 01:15:35,580

 

1609
01:15:35,580 --> 01:15:37,709

 

1610
01:15:37,709 --> 01:15:39,899

 

1611
01:15:39,899 --> 01:15:41,689

 

1612
01:15:41,689 --> 01:15:45,870

 

1613
01:15:45,870 --> 01:15:48,089

 

1614
01:15:48,089 --> 01:15:51,680

 

1615
01:15:51,680 --> 01:15:54,209

 

1616
01:15:54,209 --> 01:15:57,180

 

1617
01:15:57,180 --> 01:15:58,560

 

1618
01:15:58,560 --> 01:16:00,060

 

1619
01:16:00,060 --> 01:16:02,370

 

1620
01:16:02,370 --> 01:16:04,489

 

1621
01:16:04,489 --> 01:16:09,450

 

1622
01:16:09,450 --> 01:16:10,700

 

1623
01:16:10,700 --> 01:16:13,259

 

1624
01:16:13,259 --> 01:16:14,669

 

1625
01:16:14,669 --> 01:16:16,680

 

1626
01:16:16,680 --> 01:16:22,560

 

1627
01:16:22,560 --> 01:16:25,500

 

1628
01:16:25,500 --> 01:16:27,419

 

1629
01:16:27,419 --> 01:16:32,040

 

1630
01:16:32,040 --> 01:16:34,620

 

1631
01:16:34,620 --> 01:16:38,640

 

1632
01:16:38,640 --> 01:16:39,899

 

1633
01:16:39,899 --> 01:16:41,459

 

1634
01:16:41,459 --> 01:16:43,310

 

1635
01:16:43,310 --> 01:16:45,760

 

1636
01:16:45,760 --> 01:16:51,439

 

1637
01:16:51,439 --> 01:16:56,089

 

1638
01:16:56,089 --> 01:16:57,950

 

1639
01:16:57,950 --> 01:16:59,450

 

1640
01:16:59,450 --> 01:17:01,970

 

1641
01:17:01,970 --> 01:17:03,560

 

1642
01:17:03,560 --> 01:17:06,680

 

1643
01:17:06,680 --> 01:17:08,300

 

1644
01:17:08,300 --> 01:17:10,040

 

1645
01:17:10,040 --> 01:17:12,979

 

1646
01:17:12,979 --> 01:17:16,040

 

1647
01:17:16,040 --> 01:17:18,290

 

1648
01:17:18,290 --> 01:17:20,030

 

1649
01:17:20,030 --> 01:17:21,770

 

1650
01:17:21,770 --> 01:17:23,990

 

1651
01:17:23,990 --> 01:17:26,990

 

1652
01:17:26,990 --> 01:17:28,720

 

1653
01:17:28,720 --> 01:17:31,640

 

1654
01:17:31,640 --> 01:17:33,350

 

1655
01:17:33,350 --> 01:17:35,419

 

1656
01:17:35,419 --> 01:17:37,459

 

1657
01:17:37,459 --> 01:17:39,950

 

1658
01:17:39,950 --> 01:17:42,770

 

1659
01:17:42,770 --> 01:17:44,990

 

1660
01:17:44,990 --> 01:17:46,490

 

1661
01:17:46,490 --> 01:17:49,459

 

1662
01:17:49,459 --> 01:17:51,020

 

1663
01:17:51,020 --> 01:17:53,450

 

1664
01:17:53,450 --> 01:18:00,080

 

1665
01:18:00,080 --> 01:18:03,200

 

1666
01:18:03,200 --> 01:18:10,050

 

1667
01:18:10,050 --> 01:18:12,660

 

1668
01:18:12,660 --> 01:18:15,210

 

1669
01:18:15,210 --> 01:18:17,010

 

1670
01:18:17,010 --> 01:18:19,680

 

1671
01:18:19,680 --> 01:18:21,660

 

1672
01:18:21,660 --> 01:18:22,320

 

1673
01:18:22,320 --> 01:18:24,060

 

1674
01:18:24,060 --> 01:18:27,090

 

1675
01:18:27,090 --> 01:18:33,960

 

1676
01:18:33,960 --> 01:18:33,970

 

1677
01:18:33,970 --> 01:18:35,750


1678
01:18:35,750 --> 01:18:46,919

 

1679
01:18:46,919 --> 01:18:51,329

 

1680
01:18:51,329 --> 01:18:53,009

 

1681
01:18:53,009 --> 01:18:56,939

 

1682
01:18:56,939 --> 01:18:58,609

 

1683
01:18:58,609 --> 01:19:00,959

 

1684
01:19:00,959 --> 01:19:04,409

 

1685
01:19:04,409 --> 01:19:07,979

 

1686
01:19:07,979 --> 01:19:10,949

 

1687
01:19:10,949 --> 01:19:13,079

 

1688
01:19:13,079 --> 01:19:14,669

 

1689
01:19:14,669 --> 01:19:16,949

 

1690
01:19:16,949 --> 01:19:18,629

 

1691
01:19:18,629 --> 01:19:20,159

 

1692
01:19:20,159 --> 01:19:21,269

 

1693
01:19:21,269 --> 01:19:23,269

 

1694
01:19:23,269 --> 01:19:26,219

 

1695
01:19:26,219 --> 01:19:31,439

 

1696
01:19:31,439 --> 01:19:32,909

 

1697
01:19:32,909 --> 01:19:36,329

 

1698
01:19:36,329 --> 01:19:38,369

 

1699
01:19:38,369 --> 01:19:40,859

 

1700
01:19:40,859 --> 01:19:42,299

 

1701
01:19:42,299 --> 01:19:44,699

 

1702
01:19:44,699 --> 01:19:48,389

 

1703
01:19:48,389 --> 01:19:51,029

 

1704
01:19:51,029 --> 01:19:53,579

 

1705
01:19:53,579 --> 01:19:55,919

 

1706
01:19:55,919 --> 01:19:58,139

 

1707
01:19:58,139 --> 01:20:00,089

 

1708
01:20:00,089 --> 01:20:02,459

 

1709
01:20:02,459 --> 01:20:04,469

 

1710
01:20:04,469 --> 01:20:06,329

 

1711
01:20:06,329 --> 01:20:08,129

 

1712
01:20:08,129 --> 01:20:09,359

 

1713
01:20:09,359 --> 01:20:11,309

 

1714
01:20:11,309 --> 01:20:12,989

 

1715
01:20:12,989 --> 01:20:15,329

 

1716
01:20:15,329 --> 01:20:16,919

 

1717
01:20:16,919 --> 01:20:20,209

 

1718
01:20:20,209 --> 01:20:22,500

 

1719
01:20:22,500 --> 01:20:26,039

 

1720
01:20:26,039 --> 01:20:28,949

 

1721
01:20:28,949 --> 01:20:30,539

 

1722
01:20:30,539 --> 01:20:32,929

 

1723
01:20:32,929 --> 01:20:39,269

 

1724
01:20:39,269 --> 01:20:39,279

 

1725
01:20:39,279 --> 01:20:44,430


1726
01:20:44,430 --> 01:20:47,760

 

1727
01:20:47,760 --> 01:20:49,080

 

1728
01:20:49,080 --> 01:20:51,810

 

1729
01:20:51,810 --> 01:20:54,330

 

1730
01:20:54,330 --> 01:20:55,950

 

1731
01:20:55,950 --> 01:20:58,440

 

1732
01:20:58,440 --> 01:21:02,220

 

1733
01:21:02,220 --> 01:21:02,230

 

1734
01:21:02,230 --> 01:21:05,450


1735
01:21:05,450 --> 01:21:05,460

 

1736
01:21:05,460 --> 01:21:10,590


1737
01:21:10,590 --> 01:21:12,520

 

1738
01:21:12,520 --> 01:21:20,350

 

1739
01:21:20,350 --> 01:21:22,390

 

1740
01:21:22,390 --> 01:21:24,070

 

1741
01:21:24,070 --> 01:21:26,590

 

1742
01:21:26,590 --> 01:21:34,420

 

1743
01:21:34,420 --> 01:21:50,320

 

1744
01:21:50,320 --> 01:21:50,330

 

1745
01:21:50,330 --> 01:21:52,740


1746
01:21:52,740 --> 01:21:56,870

 

1747
01:21:56,870 --> 01:22:00,360

 

1748
01:22:00,360 --> 01:22:07,480

 

1749
01:22:07,480 --> 01:22:07,490

 

1750
01:22:07,490 --> 01:22:09,970


1751
01:22:09,970 --> 01:22:12,740

 

1752
01:22:12,740 --> 01:22:16,040

 

1753
01:22:16,040 --> 01:22:18,980

 

1754
01:22:18,980 --> 01:22:20,510

 

1755
01:22:20,510 --> 01:22:25,190

 

1756
01:22:25,190 --> 01:22:27,200

 

1757
01:22:27,200 --> 01:22:29,090

 

1758
01:22:29,090 --> 01:22:30,380

 

1759
01:22:30,380 --> 01:22:31,820

 

1760
01:22:31,820 --> 01:22:34,430

 

1761
01:22:34,430 --> 01:22:36,440

 

1762
01:22:36,440 --> 01:22:38,380

 

1763
01:22:38,380 --> 01:22:44,330

 

1764
01:22:44,330 --> 01:22:46,790

 

1765
01:22:46,790 --> 01:22:49,510

 

1766
01:22:49,510 --> 01:22:53,330

 

1767
01:22:53,330 --> 01:23:02,040

 

1768
01:23:02,040 --> 01:23:02,050

 

1769
01:23:02,050 --> 01:23:04,110