1
00:00:01,069 --> 00:00:03,189

the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  everybody it's my pleasure once again to  welcome tibi Charl who is the author of  your taper compiler to talk about the  silk runtime system thanks Charles  I can everyone hear me in the back seem  good  okay thanks for the introduction today  I'll be talking about the silicon time  system this is pretty exciting for me  this is a lecture that's not about  compilers I get to talk about something  a little different for once it should be  a fun lecture recently as I understand  it you've been looking at storage  allocation both in the serial case as  well as the parallel case and you've  already done silk programming for a  while at this point this lecture  honestly is a bit of a non sequitur can  in terms of the overall flow of the  course and it's also an advanced topic  the silk runtime system is a pretty  complicated piece of software but  nevertheless I believe you should have  enough background to at least start to  understand and appreciate some of the  aspects of the design of the silk run  time system so that's why we're talking  about that today just to quickly recall  something that you're all I'm sure  intimately familiar with by this point  what's the program El Sol programming  all about well silk is a parallel  programming language that allows allows  you to make your software run faster  using parallel processors and to use  silk it's pretty straightforward you may  start with some serial code that runs in  some running time we'll denote that as T  sub s for certain parts of the lecture  if you wanted to run in parallel using  silk you just insert silk keywords in  choice locations for example you can  paralyze the outer loop in this matrix  multiply kernel and that will allow your  code run in time TCP on P processors an  ideally tcp should be less than T sub s  now just adding keywords is all you need  to do to tell silk to execute the  computation in parallel what is silk do  in light of those keywords at a very  high level silk and it's specifically  its runtime system takes care of the  task of scheduling and load balancing  the computation on the parallel  processors and on the multi-core system  in general  so after you've denoted logical  parallelism the program using spa  silicon silk sync and so for the soaked  scheduler Maps that computation onto the  processors and it does so dynamically at  runtime based on whatever processing  resources happen to be available and so  accuses a randomized work stealing  scheduler which guarantees that that  mapping is efficient and the execution  runs efficiently now you've all been  using the silk platform for a while and  it's basic usage you write some silk  code possibly by parallelizing ordinary  serial code you feed that to a compiler  you get a binary you run the binary on  your you run the binary with some four  to your input on a mult multi core  system you get parallel performance  today we're gonna look at how exactly  does silk work what's the magic that  goes on hidden by the boxes on this  diagram and at the very the very first  thing to note is that this picture is a  little bit the first simplification that  we're gonna break is to say it's not  really just silk source and the silk  compiler there's also a runtime system  library some loops okar tsso in case  you've seen that file or messages about  that file on your system and really it's  the compiler and the runtime library  that work together to implement silks  runtime system to do the work stealing  and do the efficient scheduling and load  balancing now we might suspect that if  you just take a look at the code that  you get when you compile a soak program  but my tell you something about how silk  works  see pseudocode for the result that  results when you compile a simple piece  of silk code it's a bit complicated I  think that's fair to say there's a lot  going on here there was one function in  the original program now there are two  there are some new variables there's  some calls to functions that look a  little bit strange there's a lot going  on in the compiled results this isn't  exactly easy to interpret or understand  and this doesn't even bring into the  picture the runtime system library the  runtime system library you can find the  source code online it's a little less  than twenty thousand lines of code it's  also kind of complicated so rather than  dive into the code directly what we're  going to do today is an attempt a  top-down approach to understanding how  the so prone type system works and some  of the design considerations so we're  gonna start by talking about some of the  required functionality that we need out  of the silic runtime system as well as  some performance considerations for how  silk how the runtime system should work  and then we'll take a look at how the  worker Dex and silk get implemented how  spawning actually works how stealing a  computation works and how  synchronization works within silk that  all sound good any questions so far this  should all be review more or less okay  so let's talk a little bit about  required functionality you see in this  picture before I hope this picture  Illustrated the execution model of a  silk program here we have everyone's  favorite exponential time fibonacci  routine parallelized using silk this is  not an efficient way to compute  Fibonacci numbers but it's a nice  didactic example for understanding  parallel computation especially the silk  model and as we saw many lectures ago  when you run this program on a given  input the execution of the program can  be modeled using as a computation dag  and this computation dag unfolds  dynamically as the program executes but  I want to stop and die take a hard look  at exactly what that dynamic execution  looks like when we've got parallel  processors and work-stealing  all coming into play so we'll stick with  this Fibonacci routine and we'll imagine  we've just got one processor on the  system to start and we're just gonna use  this one processor to execute for before  and it's gonna take some time to do it  just to make thee just to make the story  interesting so we start exceeding this  computation and that one processor is  just going to execute the Fibonacci  routine from beginning up to the silk  spawned statement as if it's ordinary  serial code because it is ordinary  serial code at this point the processor  hits the silk spawn statement what  happens what happens now anyone remember  what happens to the dag  [Music]  if bridge is downward and spawns another  process more or less the way we model  that the silks bond is of a routine fib  of n minus 1 in this case I would be fib  of 3 and so like an ordinary function  call we're going to get a brand new  frame for a fib of 3 and that's going to  have some strand that's available to  execute but the spawn is not your  typical function call it actually allows  some other computation to run in  parallel and so the way we modeled out  in this picture so we got a new frame  for fib of 3 there's a strand available  to execute there and the continuation  the green strand is now available in in  the frame for fit before but no one's  necessarily executing it it's kind of  faded in the picture so once this spawn  has occurred what's the process are  gonna do the processor is actually going  to dive in and start executing fib of 3  as if it were an ordinary function call  yes there's a strand available within  the frame of fib before but the  processor isn't gonna worry about that  strand it's just gonna say Oh fit before  calls fib of 3 gonna start computing fib  of 3 it's not good this is a processor  dies down from pink strand to pink  strand the instruction pointer for the  processor returns to the beginning of  the fib routine because we're now  calling fib once again and this process  repeats it executes the pink strand up  until the silks bond just like ordinary  old code the spawn occurs we've already  seen this picture before the spawn  allows another strand to execute in  parallel but it also creates a frame for  fib of 2 and the processor dives into  fib of 2 resetting the instruction  pointer to the beginning of fib p1  executes sup to the spawn once again we  get another strand to execute as well as  an invocation of fib of one processor  dives even further so that's fine this  is just the processor doing more or less  ordinary serial execution on this fit  routine but it's also allowing some  strands to be executed in parallel  this is the one processor situation  looks pretty good so far right and in  the fit of one case it doesn't make it  as far through the pink strand because  in fact we hit the base case but now  let's bring in some more processors  suppose that another processor finally  shows up and says I'm bored I wanted to  do some work and decides to steal some  computation it's gonna discover the  green strand in the frame fib of 4 and  PT is just going to jump in there and  start executing that strand and if we  think really hard about what this means  P 2 is another processor on the system  it has its own set of registers it has  its own instruction pointer and so  what's silk somehow allows to happen is  for P 2 to just jump right into the  middle of this fit before routine which  is already executing it just sets the  instruction pointer to point out that at  that green instruction at the call to  fib of n minus 2 and it's gonna pick up  where processor 1 left off when it  executed up to this point into before  somehow in this case it executed n minus  2 that calls fib of 2 creates a new  strand it's just an ordinary function  call it's going to descend into that new  frame it's gonna return to the beginning  of fib all that's well well and good  another processor might come along and  steal another piece of the computation  it steals another green strand so once  again this processor needs to jump into  the middle of an executing function its  instruction pointer is just going to  point at this call to fib of n minus 2  somehow it's going to have the state of  this executing function available  despite having independent registers and  needs to just start from this location  with all the parameters set  appropriately and start executing this  function as if it's an ordinary function  it calls fib of 3 minus 2 is 1 and now  these processors might start executing  in parallel P 1 might return from its  base case routine up to the parent call  if F of minus 2 and started executing  its continuation because that wasn't  stolen meanwhile P 3 to  into the execution of a fit of one and  then another step P 3 and P to make some  progress executing their computation P 2  encounters a silk spawn statement which  creates a new frame and allows another  strand X Union parallel P 3 encounters  this base case routine and says ok it's  time to return and all of that can  happen in parallel and somehow the Silk  system has to coordinate all of this and  so already but we already have one  mystery how does a processor start  executing from the middle of a running  function the running function and it's  States lived on lived on P 1 initially  and then P 2 and P 3 somehow find that  states hop into the middle of the  function and just start running that's  kind of strange how does that happen how  does this so chrono system make that  happen this is one thing to consider  another thing to consider is what  happens when we hit a sync we'll talk  about how these issues get addressed  later on but let's lay out all the  considerations upfront before we just  see how bad the problem is before we try  to solve solve it bit by bit so now  let's take this picture again and  progress it a little bit further let's  suppose that processor 3 decides to xq  the return it's gonna return to an  invocation of fib of 3 and the return  statement is this silk sync is a silk  sync statement but processor 3 can't  execute the sync because the computation  of fib of 2 in this case that's being  done by processor 1 that computation is  not done yet so the execution can't  proceed past the sync so somehow P 3  needs to say ok there's a sink statement  Bo can execute beyond this point because  specifically it's waiting on processor 1  it doesn't care what processor 2 is  doing prostitue is having a dandy time  executing fib of 2 on the other side of  the tree process of 3 shouldn't care so  processor 3 can't do something like ok  all processors need to stop get to this  point in the code  their execution can proceed no no no it  just needs to wait on Prosser what  somehow the Silk system has to allow  that fine grain synchronization to  happen in this nested pattern so how  does this silk sync wait on only the  nested sub computations within the  program how does it figure out how to do  that how does this silk run time system  implement this so that's another  consideration okay so at this point we  have three top level considerations a  single worker needs to be able to  execute this program as if it's an  orrery serial program thieves have to be  able to jump into the middle of  executing functions and pick up from  where they left off from where other  processors in the system left off syncs  have to be able to stall stall functions  appropriately based only on those  functions nested child sub computations  so we have three big considerations that  we need to pick apart so far that's not  the whole story though any ideas what  other functionality we need to worry  about for implementing the Silk system  it's kind of an open-ended question but  any thoughts  we have serial execution spawning or  stealing and sinking as top-level  concerns anyone remember some other  features of silk that the runtime system  magically makes happen correctly it's  probably been a while has you seen those  yeah the silk for loops divide and  conquer  somehow the runtime system does have to  have to implement silk fours the silk  fours end up getting implemented  internally with spawns and syncs that's  courtesy of the compiler yeah courtesy  of the compiler so we won't look too  hard at so for today but that's a that's  definitely one concern so good  observation any other thoughts sort of  low-level system details that silk needs  to implement correctly  cache coherence it actually doesn't need  to worry too much about cache coherence  although given the latest runtime you  know given latest performance numbers  I've seen from silk maybe it should  worry more about the cache but it turns  out the hardware does a pretty good job  maintaining the cache coherence protocol  itself but good guess  there's not be a tough question because  it's really just calling back memories  of old lectures I think you recently had  a quiz on this material so it's probably  safe to say that all that material has  been paged out of of your brain at this  point so I'll just spoil the fun for you  silk has a notion of a cactus stack so  we talked a little bit about processors  jumping into the middle of an executing  function and somehow having the state of  that function available one  consideration is register state but  another consideration is the stack  itself and silk supports the C C's rule  for pointers namely that children can  see pointers into parent frames but  parents can't see pointers into child  frames now each processor each worker in  a silk system needs to have its own view  of the stack but those views aren't  necessarily independent in this picture  all five processors share the same view  of the frame for function a  instantiation a and then processor is  three through five all share the same  view for the instantiation C so somehow  silk has to make all of those views  available and consistent but not quite  the same sort of consistent as we get  with cache coherence silk somehow has to  implement this cactus stack so that's  another consideration that we have to  worry about and then there's one more  kind of funny detail if we take another  look at work-stealing itself may  remember we had this picture from  several lectures ago where we have  processors on the system each house each  maintains its own deck of frames and  workers are allowed to steal steel  frames from each other but if we take a  look at how this all unfolds yes we may  have a a processor that performs a call  and that'll push another frame for a  called function onto its deck on the  bottom you may spawn and then it'll push  a spawn frame onto the bottom of its  deck but we fast forward a little bit  and we got end up with a worker  nothing to do that workers gonna go  ahead and steal picking or picking  another worker in the system at random  and it's gonna steal from the top of the  deck but it's not just gonna steal the  topmost item on the deck it's actually  going to steal a chunk of items from the  deck in particular if it selects the  third processor in this picture third  from the left this thief is gonna steal  everything up to everything through the  parent of the next spawned frame it  needs to take this whole stack of frames  and it's not clear a priori how many  frames the workers gonna have to steal  in this case but nevertheless it needs  to take all those frames and resume  execution after all that bottom is  called frame that I just stole that's  where there's a continuation with work  available to be done in parallel and so  if we think about it there are a lot of  questions that arise what's involved in  stealing frames what synchronization  does the system have to implement what  happens to the stack we it looks like we  just shifted some frames from one  processor to another but the first  processor the victim still needs access  to the data in that stack so how does  that part work and how does any of this  actually become efficient so now we have  a pretty decent list of functionality  that we need out of the silk runtime  system we need serial execution to work  we need thieves to be able to jump into  the middle of running functions we need  syncs to synchronize in this nested  fine-grain way we need to implement a  cactus stack for all the workers to see  and things have to deal with mixtures of  spawn frames and called frames that may  may be available when they steal  computation so that's a bunch of  considerations is this the whole picture  well there's a little bit more to it  than that so before I give you any  answers I'm just going to keep raising  questions and now I want to raise some  questions concerning the performance of  this system how do we want to design the  system to get  to get good parallel execution times  well if we take a look at the  work-stealing bounds for silk the silk  silks work stealing scheduler achieves  an expected running time of TCP on pew  processors which is proportional to the  work of the computation divided by the  number of processors plus something on  the order of the span of the computation  now if we think if we take a look at  this running time bound we can decompose  it into two pieces the t1 over P part  that's really the time that the parallel  workers on the system spend doing actual  work there P of those workers that are  all making progress on the work of the  computation that comes out to t1 over P  the other part of the bound order T  infinity that's the time that turns out  to be the time that workers spend  stealing computation from each other and  ideally what we want when we paralyze a  program using silk is we want to see  this program achieve linear speed-up  that means that if we give the give the  program more processors to run and if we  increase P we want to see the execution  time decrease linearly with P and that  means we want the program to do x' to  spend most the workers in the Silk  system to spend most of his time doing  useful work we don't want the workers  spending a lot of time stealing from  each other in fact it's we have one even  more than this we don't just want work  divided by number of processors we  really care about how the performance  compares to the running time of the  original serial code that we were given  that we parallelized that original  serial code ran in time T sub s and now  we paralyze it using silks ball and  soaked sink or in this case silk 4 and  ideally with sufficient parallelism  we're guarantee that the running time is  going to be tcp proportional to the work  of a processors t1 divided by p but we  really want speed up compared to T sub s  so that's our goal we want TC be to  proportional to T sub s over P that says  that we want the serial run  to be pretty close to the work of the  parallel computation so the one  processor running time of our silk code  ideally should look pretty close to the  writing time of the original serial code  so just to put these pieces together if  we had in a if we were originally given  a serial program that ran a time T sub s  and we paralyze it using silk we end up  with a parallel program with work t1 and  span T infinity we want to achieve  linear speed-up on P processors compared  to the original serial running time in  order to do that we need two things we  need ample parallelism T 1 over T  infinity should be a lot bigger than P  and we've seen why that's the case in  lectures past and we also want what's  called high work efficiency we want the  ratio of the serial running time divided  by the work of the SIL computation to be  pretty close to 1 as close as possible  now the silk runtime system is designed  with these two observations in mind in a  particular the slope runtime system says  suppose that we have a silk program that  has ample parallelism it has sufficient  parallelism to make good use of the  available parallel processors then the  goal of then in implementing the sope  runtime we have a goal to maintain high  work efficiency and to maintain high  work efficiency the silk runtime system  abides by what's called the work first  principle which is to optimize the  ordinary serial execution of the program  even at the expense of some additional  cost to steels now at 30,000 feet the  way that the sope runtime system  implements the work first principle and  makes all these components work is by  dividing the dividing the job between  both the compiler and the runtime system  library the compiler uses a handful of  small data structures including workers  and stack frames and implements  optimized fast paths for execution  functions which should be executed when  no steals occur  the runtime system library handles  issues with the parallel execution it  uses larger data structures that  maintain parallel running time state and  it handles slower paths of execution in  particular when Steel's actually occur  so those are all the considerations we  have some we have a lot of functionality  requirements and we have some  performance considerations we want to  optimize the work even at the expense of  some steels let's finally take a look at  how silk works how do we deal with all  these problems I imagine some of you may  have some ideas as to how you might  tackle one issue or another but let's  see what what really happens let start  from the beginning how do we implement a  work or deck now for this discussion  we're gonna use a running example which  is just a really really simple silk  routine it's not even as complicated as  fib we're gonna have a function foo that  at one point spawns a function bar in  the continuation calls Baz performs the  sync and then returns and just to  establish some terminology foo will be  what we call a spawning function meaning  that foo is capable of executing a silk  spawn statement the function bar is  spawned by foo we can see that from the  silk spawn in front of bar and the call  to baz occurs in the continuation of  that silk spawn simple picture everyone  good so far any questions about the  functionality requirements terminology  performance considerations  okay so now we're gonna take take a hard  look at just one worker we're gonna say  conceptually we have this deck like  structure which has spawn frames and  called frames let's ignore the rest of  the workers on the system let's not  worry about well we'll worry a little  bit about how Steel's can't work but  we're just going to focus on the actions  that one worker performs how do we  implement this deck and we want the  worker to operate on its own deck a lot  like a stack is going to push and pop  frames from the bottom of the deck  Steel's need to be able to transfer  ownership of several consecutive frames  from the top of the deck and these need  to be able to resume a continuation so  the way that the silk system does this  to bring this concept into an  implementation is that it's going to  implement the deck externally from the  actual call stack those frames will  still be in a stack somewhere and will  be managed roughly speaking with the  with a standard calling convention but  the worker is going to maintain a  separate deck data structure which will  contain pointers into this stack and the  worker itself will maintain the deck  using head and tail pointers now in  addition to this picture the frames that  are available to be stolen the frames  that have computation that a thief can  come along and execute those frames will  store an additional local structure that  will contain information that's  necessary for stealing to occur does  this make sense questions so far  ordinary call stack deck lives outside  of it  worker points at the deck pretty simple  design  so I mentioned that the that the  compiler and run that the compiler used  relatively lightweight structures this  is essentially one of them and if we  take a look at the at the implementation  of the sope runtime system this is the  essence of it there are some additional  implementation details but this is these  are the core this is in a sense the core  piece of the design so the rest is just  details the Intel so plus runtime system  takes this design and elaborates on it  for a variety of in a variety of ways  and we're gonna take a look at those  elaborations first off well we'll see is  that every spawn some computation ends  up being executed within its own helper  function which the compiler will  generate that's called the spawn a spawn  helper function and then the runtime  system is going to maintain a few basic  data structures as the workers execute  their work there'll be a structure for  the worker which will look similar to  what we just saw in the previous slide  there'll be a silk stack frame structure  for each instantiation of a spawning  function some function that can perform  a spawn and there will be a stack frame  structure for each spawn helper each  instantiation that is spawned now if we  take another look at the compiled code  we had before some of it starts to make  some sense originally we had our  spawning function foo and a statement  that's spawned off a called a bar and in  the c pseudocode of the compiled results  we see that we have two functions the  first function foo that's our spawning  function it's got a bunch of stuff in it  and we'll figure out what that's doing  in a second but there's a second  function and that second function is the  spawned helper and that spawn helper  actually contains a statement which  calls bar and ultimately saves a result  make sense now we're starting to  understand some of the confusing c  pseudocode we saw before  okay and if we take a look at each of  these routines we see indeed there is a  stack frame structure and so in Intelli  so plus it's called the silk RTS stack  frame very creative name I know and it's  just added as an extra local variable in  each of these functions you've got one  inside of foo because that's a spawning  function and you get one inside of the  spawn helper now if we dive into the  silk stack frame structure itself by  cracking open the source code for the  Intel so plus runtime we see that there  are a lot of fields in this structure  the main fields are as follows there is  a buffer a context buffer and that's  going to contain enough information to  resume a function at a continuation  particularly mean after.silk spawn or in  fact after a silk sink statement there's  an additional integer in the in the  stack frame called flags which will  summarize the state of the Silk stock  frame we'll see a little bit more about  that later and there's going to be a  pointer to a parent silk stack frame  that's somewhere above this above this  so Carty a stack frame somewhere in the  call stack so these silk RTS stack  frames these are the extra bit of states  that the silk runtime system adds to the  ordinary call stack if we take a look at  the actual worker structure it's a lot  like what we saw before we have a deck  that's external to the call stack the  silk worker maintains head and pointer  to head and tail pointers to the deck  this little worker is also going to  maintain a pointer to the current soak  RTS stack frame which will tend to be  somewhere near the bottom of the stack  okay so there's the basic data  structures that a single worker is going  to maintain that includes the deck let's  see a ball of action shall we  any questions about that so far before  we start watching pointers fly yeah what  are the arrows among the elements on the  call stack mean so in this picture of  the call stack function instantiations  are actually in green and local  variables specifically the silk RT has  stack frames those show up in beige so  foo SF is the silk RT a stack frame  inside instantiation of foo it's just a  local variable it's also stored in the  stack right now the silk Rd a stack  frame maintains a parent pointer and it  maintains a pointer up to some so  cartier stack frame above it on the  stack it's just another local variable  also storing the stack so when we step  away and look at the whole call stack  with all the function frames and the  silk RTS stack frames that's where we  got the pointers climbing up the stack  okay other questions  all right let's make some pointers fly  okay that's that's good this is gonna be  kind of a letdown because the first  thing we're gonna look at is some code  so we're not gonna have pointers fly  just yet  we can take a look at the code for the  spawning function foo at this point and  there's a lot of extra code in here  clearly I've hide a lot of stuff on this  slide and all this all the highlighted  material is related to the execution of  the silk runtime system but basically if  we look at this code we can understand  each of these pieces each of them has  some role to play in making this whole  process in work so at the very beginning  we have our silk stack frame structure  and there's a call to this enter frame  function which all that really does is  initialize the stack frame that's all  the functions doing later on we find  that there's this set jump routine we'll  talk a lot more about such up in a bit  that at this point we can say the set  jump prepares the function for a spawn  and inside the the conditional where the  setup where the such I've occurs is a  predicate we have a call to spawn bar if  we remember from a couple slides ago a  spawn bar was our spawn helper function  so we're here we're just invoking the  spot helper later on the code we have  another blob of conditionals with a soak  RTS sink call deep inside all that code  performs a sink we'll talk about that a  bit near the end of the lecture and  finally at the end of the spawning  function we have a call to pop frame  which just cleans up the silk stack  frame structure within this function and  then there's a call to leaf frame which  essentially cleans up the deck that's  the spawning function this is the spawn  helper it looks somewhat similar I've  added extra whitespace just to make this  slide a little bit prettier and in some  ways it's similar to the spawning  function itself we have a so cárdenas a  crane before the spawn helper another  call to enter frame which is just a  little bit different but essentially it  initializes this stack for  it's its reason to be is similar to the  enter frame call we saw before there's a  call to silk RTS detach which performs a  bunch of updates on the deck then  there's the actual invocation of the  spawns subroutine this is where we're  calling bar and finally at the end of  the function there's a call to pop frame  to clean up the stack structure and a  call to Li frame which will clean up the  deck and possibly return it'll try to  return we'll see more about that so  let's watch all this an action question  okay cool  but well let's let's see all of this in  action we'll start off with a pretty  boring picture all we've got on our call  attack is main and our soak worker has  nothing on its deck but now we suppose  that main calls our spawning function  foo and the spawning function foo  contains a silk RT a stack frame what  we're gonna do in the soap worker what  that enter frame call is going to going  to perform all is gonna do is update the  current stack frame we now have a so  card to a stack frame make sure their  worker points at it that's all fast  forward a little bit and few encounters  this call to silk spawn at bar and in  the c pseudocode that's compiled for foo  we have a setup routine this setup is  kind of a magical function this is the  function that allows thieves to steal  the continuation and in particular the  setup takes as an argument a buffer in  this case it's the context buffer that  we have in the silk RTS stack frame and  what the setup will do is it'll store  information that's necessary to resume  the function at the location of the set  jump and it stores that information is  the buffer can I know what guess what  that information might be  instruction on the step instruction  pointer stack pointer I believe both of  those are in the frame in a starting  point yeah  both of those are in the frame good what  else  all the registers are currently in use  does it need all the registers you're  absolutely on the right track but is  there any way it could restrict the set  of registers and needs to save that's  part of it such a biz in that clever  though so it it just stores a  predetermined set of registers but  there's another way to restrict the set  only registers uses parameters in the  called function yeah close enough Kali  saved registers so register is that the  that the function might that it's the  responsibility of food is safe this goes  all the way back to that discussion in  lecture I don't remember which which  small number talking about the calling  convention these registers need to be  saved as well as the instruction pointer  and various stack pointers those are  what gets saved into the buffer the  other registers what we're about to call  a function it's up to that other  function to save the registers  appropriately so we're not going to we  don't need to worry about those yeah so  good any questions about that all right  so this edge up routine let's take it  for granted that when we call a set John  Ball in this given buffer it returns  zero that's a good lie for now we'll  just run with it so such up return zero  the condition says if not zero which is  which turns out to be true and so the  next thing that happens is this call to  the spawn helper spawn underscore bar in  this case when we call spawn spawn  underscore bar what happens to our stack  some of this should look pretty routine  for doing a function call and so we push  the frame for the called function onto  the stack and that called functions bond  bar contains a local variable which is  this so cardia stack frame so that also  gets pushed onto the stack  pretty straightforward we've seen  function calls many times before this  should look pretty familiar now we do  this silk RTS enter frame fast routine  and I mentioned before that that's going  to update the worker structure and the  deck or no not the deck just a worker  structure so what's going to happen here  well we have a brand-new so card to a  stack frame on the on the stack any  guesses as to what change we make  what would enter into yeah  [Music]  point currents a crane to spawn bar  Sacre you're right anything else  hope I got this animation right  what are the various fields within the  stock frame and what did sorry I don't  know your name what's your name Greg  what did Greg ask about before when we  saw an earlier picture of the call stack  set a pointer to the parent exactly so  we're gonna do is we're gonna take this  call stack we do the enter frame fast  routine that establishes this parent  pointer in our brand new calls in our  brand new stack frame and we update the  workers current stack frame to point at  the bottom yes question  [Music]  how does interframe know what the parent  is good question enter frame knows the  worker or rather enter frame can do a  call which will give it access to the  Silk worker structure and because it can  do a call it can read the current  saccharin pointer in the worker  [Music]  yeah in this case we do yep so we add  the parent pointer then we delete and  update so good catch any other questions  cool alright now we encounter this thing  so card taste attached this one's kind  of exciting finally we got to do do  something to the deck any guesses what  we do how do we update the deck so card  is here here's a hint  so card taste attached allows this is  the function that allows some  computation to be stolen once once silk  artistic is done executing a thief could  come along and steal the continuation of  the silk spawn so what would silk RTS  detach due to our worker and it's  structures yeah and back  push the sacrum to the worker deck  specifically at the tail now right I  gave it away with by clicking the  animation oh now the thing that's  available to be stolen is inside of Fuu  so what ends up getting pushed on to the  deck is not the current stack frame but  in fact its immediate parent so the the  sacrum foo that gets pushed onto the  tail of the deck and we now push  something onto the tail of a deck and so  we advanced the table pointer is so good  everyone I see some nods I see at least  one nod I'll take it but feel free to  ask questions of course and then of  course there's this invocation of bar  this does what you might expect  it calls bar no magic here well no new  magic here okay fast-forward let's  suppose that bar finally returns and now  we return to the two same adapter bar in  the spawn helper that statement is the  pop frame actually says we just returned  from bar we need get rid of bar from the  stack frame good now we can actually get  the pop frame  what would the pop frame do it's going  to clean up the stack frame structure so  what would what would that entail any  guesses  move the currents a crane back to the  parent very good they think  that's largely it I guess there's one  other thing you can do this is kind of  optional given that it's gonna garbage  the garbage the memory anyway so it  updates the current stack frame to point  to the parent and now I no longer needs  that parent pointer so we can clean that  up in principle and then there's this  call to soak RTS leave frame this is  magic well not really but it's not  obvious this is a function call that may  or may not return  welcome to the silk runtime system you  end up with calls to functions that you  may never return from this happens all  the time  and these soak our DS leave frame may or  may not return based entirely on what's  on the status of the deck what content  is currently sitting on this on the  workers deck so why anyone have a guess  as to why the leave frame routine might  not return in the conventional sense  there's nothing if there's nothing left  to do on the deck then it's going to  start shakin right if there's nothing on  the deck then it then it has nowhere to  return to and so naturally as we've seen  from silk workers in this path in the  past it discovers there's nothing on the  deck there's no work to do time to turn  to a life of crime and try to steal work  from someone else so there are two  possible scenarios it could the pop  because the seed and execution power  continues as normal or it fails and it  becomes a thief now which of these two  cases do you think is more important for  the runtime system to optimize  [Music]  success case one exactly so why is that  at least we hope so yeah we assumed this  hearkens all the way back to the two  that work first principle we assume that  in the common case workers are doing  useful work they're not just spending  their time stealing from each other and  therefore ideally we want we want to  assume that the worker will do what's  normal just an ordinary serial execution  in a normal serial execution there's  something on the deck the pop succeeds  that's case one and so what we'll see is  that the runtime system in fact does a  little bit of optimization on case one  let's talk about something a little more  exciting  how about stealing computation we like  stealing stuff from each other yes  yep where does it return the results so  where does it return the result in the  spawn bar this actually the answer you  can kind of see two lines above this so  in this case in the original SIL code we  had x equals silk spawn of bar and here  what are the parameters to our spawn bar  our function  X and n now n is the input to bar right  so what's X  you can rewind a little bit and see  there you are correct  oops there you yeah so the original silk  code we had x equals silk spawn bar  that's the same X all that silk does is  pass a pointer to the memory allocator  for that variable down to the spawn  helper and now the spawn helper when it  calls bar and that returns it gets  stored into that storage in the parent  stack frame good good catch  good observation any questions about  that that makes sense cool  probably use too many animations in  these slides alright now let's talk  about stealing how do we how does a  worker steal computation now the  conceptual diagram we had before saw  this one worker with nothing on its deck  take a couple frames from another  workers deck and just slide them on over  what does that actually look like in the  implementation well we're still going to  take from the top of the deck but now we  have a picture that's a little bit more  accurate in terms of the structures that  are really implemented in the system so  we have the call sack of the victim and  the victim also has a deck data  structure and a silk worker data  structure with head and tail pointers  and a current stack frame so what  happens when a thief comes along out of  nowhere its bored it has nothing on its  deck head and tail pointers both point  to the top current stack frame has  nothing  what's the thief going to do any guesses  how does this thief take the content  from the workers deck  [Music]  that's their Instagram  [Music]  exactly right yep sorry was that I  didn't mean to interrupt  all right cool so the red highlighting  should give a little bit of a hint the  current stack frame in the thief is  going to end up pointing to the stack  frame at the top of the deck pointed to  you by the top of the deck and the head  of the deck needs to be updated so let's  just see all those pointers shuffle the  thief is going to take is going to  target the head of the deck it's going  to DQ that item from the from the top of  the deck it's going to set the current  stack frame to point to that item and  it'll delete the pointer on the deck  that makes sense cool now the victim and  the thief are on different processors  and this scenario involves shuffling a  lot of pointers around so we think about  if we think about this process there  needs to be some way to handle the  concurrent accesses that are gonna occur  on the head of the deck you haven't  talked about synchronization yet in this  class that's going to be a couple of  lectures down the road there are a  couple I'll give you a couple spoilers  for those synchronization lectures first  off synchronization is expensive and  second reason about synchronization is a  source of massive headaches  congratulations you now know those two  lectures I'm just gonna go to the  lectures you'll learn a lot they're  great in this silk in the silk runtime  system the way that those concurrent  accesses are handled the pro is by using  a protocol known as the t-h-e protocol  this is pseudocode for most logic in the  THC protocol there's a protocol that the  worker executing work normally follows  and there's the protocol for the thief  I'm not gonna walk through all the lines  of code here and and describe what they  do I'll just give you the very  high-level view of this protocol from  the theists perspective  the thief always grabs a lock on the  deck before doing any operations on the  deck always acquire the lock first for  the worker it's a little bit more  optimized so what the worker will do is  optimistically try to pop something from  the bottom of the deck and only if it  looks like that pop operation fails does  the worker do something more complicated  only then does it try to acquire a lock  on the deck then try to pop something  off see if it really succeeds or fails  and you know possibly turn to a life of  crime so the workers protocol looks  longer but that's just because the the  worker implements a special case which  which is optimized for the common case  this is essentially where the leave  frame routine that we saw before is  optimized for case one optimize for the  pop from the deck succeeding any  questions about that seem clear from  30,000 feet cool okay so that's how a  worker steals work from the top of  another from the top of a victim's deck  now that thief needs to resume a  continuation and this is that that whole  process about jumping into the middle of  an executing function it already has a  frame it already has a bunch of state  going on and somehow and all that was  established by a different processor so  somehow that thief has to magically come  up with the right state and start  executing that function how does that  happen well this has to do with a  routine that's the complement of the set  up routine we saw before the the  complement of set jump is what's called  long jump  so silk uses in particular silk thieves  use the long jump function in order to  resume a sole in continuation previously  in our spawning function foo we had this  set jump call and that set jump save  some states to a local buffer in  particularly the buffer  in the stack frame of foo now the thief  has just created this silk worker  structure where the current stack frame  is pointing at the stack frame of foo  and so what the thief will do is it will  execute a call to the it'll execute the  statement long jump pass it Aleksey the  long jump function passing that  particular stack frames buffer and an  additional argument and that long jump  will take the register states throw it  in the buffer put that register state  into the worker and then what the worker  proceed that makes sense  any questions about that this is kind of  a wacky routine because if you remember  one of the one of the registers stored  in that buffer is an instruction pointer  and so it's going to read and read the  instruction pointer out of the buffer  it's also gonna read a bunch of kali  saved registers and stack pointers out  of the buffer and i'm just gonna say  those are my those are that's my  register state now that's what the thief  says it just stole that register state  and it's going to set its RI p to be the  RIPD it just read so where does that so  what does that mean for where the long  jump routine returns  yeah returns in the stock frame above  the one who just stole uh more or less  can but more specifically we're in that  function does it return which call to  the the spawn bar here almost very very  close very very close what ends up  happening is that the long jump  effectively returns from the set jump a  second time this is the weird protocol  between set jump and long jump set jump  you pass it a buffer it saves some  register state and then it returns and  it returns immediately and on it's  direct invocation that set jump call  returns the value zero as we mentioned  before now if you invoke a long jump  using the same buffer that causes the  processor to effectively return from the  same set jump call they use the same  buffer but now it's going to return with  a different value and it's going to  return with the value specified in the  second argument so invoking long jump of  buffer X returns from that set jump with  the value X so when the so when the  thief executes a long jump with the  appropriate buffer and the second  argument is one what happens anyone walk  me through this oh well it's on the  slide okay so now set that set of  effectively returns a second time but  now it returns with the value one and  now the predicate the predicate gets  evaluated so if not one which would be  if false well don't do the consequent  because you're just because the  predicate was false and that means it's  gonna skip the call to spawn bar and  it'll just fall through and execute the  stuff right after that condition that  conditional  which happens to be the continuation of  the spawn that's kind of neat I think  that's kind of neat maybe I'm biased  anyone else think that I need actually  anyone desperately confused about this  such a long term nonsense any questions  you want to ask or just generally  confused about why these things exist in  modern computing yeah is there any  reason you couldn't just add some fixed  offset to the instruction pointer to  jump over the call in principle I think  if you can statically compute the  distance you need to jump then you can  just add that to our IP and let the long  jump  do its thing or rather the thief will  just adopt that our IP and end up in the  right place what's done here is  basically this was the protocol that  they set existing such up and long term  routines implement and I imagine it's a  more it's a bit more flexible of a  protocol then well you strictly need for  the silk runtime and so you know it adds  up working on but if you can statically  compute that offset there's no reason in  principle you couldn't adopt a different  approach so good observation any  questions any other questions  it's fine to be generally confused why  their routines such a been long jump  with this whacky behavior compiler  writers have that reaction all the time  these are a nightmare to compile but  anyway okay so we've seen how the how a  thief can take some computation off of a  victim's deck and we've seen how the  thief can jump right into the middle of  an executing function with the  appropriate register state is this the  end of the story is there anything else  we need to talk about with respect to  stealing or more pointedly what else do  we need to talk about with respect to  stealing  you're welcome to answer you like  okay I remember that list of concerns we  had at the beginning list of  requirements things about it's called  we will talk about sinks but not just  yet what other thing was brought up as  remember this slide from a previous  lecture here's another hint so the  register register state is certainly  part of the state of an executing  function what else defines a state of an  executing function whereas the other  state of the function where does that  live it lives on the stack so what is  there to talk about regarding the stack  the cactus stack exactly so we mentioned  before that thieves need to implement  this cactus stack abstraction for so for  the silk runtime system what exact why  exactly do we need this cactus stack  what's wrong was just having the thief  have stock used the victims stack yep  the victim might just free up a bunch of  stuff ends that it's no longer  accessible so it can free some amount of  some amount of stuff in particular  everything up to the function foo but in  fact it can't return from the function  foo because some other well assuming  that the silk RT has leaf frame thing is  implemented the function foo is no  longer in a stack it won't ever reach it  bla bla bla  so it won't return from the function foo  while another worker is working on it  but good observation there's something  else that can go wrong if the thief just  directly uses the victim's stack  well let's take a hint from the slide we  have so far so the example that's going  to be shown is that the thief steals the  continuation of foo and then the thief  is going to call a function Baz  so the thief is using the victim's stack  and then it calls a function Baz what  goes wrong  [Music]  exactly the victim in this picture for  example has some other functions on its  stack below foo so if the thief does any  function calls and is using the same  stack it's gonna scribble all over the  state of in this case spawn bar and bar  which the victim is trying to use and  maintain so the thief will end up  corrupting the victims stack and if you  think about it it's also possible for  the victim to grab the thief's stack  involved they can't they can't share a  stack but they do want to share some  amount of some amount of data on the  stack they do both care about the state  of foo and that needs to be consistent  across all the workers but you at least  need a separate call stack for the thief  we'd rather not do unnecessary work in  order to initialize this call stack  however we really need this call stack  for things that the thief might might  evoke variable is that the local  variables the thief might need or  functions that the thief might call or  spawn okay so how do we implement the  call stack sorry the cactus stuck we  have a victim sack the thief stack and  we have a pretty cute trick in my  opinion so the thief steals this  continuation it's not it's gonna do a  little bit of magic with its stack  pointers what it's gonna do is it's  going to use the rbp it was given which  points out the victim stack and it's  gonna set the stack pointer to point at  its own stack so RB P is over there and  RSP for the thief is pointing to the  beginning of the thief's call stack and  that is basically fine  the thief can access all the state in  the function foo as offsets from RB p  but if the thief needs to do any  function calls we have a calling  convention that involves saving our VP  and updating our SP in order to execute  the call so in particular the thief  calls a function bars  it saves its current value of our VP on  to its own stack okay advances RRSP it  says RB P equal to RSP it pushes the  stack frame for Baz onto the stack and  it advances RSP a little bit further and  just like that the thief is churning  away on its own stack so just with this  magic of our VP pointing there and rst  pointing here we get our cactus stack  everyone followed that anyone  desperately confused by this stack  pointer base pointer nonsense  who thinks is kind of a neat trick all  right cool  everything is a really mundane trick  hopefully no things are the mundane  trick okay there's like half a hand  there that's fine noise I think this is  a neat trick just messing around with  the stack pointers are there any worries  about using our VP and our SP this way  any any concerns that you might think of  from using using these two stack  pointers as described in a past lecture  briefly mentioned was a compiler  optimization for dealing with stacks  yeah  three of them  right there was a compiler optimization  that said in certain cases you don't  need both the base pointer and the stack  pointer you can do all offsets I think  it's actually off the stack pointer and  then the base pointer becomes an  additional general purpose register that  optimization clearly does not work if  you need the base pointer stack pointer  to do this wacky to do this wacky trick  the answer is that the SIL compiler  specifically specifically says if this  function has a continuation that could  be stolen don't do that optimization  it's super illegal it's very bad don't  do the optimization it's not about being  the answer and it costs us a general  purpose register for silk functions not  the biggest loss in the world  all right there's a little bit of time  left so we can talk about synchronizing  computation I'll give you the brief  version of this this part gets fairly  fairly complicated and so I'll give you  a high-level summary of how all this  works  so just to page back in some contexts we  have this scenario where different  processors are executing different parts  of our computation dag and one processor  might encounter a silk sync statement  that it can't execute because some other  processor is busy executing some spawned  sub computation now in this case P 3 is  waiting on P 1 to finish its execution  before the sync can proceed and  synchronization needs to happen really  only on the sub computation that P 1 is  executing P 2 shouldn't play a role in  this so what exactly happens when a  worker reaches the silk sync before all  the spawns sub computations return well  we'd like to worker to become a thief  we'd rather the worker not just sit  there and wait until all this balance of  computation is returned that's a waste  of a perfectly good worker but we also  can't let the workers current function  frame disappear there's a spawn sub  computation that's using that frame that  frame is its parent  it may be accessing state in that frame  and maybe trying to save a return value  to some location in that frame and so  the frame has to has to persist even if  the worker that's working on the frame  goes off and becomes a thief moreover in  the future that's up computation we  believe should return and that worker  must resume the frame and actually  execute past the silk sync finally the  silk sync should only apply to the  nested sub computations underneath this  function not the program in general and  so we don't we don't allow ourselves  synchronization just among all the  workers wholesale we don't say okay  we've hit a sync every worker in the  system must reach some point in the  execution we only care about this nested  synchronization so if we think about  this and we're talking about nested  synchronization for computations under a  function we have this notion of cactus  stack we have this notion of a tree of  function invocations we may immediately  start to think about well what if we  just maintain some state in a tree to  keep track of who needs to synchronize  with whom which computations are waiting  on which other computations to finish  and in fact that's essentially what this  so part-time system does it maintains a  tree of states called full frames and  those full frames store state for the  parallel of sub computations and it  keeps track those full frames keep track  which sub computations are outstanding  and how they relate to each other this  is a high-level picture of a full frame  there are some there lots of details  delighted to be honest but at 30,000  feet a full frame keeps track of a bunch  of information for the parallel  execution I know I'm giving you the  quick version of this including pointers  to parent frames and possibly pointers  to child frames or at least the number  of outstanding child frames the  processors when visit system work on  what are called active full frames in  the diagram those full frames  there are the rounded rectangles  highlight and dark blue other full  frames in the system are what we call  suspended there they're waiting on some  sub computation to return that's what a  full frame tree can look like under some  execution let's see how a full frame  tree can come into being just by working  through an animation so suppose we have  some worker with a bunch of spawn and  called frames on its deck no one else  has no other workers have anything on  their decks and and finally some worker  says it wants to steal sub car it wants  a steal and I'll admit this animation is  crafted slightly just to make the  pictures a little bit nicer it can look  more complicated in practice don't worry  if that was actually worried of yours so  what's gonna so what's gonna happen the  thief is gonna take some frames from the  top of the victim's deck and it's  actually going to steal not just those  frames but the whole full frame  structure along with it the full frame  structure is just represented with this  rounded rectangle in fact it's a  constant size thing but the theme is  gonna take the whole full frame  structure and it's going to give the  victim a brand new full frame and  establish the the child to parent  pointer in that in the victims new full  frame that's kind of weird it's not  obvious why the thief would take the  full frame as a stealing computation at  least not from one step but we can see  why it helps just given one more step so  let's fast forward this picture a little  bit and now we have another worker try  to steal some sub computation some  computation and we have a little bit  more stuff going on so this worker might  randomly select the last worker on the  right steal a computation from the top  of its deck and it's gonna steal the  full frame along with those along with  the the deck frames  and because it stole the full-frame all  pointers to that full-frame from any  child sub computations are still valid  the child self computation on the Left  still points to the correct full-frame  the the full frame that was stolen has  the parent context of that child and so  we need to make sure that pointer is  still it's still good if if it created a  new full frame for itself then you would  have to update the child pointers  somehow and that requires more  synchronization and a more complicated  protocol synchronization is expensive  protocols are complicated this ends up  saving some complexity and then it  creates a frame for the child and we can  see this process unfold just a little  bit further  hello behold after a few steals we end  up with a tree we have to two children  pointing to one parent and one of those  children has its own child  great now suppose that some worker says  oh I encountered a sink can i  synchronize in this case the worker has  an outstanding child computation so I  can't synchronize and so we can't  recycle the full frame we can't recycle  any of the any of the stack for this  child and so instead the worker will  suspend its full frame turn you from  dark blue light blue in our picture and  it goes and becomes a thief  the program has ample parallelism what  do you expect to typically happen when  the program execution reaches a silk  sync we're kind of out of time so I  think I'm just gonna spoil the answer  for this unless anyone has it has a  guest handy so what's the common case  for a silk sync  for the sake of time the common case is  that the executing function has no  outstanding children all the workers on  the system we're busy doing their own  thing  there's no synchronization that's  necessary and so how is the runtime  optimize this case it ends up using some  bits yeah ends up having the full frame  use some bits of an Associated stack  frame in particular flag field  and that's why when we look at the  compiled code for a silk sync we see  some conditions that that evaluate the  flags within the within the local stack  frame that's just an optimization to say  if you don't need a sync don't do any  computation otherwise otherwise some  steals really did occur go ahead and  execute the silk RTS sync routine there  are a bunch of other runtime features if  you take a look at that picture for a  long time you may be dissatisfied with  what that implies about some of the  protocols and there's a lot more code  within the runtime system itself to  implement a variety of other features  such as support for C++ exceptions  reducer hyper objects and a form of ID  ID is called pedigrees we won't talk  about that today I'm actually all out of  time thanks for listening to all the all  this about the soap run type system feel  free to ask any questions after your  class

2
00:00:03,189 --> 00:00:05,769
the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  everybody it's my pleasure once again to  welcome tibi Charl who is the author of  your taper compiler to talk about the  silk runtime system thanks Charles  I can everyone hear me in the back seem  good  okay thanks for the introduction today  I'll be talking about the silicon time  system this is pretty exciting for me  this is a lecture that's not about  compilers I get to talk about something  a little different for once it should be  a fun lecture recently as I understand  it you've been looking at storage  allocation both in the serial case as  well as the parallel case and you've  already done silk programming for a  while at this point this lecture  honestly is a bit of a non sequitur can  in terms of the overall flow of the  course and it's also an advanced topic  the silk runtime system is a pretty  complicated piece of software but  nevertheless I believe you should have  enough background to at least start to  understand and appreciate some of the  aspects of the design of the silk run  time system so that's why we're talking  about that today just to quickly recall  something that you're all I'm sure  intimately familiar with by this point  what's the program El Sol programming  all about well silk is a parallel  programming language that allows allows  you to make your software run faster  using parallel processors and to use  silk it's pretty straightforward you may  start with some serial code that runs in  some running time we'll denote that as T  sub s for certain parts of the lecture  if you wanted to run in parallel using  silk you just insert silk keywords in  choice locations for example you can  paralyze the outer loop in this matrix  multiply kernel and that will allow your  code run in time TCP on P processors an  ideally tcp should be less than T sub s  now just adding keywords is all you need  to do to tell silk to execute the  computation in parallel what is silk do  in light of those keywords at a very  high level silk and it's specifically  its runtime system takes care of the  task of scheduling and load balancing  the computation on the parallel  processors and on the multi-core system  in general  so after you've denoted logical  parallelism the program using spa  silicon silk sync and so for the soaked  scheduler Maps that computation onto the  processors and it does so dynamically at  runtime based on whatever processing  resources happen to be available and so  accuses a randomized work stealing  scheduler which guarantees that that  mapping is efficient and the execution  runs efficiently now you've all been  using the silk platform for a while and  it's basic usage you write some silk  code possibly by parallelizing ordinary  serial code you feed that to a compiler  you get a binary you run the binary on  your you run the binary with some four  to your input on a mult multi core  system you get parallel performance  today we're gonna look at how exactly  does silk work what's the magic that  goes on hidden by the boxes on this  diagram and at the very the very first  thing to note is that this picture is a  little bit the first simplification that  we're gonna break is to say it's not  really just silk source and the silk  compiler there's also a runtime system  library some loops okar tsso in case  you've seen that file or messages about  that file on your system and really it's  the compiler and the runtime library  that work together to implement silks  runtime system to do the work stealing  and do the efficient scheduling and load  balancing now we might suspect that if  you just take a look at the code that  you get when you compile a soak program  but my tell you something about how silk  works  see pseudocode for the result that  results when you compile a simple piece  of silk code it's a bit complicated I  think that's fair to say there's a lot  going on here there was one function in  the original program now there are two  there are some new variables there's  some calls to functions that look a  little bit strange there's a lot going  on in the compiled results this isn't  exactly easy to interpret or understand  and this doesn't even bring into the  picture the runtime system library the  runtime system library you can find the  source code online it's a little less  than twenty thousand lines of code it's  also kind of complicated so rather than  dive into the code directly what we're  going to do today is an attempt a  top-down approach to understanding how  the so prone type system works and some  of the design considerations so we're  gonna start by talking about some of the  required functionality that we need out  of the silic runtime system as well as  some performance considerations for how  silk how the runtime system should work  and then we'll take a look at how the  worker Dex and silk get implemented how  spawning actually works how stealing a  computation works and how  synchronization works within silk that  all sound good any questions so far this  should all be review more or less okay  so let's talk a little bit about  required functionality you see in this  picture before I hope this picture  Illustrated the execution model of a  silk program here we have everyone's  favorite exponential time fibonacci  routine parallelized using silk this is  not an efficient way to compute  Fibonacci numbers but it's a nice  didactic example for understanding  parallel computation especially the silk  model and as we saw many lectures ago  when you run this program on a given  input the execution of the program can  be modeled using as a computation dag  and this computation dag unfolds  dynamically as the program executes but  I want to stop and die take a hard look  at exactly what that dynamic execution  looks like when we've got parallel  processors and work-stealing  all coming into play so we'll stick with  this Fibonacci routine and we'll imagine  we've just got one processor on the  system to start and we're just gonna use  this one processor to execute for before  and it's gonna take some time to do it  just to make thee just to make the story  interesting so we start exceeding this  computation and that one processor is  just going to execute the Fibonacci  routine from beginning up to the silk  spawned statement as if it's ordinary  serial code because it is ordinary  serial code at this point the processor  hits the silk spawn statement what  happens what happens now anyone remember  what happens to the dag  [Music]  if bridge is downward and spawns another  process more or less the way we model  that the silks bond is of a routine fib  of n minus 1 in this case I would be fib  of 3 and so like an ordinary function  call we're going to get a brand new  frame for a fib of 3 and that's going to  have some strand that's available to  execute but the spawn is not your  typical function call it actually allows  some other computation to run in  parallel and so the way we modeled out  in this picture so we got a new frame  for fib of 3 there's a strand available  to execute there and the continuation  the green strand is now available in in  the frame for fit before but no one's  necessarily executing it it's kind of  faded in the picture so once this spawn  has occurred what's the process are  gonna do the processor is actually going  to dive in and start executing fib of 3  as if it were an ordinary function call  yes there's a strand available within  the frame of fib before but the  processor isn't gonna worry about that  strand it's just gonna say Oh fit before  calls fib of 3 gonna start computing fib  of 3 it's not good this is a processor  dies down from pink strand to pink  strand the instruction pointer for the  processor returns to the beginning of  the fib routine because we're now  calling fib once again and this process  repeats it executes the pink strand up  until the silks bond just like ordinary  old code the spawn occurs we've already  seen this picture before the spawn  allows another strand to execute in  parallel but it also creates a frame for  fib of 2 and the processor dives into  fib of 2 resetting the instruction  pointer to the beginning of fib p1  executes sup to the spawn once again we  get another strand to execute as well as  an invocation of fib of one processor  dives even further so that's fine this  is just the processor doing more or less  ordinary serial execution on this fit  routine but it's also allowing some  strands to be executed in parallel  this is the one processor situation  looks pretty good so far right and in  the fit of one case it doesn't make it  as far through the pink strand because  in fact we hit the base case but now  let's bring in some more processors  suppose that another processor finally  shows up and says I'm bored I wanted to  do some work and decides to steal some  computation it's gonna discover the  green strand in the frame fib of 4 and  PT is just going to jump in there and  start executing that strand and if we  think really hard about what this means  P 2 is another processor on the system  it has its own set of registers it has  its own instruction pointer and so  what's silk somehow allows to happen is  for P 2 to just jump right into the  middle of this fit before routine which  is already executing it just sets the  instruction pointer to point out that at  that green instruction at the call to  fib of n minus 2 and it's gonna pick up  where processor 1 left off when it  executed up to this point into before  somehow in this case it executed n minus  2 that calls fib of 2 creates a new  strand it's just an ordinary function  call it's going to descend into that new  frame it's gonna return to the beginning  of fib all that's well well and good  another processor might come along and  steal another piece of the computation  it steals another green strand so once  again this processor needs to jump into  the middle of an executing function its  instruction pointer is just going to  point at this call to fib of n minus 2  somehow it's going to have the state of  this executing function available  despite having independent registers and  needs to just start from this location  with all the parameters set  appropriately and start executing this  function as if it's an ordinary function  it calls fib of 3 minus 2 is 1 and now  these processors might start executing  in parallel P 1 might return from its  base case routine up to the parent call  if F of minus 2 and started executing  its continuation because that wasn't  stolen meanwhile P 3 to  into the execution of a fit of one and  then another step P 3 and P to make some  progress executing their computation P 2  encounters a silk spawn statement which  creates a new frame and allows another  strand X Union parallel P 3 encounters  this base case routine and says ok it's  time to return and all of that can  happen in parallel and somehow the Silk  system has to coordinate all of this and  so already but we already have one  mystery how does a processor start  executing from the middle of a running  function the running function and it's  States lived on lived on P 1 initially  and then P 2 and P 3 somehow find that  states hop into the middle of the  function and just start running that's  kind of strange how does that happen how  does this so chrono system make that  happen this is one thing to consider  another thing to consider is what  happens when we hit a sync we'll talk  about how these issues get addressed  later on but let's lay out all the  considerations upfront before we just  see how bad the problem is before we try  to solve solve it bit by bit so now  let's take this picture again and  progress it a little bit further let's  suppose that processor 3 decides to xq  the return it's gonna return to an  invocation of fib of 3 and the return  statement is this silk sync is a silk  sync statement but processor 3 can't  execute the sync because the computation  of fib of 2 in this case that's being  done by processor 1 that computation is  not done yet so the execution can't  proceed past the sync so somehow P 3  needs to say ok there's a sink statement  Bo can execute beyond this point because  specifically it's waiting on processor 1  it doesn't care what processor 2 is  doing prostitue is having a dandy time  executing fib of 2 on the other side of  the tree process of 3 shouldn't care so  processor 3 can't do something like ok  all processors need to stop get to this  point in the code  their execution can proceed no no no it  just needs to wait on Prosser what  somehow the Silk system has to allow  that fine grain synchronization to  happen in this nested pattern so how  does this silk sync wait on only the  nested sub computations within the  program how does it figure out how to do  that how does this silk run time system  implement this so that's another  consideration okay so at this point we  have three top level considerations a  single worker needs to be able to  execute this program as if it's an  orrery serial program thieves have to be  able to jump into the middle of  executing functions and pick up from  where they left off from where other  processors in the system left off syncs  have to be able to stall stall functions  appropriately based only on those  functions nested child sub computations  so we have three big considerations that  we need to pick apart so far that's not  the whole story though any ideas what  other functionality we need to worry  about for implementing the Silk system  it's kind of an open-ended question but  any thoughts  we have serial execution spawning or  stealing and sinking as top-level  concerns anyone remember some other  features of silk that the runtime system  magically makes happen correctly it's  probably been a while has you seen those  yeah the silk for loops divide and  conquer  somehow the runtime system does have to  have to implement silk fours the silk  fours end up getting implemented  internally with spawns and syncs that's  courtesy of the compiler yeah courtesy  of the compiler so we won't look too  hard at so for today but that's a that's  definitely one concern so good  observation any other thoughts sort of  low-level system details that silk needs  to implement correctly  cache coherence it actually doesn't need  to worry too much about cache coherence  although given the latest runtime you  know given latest performance numbers  I've seen from silk maybe it should  worry more about the cache but it turns  out the hardware does a pretty good job  maintaining the cache coherence protocol  itself but good guess  there's not be a tough question because  it's really just calling back memories  of old lectures I think you recently had  a quiz on this material so it's probably  safe to say that all that material has  been paged out of of your brain at this  point so I'll just spoil the fun for you  silk has a notion of a cactus stack so  we talked a little bit about processors  jumping into the middle of an executing  function and somehow having the state of  that function available one  consideration is register state but  another consideration is the stack  itself and silk supports the C C's rule  for pointers namely that children can  see pointers into parent frames but  parents can't see pointers into child  frames now each processor each worker in  a silk system needs to have its own view  of the stack but those views aren't  necessarily independent in this picture  all five processors share the same view  of the frame for function a  instantiation a and then processor is  three through five all share the same  view for the instantiation C so somehow  silk has to make all of those views  available and consistent but not quite  the same sort of consistent as we get  with cache coherence silk somehow has to  implement this cactus stack so that's  another consideration that we have to  worry about and then there's one more  kind of funny detail if we take another  look at work-stealing itself may  remember we had this picture from  several lectures ago where we have  processors on the system each house each  maintains its own deck of frames and  workers are allowed to steal steel  frames from each other but if we take a  look at how this all unfolds yes we may  have a a processor that performs a call  and that'll push another frame for a  called function onto its deck on the  bottom you may spawn and then it'll push  a spawn frame onto the bottom of its  deck but we fast forward a little bit  and we got end up with a worker  nothing to do that workers gonna go  ahead and steal picking or picking  another worker in the system at random  and it's gonna steal from the top of the  deck but it's not just gonna steal the  topmost item on the deck it's actually  going to steal a chunk of items from the  deck in particular if it selects the  third processor in this picture third  from the left this thief is gonna steal  everything up to everything through the  parent of the next spawned frame it  needs to take this whole stack of frames  and it's not clear a priori how many  frames the workers gonna have to steal  in this case but nevertheless it needs  to take all those frames and resume  execution after all that bottom is  called frame that I just stole that's  where there's a continuation with work  available to be done in parallel and so  if we think about it there are a lot of  questions that arise what's involved in  stealing frames what synchronization  does the system have to implement what  happens to the stack we it looks like we  just shifted some frames from one  processor to another but the first  processor the victim still needs access  to the data in that stack so how does  that part work and how does any of this  actually become efficient so now we have  a pretty decent list of functionality  that we need out of the silk runtime  system we need serial execution to work  we need thieves to be able to jump into  the middle of running functions we need  syncs to synchronize in this nested  fine-grain way we need to implement a  cactus stack for all the workers to see  and things have to deal with mixtures of  spawn frames and called frames that may  may be available when they steal  computation so that's a bunch of  considerations is this the whole picture  well there's a little bit more to it  than that so before I give you any  answers I'm just going to keep raising  questions and now I want to raise some  questions concerning the performance of  this system how do we want to design the  system to get  to get good parallel execution times  well if we take a look at the  work-stealing bounds for silk the silk  silks work stealing scheduler achieves  an expected running time of TCP on pew  processors which is proportional to the  work of the computation divided by the  number of processors plus something on  the order of the span of the computation  now if we think if we take a look at  this running time bound we can decompose  it into two pieces the t1 over P part  that's really the time that the parallel  workers on the system spend doing actual  work there P of those workers that are  all making progress on the work of the  computation that comes out to t1 over P  the other part of the bound order T  infinity that's the time that turns out  to be the time that workers spend  stealing computation from each other and  ideally what we want when we paralyze a  program using silk is we want to see  this program achieve linear speed-up  that means that if we give the give the  program more processors to run and if we  increase P we want to see the execution  time decrease linearly with P and that  means we want the program to do x' to  spend most the workers in the Silk  system to spend most of his time doing  useful work we don't want the workers  spending a lot of time stealing from  each other in fact it's we have one even  more than this we don't just want work  divided by number of processors we  really care about how the performance  compares to the running time of the  original serial code that we were given  that we parallelized that original  serial code ran in time T sub s and now  we paralyze it using silks ball and  soaked sink or in this case silk 4 and  ideally with sufficient parallelism  we're guarantee that the running time is  going to be tcp proportional to the work  of a processors t1 divided by p but we  really want speed up compared to T sub s  so that's our goal we want TC be to  proportional to T sub s over P that says  that we want the serial run  to be pretty close to the work of the  parallel computation so the one  processor running time of our silk code  ideally should look pretty close to the  writing time of the original serial code  so just to put these pieces together if  we had in a if we were originally given  a serial program that ran a time T sub s  and we paralyze it using silk we end up  with a parallel program with work t1 and  span T infinity we want to achieve  linear speed-up on P processors compared  to the original serial running time in  order to do that we need two things we  need ample parallelism T 1 over T  infinity should be a lot bigger than P  and we've seen why that's the case in  lectures past and we also want what's  called high work efficiency we want the  ratio of the serial running time divided  by the work of the SIL computation to be  pretty close to 1 as close as possible  now the silk runtime system is designed  with these two observations in mind in a  particular the slope runtime system says  suppose that we have a silk program that  has ample parallelism it has sufficient  parallelism to make good use of the  available parallel processors then the  goal of then in implementing the sope  runtime we have a goal to maintain high  work efficiency and to maintain high  work efficiency the silk runtime system  abides by what's called the work first  principle which is to optimize the  ordinary serial execution of the program  even at the expense of some additional  cost to steels now at 30,000 feet the  way that the sope runtime system  implements the work first principle and  makes all these components work is by  dividing the dividing the job between  both the compiler and the runtime system  library the compiler uses a handful of  small data structures including workers  and stack frames and implements  optimized fast paths for execution  functions which should be executed when  no steals occur  the runtime system library handles  issues with the parallel execution it  uses larger data structures that  maintain parallel running time state and  it handles slower paths of execution in  particular when Steel's actually occur  so those are all the considerations we  have some we have a lot of functionality  requirements and we have some  performance considerations we want to  optimize the work even at the expense of  some steels let's finally take a look at  how silk works how do we deal with all  these problems I imagine some of you may  have some ideas as to how you might  tackle one issue or another but let's  see what what really happens let start  from the beginning how do we implement a  work or deck now for this discussion  we're gonna use a running example which  is just a really really simple silk  routine it's not even as complicated as  fib we're gonna have a function foo that  at one point spawns a function bar in  the continuation calls Baz performs the  sync and then returns and just to  establish some terminology foo will be  what we call a spawning function meaning  that foo is capable of executing a silk  spawn statement the function bar is  spawned by foo we can see that from the  silk spawn in front of bar and the call  to baz occurs in the continuation of  that silk spawn simple picture everyone  good so far any questions about the  functionality requirements terminology  performance considerations  okay so now we're gonna take take a hard  look at just one worker we're gonna say  conceptually we have this deck like  structure which has spawn frames and  called frames let's ignore the rest of  the workers on the system let's not  worry about well we'll worry a little  bit about how Steel's can't work but  we're just going to focus on the actions  that one worker performs how do we  implement this deck and we want the  worker to operate on its own deck a lot  like a stack is going to push and pop  frames from the bottom of the deck  Steel's need to be able to transfer  ownership of several consecutive frames  from the top of the deck and these need  to be able to resume a continuation so  the way that the silk system does this  to bring this concept into an  implementation is that it's going to  implement the deck externally from the  actual call stack those frames will  still be in a stack somewhere and will  be managed roughly speaking with the  with a standard calling convention but  the worker is going to maintain a  separate deck data structure which will  contain pointers into this stack and the  worker itself will maintain the deck  using head and tail pointers now in  addition to this picture the frames that  are available to be stolen the frames  that have computation that a thief can  come along and execute those frames will  store an additional local structure that  will contain information that's  necessary for stealing to occur does  this make sense questions so far  ordinary call stack deck lives outside  of it  worker points at the deck pretty simple  design  so I mentioned that the that the  compiler and run that the compiler used  relatively lightweight structures this  is essentially one of them and if we  take a look at the at the implementation  of the sope runtime system this is the  essence of it there are some additional  implementation details but this is these  are the core this is in a sense the core  piece of the design so the rest is just  details the Intel so plus runtime system  takes this design and elaborates on it  for a variety of in a variety of ways  and we're gonna take a look at those  elaborations first off well we'll see is  that every spawn some computation ends  up being executed within its own helper  function which the compiler will  generate that's called the spawn a spawn  helper function and then the runtime  system is going to maintain a few basic  data structures as the workers execute  their work there'll be a structure for  the worker which will look similar to  what we just saw in the previous slide  there'll be a silk stack frame structure  for each instantiation of a spawning  function some function that can perform  a spawn and there will be a stack frame  structure for each spawn helper each  instantiation that is spawned now if we  take another look at the compiled code  we had before some of it starts to make  some sense originally we had our  spawning function foo and a statement  that's spawned off a called a bar and in  the c pseudocode of the compiled results  we see that we have two functions the  first function foo that's our spawning  function it's got a bunch of stuff in it  and we'll figure out what that's doing  in a second but there's a second  function and that second function is the  spawned helper and that spawn helper  actually contains a statement which  calls bar and ultimately saves a result  make sense now we're starting to  understand some of the confusing c  pseudocode we saw before  okay and if we take a look at each of  these routines we see indeed there is a  stack frame structure and so in Intelli  so plus it's called the silk RTS stack  frame very creative name I know and it's  just added as an extra local variable in  each of these functions you've got one  inside of foo because that's a spawning  function and you get one inside of the  spawn helper now if we dive into the  silk stack frame structure itself by  cracking open the source code for the  Intel so plus runtime we see that there  are a lot of fields in this structure  the main fields are as follows there is  a buffer a context buffer and that's  going to contain enough information to  resume a function at a continuation  particularly mean after.silk spawn or in  fact after a silk sink statement there's  an additional integer in the in the  stack frame called flags which will  summarize the state of the Silk stock  frame we'll see a little bit more about  that later and there's going to be a  pointer to a parent silk stack frame  that's somewhere above this above this  so Carty a stack frame somewhere in the  call stack so these silk RTS stack  frames these are the extra bit of states  that the silk runtime system adds to the  ordinary call stack if we take a look at  the actual worker structure it's a lot  like what we saw before we have a deck  that's external to the call stack the  silk worker maintains head and pointer  to head and tail pointers to the deck  this little worker is also going to  maintain a pointer to the current soak  RTS stack frame which will tend to be  somewhere near the bottom of the stack  okay so there's the basic data  structures that a single worker is going  to maintain that includes the deck let's  see a ball of action shall we  any questions about that so far before  we start watching pointers fly yeah what  are the arrows among the elements on the  call stack mean so in this picture of  the call stack function instantiations  are actually in green and local  variables specifically the silk RT has  stack frames those show up in beige so  foo SF is the silk RT a stack frame  inside instantiation of foo it's just a  local variable it's also stored in the  stack right now the silk Rd a stack  frame maintains a parent pointer and it  maintains a pointer up to some so  cartier stack frame above it on the  stack it's just another local variable  also storing the stack so when we step  away and look at the whole call stack  with all the function frames and the  silk RTS stack frames that's where we  got the pointers climbing up the stack  okay other questions  all right let's make some pointers fly  okay that's that's good this is gonna be  kind of a letdown because the first  thing we're gonna look at is some code  so we're not gonna have pointers fly  just yet  we can take a look at the code for the  spawning function foo at this point and  there's a lot of extra code in here  clearly I've hide a lot of stuff on this  slide and all this all the highlighted  material is related to the execution of  the silk runtime system but basically if  we look at this code we can understand  each of these pieces each of them has  some role to play in making this whole  process in work so at the very beginning  we have our silk stack frame structure  and there's a call to this enter frame  function which all that really does is  initialize the stack frame that's all  the functions doing later on we find  that there's this set jump routine we'll  talk a lot more about such up in a bit  that at this point we can say the set  jump prepares the function for a spawn  and inside the the conditional where the  setup where the such I've occurs is a  predicate we have a call to spawn bar if  we remember from a couple slides ago a  spawn bar was our spawn helper function  so we're here we're just invoking the  spot helper later on the code we have  another blob of conditionals with a soak  RTS sink call deep inside all that code  performs a sink we'll talk about that a  bit near the end of the lecture and  finally at the end of the spawning  function we have a call to pop frame  which just cleans up the silk stack  frame structure within this function and  then there's a call to leaf frame which  essentially cleans up the deck that's  the spawning function this is the spawn  helper it looks somewhat similar I've  added extra whitespace just to make this  slide a little bit prettier and in some  ways it's similar to the spawning  function itself we have a so cárdenas a  crane before the spawn helper another  call to enter frame which is just a  little bit different but essentially it  initializes this stack for  it's its reason to be is similar to the  enter frame call we saw before there's a  call to silk RTS detach which performs a  bunch of updates on the deck then  there's the actual invocation of the  spawns subroutine this is where we're  calling bar and finally at the end of  the function there's a call to pop frame  to clean up the stack structure and a  call to Li frame which will clean up the  deck and possibly return it'll try to  return we'll see more about that so  let's watch all this an action question  okay cool  but well let's let's see all of this in  action we'll start off with a pretty  boring picture all we've got on our call  attack is main and our soak worker has  nothing on its deck but now we suppose  that main calls our spawning function  foo and the spawning function foo  contains a silk RT a stack frame what  we're gonna do in the soap worker what  that enter frame call is going to going  to perform all is gonna do is update the  current stack frame we now have a so  card to a stack frame make sure their  worker points at it that's all fast  forward a little bit and few encounters  this call to silk spawn at bar and in  the c pseudocode that's compiled for foo  we have a setup routine this setup is  kind of a magical function this is the  function that allows thieves to steal  the continuation and in particular the  setup takes as an argument a buffer in  this case it's the context buffer that  we have in the silk RTS stack frame and  what the setup will do is it'll store  information that's necessary to resume  the function at the location of the set  jump and it stores that information is  the buffer can I know what guess what  that information might be  instruction on the step instruction  pointer stack pointer I believe both of  those are in the frame in a starting  point yeah  both of those are in the frame good what  else  all the registers are currently in use  does it need all the registers you're  absolutely on the right track but is  there any way it could restrict the set  of registers and needs to save that's  part of it such a biz in that clever  though so it it just stores a  predetermined set of registers but  there's another way to restrict the set  only registers uses parameters in the  called function yeah close enough Kali  saved registers so register is that the  that the function might that it's the  responsibility of food is safe this goes  all the way back to that discussion in  lecture I don't remember which which  small number talking about the calling  convention these registers need to be  saved as well as the instruction pointer  and various stack pointers those are  what gets saved into the buffer the  other registers what we're about to call  a function it's up to that other  function to save the registers  appropriately so we're not going to we  don't need to worry about those yeah so  good any questions about that all right  so this edge up routine let's take it  for granted that when we call a set John  Ball in this given buffer it returns  zero that's a good lie for now we'll  just run with it so such up return zero  the condition says if not zero which is  which turns out to be true and so the  next thing that happens is this call to  the spawn helper spawn underscore bar in  this case when we call spawn spawn  underscore bar what happens to our stack  some of this should look pretty routine  for doing a function call and so we push  the frame for the called function onto  the stack and that called functions bond  bar contains a local variable which is  this so cardia stack frame so that also  gets pushed onto the stack  pretty straightforward we've seen  function calls many times before this  should look pretty familiar now we do  this silk RTS enter frame fast routine  and I mentioned before that that's going  to update the worker structure and the  deck or no not the deck just a worker  structure so what's going to happen here  well we have a brand-new so card to a  stack frame on the on the stack any  guesses as to what change we make  what would enter into yeah  [Music]  point currents a crane to spawn bar  Sacre you're right anything else  hope I got this animation right  what are the various fields within the  stock frame and what did sorry I don't  know your name what's your name Greg  what did Greg ask about before when we  saw an earlier picture of the call stack  set a pointer to the parent exactly so  we're gonna do is we're gonna take this  call stack we do the enter frame fast  routine that establishes this parent  pointer in our brand new calls in our  brand new stack frame and we update the  workers current stack frame to point at  the bottom yes question  [Music]  how does interframe know what the parent  is good question enter frame knows the  worker or rather enter frame can do a  call which will give it access to the  Silk worker structure and because it can  do a call it can read the current  saccharin pointer in the worker  [Music]  yeah in this case we do yep so we add  the parent pointer then we delete and  update so good catch any other questions  cool alright now we encounter this thing  so card taste attached this one's kind  of exciting finally we got to do do  something to the deck any guesses what  we do how do we update the deck so card  is here here's a hint  so card taste attached allows this is  the function that allows some  computation to be stolen once once silk  artistic is done executing a thief could  come along and steal the continuation of  the silk spawn so what would silk RTS  detach due to our worker and it's  structures yeah and back  push the sacrum to the worker deck  specifically at the tail now right I  gave it away with by clicking the  animation oh now the thing that's  available to be stolen is inside of Fuu  so what ends up getting pushed on to the  deck is not the current stack frame but  in fact its immediate parent so the the  sacrum foo that gets pushed onto the  tail of the deck and we now push  something onto the tail of a deck and so  we advanced the table pointer is so good  everyone I see some nods I see at least  one nod I'll take it but feel free to  ask questions of course and then of  course there's this invocation of bar  this does what you might expect  it calls bar no magic here well no new  magic here okay fast-forward let's  suppose that bar finally returns and now  we return to the two same adapter bar in  the spawn helper that statement is the  pop frame actually says we just returned  from bar we need get rid of bar from the  stack frame good now we can actually get  the pop frame  what would the pop frame do it's going  to clean up the stack frame structure so  what would what would that entail any  guesses  move the currents a crane back to the  parent very good they think  that's largely it I guess there's one  other thing you can do this is kind of  optional given that it's gonna garbage  the garbage the memory anyway so it  updates the current stack frame to point  to the parent and now I no longer needs  that parent pointer so we can clean that  up in principle and then there's this  call to soak RTS leave frame this is  magic well not really but it's not  obvious this is a function call that may  or may not return  welcome to the silk runtime system you  end up with calls to functions that you  may never return from this happens all  the time  and these soak our DS leave frame may or  may not return based entirely on what's  on the status of the deck what content  is currently sitting on this on the  workers deck so why anyone have a guess  as to why the leave frame routine might  not return in the conventional sense  there's nothing if there's nothing left  to do on the deck then it's going to  start shakin right if there's nothing on  the deck then it then it has nowhere to  return to and so naturally as we've seen  from silk workers in this path in the  past it discovers there's nothing on the  deck there's no work to do time to turn  to a life of crime and try to steal work  from someone else so there are two  possible scenarios it could the pop  because the seed and execution power  continues as normal or it fails and it  becomes a thief now which of these two  cases do you think is more important for  the runtime system to optimize  [Music]  success case one exactly so why is that  at least we hope so yeah we assumed this  hearkens all the way back to the two  that work first principle we assume that  in the common case workers are doing  useful work they're not just spending  their time stealing from each other and  therefore ideally we want we want to  assume that the worker will do what's  normal just an ordinary serial execution  in a normal serial execution there's  something on the deck the pop succeeds  that's case one and so what we'll see is  that the runtime system in fact does a  little bit of optimization on case one  let's talk about something a little more  exciting  how about stealing computation we like  stealing stuff from each other yes  yep where does it return the results so  where does it return the result in the  spawn bar this actually the answer you  can kind of see two lines above this so  in this case in the original SIL code we  had x equals silk spawn of bar and here  what are the parameters to our spawn bar  our function  X and n now n is the input to bar right  so what's X  you can rewind a little bit and see  there you are correct  oops there you yeah so the original silk  code we had x equals silk spawn bar  that's the same X all that silk does is  pass a pointer to the memory allocator  for that variable down to the spawn  helper and now the spawn helper when it  calls bar and that returns it gets  stored into that storage in the parent  stack frame good good catch  good observation any questions about  that that makes sense cool  probably use too many animations in  these slides alright now let's talk  about stealing how do we how does a  worker steal computation now the  conceptual diagram we had before saw  this one worker with nothing on its deck  take a couple frames from another  workers deck and just slide them on over  what does that actually look like in the  implementation well we're still going to  take from the top of the deck but now we  have a picture that's a little bit more  accurate in terms of the structures that  are really implemented in the system so  we have the call sack of the victim and  the victim also has a deck data  structure and a silk worker data  structure with head and tail pointers  and a current stack frame so what  happens when a thief comes along out of  nowhere its bored it has nothing on its  deck head and tail pointers both point  to the top current stack frame has  nothing  what's the thief going to do any guesses  how does this thief take the content  from the workers deck  [Music]  that's their Instagram  [Music]  exactly right yep sorry was that I  didn't mean to interrupt  all right cool so the red highlighting  should give a little bit of a hint the  current stack frame in the thief is  going to end up pointing to the stack  frame at the top of the deck pointed to  you by the top of the deck and the head  of the deck needs to be updated so let's  just see all those pointers shuffle the  thief is going to take is going to  target the head of the deck it's going  to DQ that item from the from the top of  the deck it's going to set the current  stack frame to point to that item and  it'll delete the pointer on the deck  that makes sense cool now the victim and  the thief are on different processors  and this scenario involves shuffling a  lot of pointers around so we think about  if we think about this process there  needs to be some way to handle the  concurrent accesses that are gonna occur  on the head of the deck you haven't  talked about synchronization yet in this  class that's going to be a couple of  lectures down the road there are a  couple I'll give you a couple spoilers  for those synchronization lectures first  off synchronization is expensive and  second reason about synchronization is a  source of massive headaches  congratulations you now know those two  lectures I'm just gonna go to the  lectures you'll learn a lot they're  great in this silk in the silk runtime  system the way that those concurrent  accesses are handled the pro is by using  a protocol known as the t-h-e protocol  this is pseudocode for most logic in the  THC protocol there's a protocol that the  worker executing work normally follows  and there's the protocol for the thief  I'm not gonna walk through all the lines  of code here and and describe what they  do I'll just give you the very  high-level view of this protocol from  the theists perspective  the thief always grabs a lock on the  deck before doing any operations on the  deck always acquire the lock first for  the worker it's a little bit more  optimized so what the worker will do is  optimistically try to pop something from  the bottom of the deck and only if it  looks like that pop operation fails does  the worker do something more complicated  only then does it try to acquire a lock  on the deck then try to pop something  off see if it really succeeds or fails  and you know possibly turn to a life of  crime so the workers protocol looks  longer but that's just because the the  worker implements a special case which  which is optimized for the common case  this is essentially where the leave  frame routine that we saw before is  optimized for case one optimize for the  pop from the deck succeeding any  questions about that seem clear from  30,000 feet cool okay so that's how a  worker steals work from the top of  another from the top of a victim's deck  now that thief needs to resume a  continuation and this is that that whole  process about jumping into the middle of  an executing function it already has a  frame it already has a bunch of state  going on and somehow and all that was  established by a different processor so  somehow that thief has to magically come  up with the right state and start  executing that function how does that  happen well this has to do with a  routine that's the complement of the set  up routine we saw before the the  complement of set jump is what's called  long jump  so silk uses in particular silk thieves  use the long jump function in order to  resume a sole in continuation previously  in our spawning function foo we had this  set jump call and that set jump save  some states to a local buffer in  particularly the buffer  in the stack frame of foo now the thief  has just created this silk worker  structure where the current stack frame  is pointing at the stack frame of foo  and so what the thief will do is it will  execute a call to the it'll execute the  statement long jump pass it Aleksey the  long jump function passing that  particular stack frames buffer and an  additional argument and that long jump  will take the register states throw it  in the buffer put that register state  into the worker and then what the worker  proceed that makes sense  any questions about that this is kind of  a wacky routine because if you remember  one of the one of the registers stored  in that buffer is an instruction pointer  and so it's going to read and read the  instruction pointer out of the buffer  it's also gonna read a bunch of kali  saved registers and stack pointers out  of the buffer and i'm just gonna say  those are my those are that's my  register state now that's what the thief  says it just stole that register state  and it's going to set its RI p to be the  RIPD it just read so where does that so  what does that mean for where the long  jump routine returns  yeah returns in the stock frame above  the one who just stole uh more or less  can but more specifically we're in that  function does it return which call to  the the spawn bar here almost very very  close very very close what ends up  happening is that the long jump  effectively returns from the set jump a  second time this is the weird protocol  between set jump and long jump set jump  you pass it a buffer it saves some  register state and then it returns and  it returns immediately and on it's  direct invocation that set jump call  returns the value zero as we mentioned  before now if you invoke a long jump  using the same buffer that causes the  processor to effectively return from the  same set jump call they use the same  buffer but now it's going to return with  a different value and it's going to  return with the value specified in the  second argument so invoking long jump of  buffer X returns from that set jump with  the value X so when the so when the  thief executes a long jump with the  appropriate buffer and the second  argument is one what happens anyone walk  me through this oh well it's on the  slide okay so now set that set of  effectively returns a second time but  now it returns with the value one and  now the predicate the predicate gets  evaluated so if not one which would be  if false well don't do the consequent  because you're just because the  predicate was false and that means it's  gonna skip the call to spawn bar and  it'll just fall through and execute the  stuff right after that condition that  conditional  which happens to be the continuation of  the spawn that's kind of neat I think  that's kind of neat maybe I'm biased  anyone else think that I need actually  anyone desperately confused about this  such a long term nonsense any questions  you want to ask or just generally  confused about why these things exist in  modern computing yeah is there any  reason you couldn't just add some fixed  offset to the instruction pointer to  jump over the call in principle I think  if you can statically compute the  distance you need to jump then you can  just add that to our IP and let the long  jump  do its thing or rather the thief will  just adopt that our IP and end up in the  right place what's done here is  basically this was the protocol that  they set existing such up and long term  routines implement and I imagine it's a  more it's a bit more flexible of a  protocol then well you strictly need for  the silk runtime and so you know it adds  up working on but if you can statically  compute that offset there's no reason in  principle you couldn't adopt a different  approach so good observation any  questions any other questions  it's fine to be generally confused why  their routines such a been long jump  with this whacky behavior compiler  writers have that reaction all the time  these are a nightmare to compile but  anyway okay so we've seen how the how a  thief can take some computation off of a  victim's deck and we've seen how the  thief can jump right into the middle of  an executing function with the  appropriate register state is this the  end of the story is there anything else  we need to talk about with respect to  stealing or more pointedly what else do  we need to talk about with respect to  stealing  you're welcome to answer you like  okay I remember that list of concerns we  had at the beginning list of  requirements things about it's called  we will talk about sinks but not just  yet what other thing was brought up as  remember this slide from a previous  lecture here's another hint so the  register register state is certainly  part of the state of an executing  function what else defines a state of an  executing function whereas the other  state of the function where does that  live it lives on the stack so what is  there to talk about regarding the stack  the cactus stack exactly so we mentioned  before that thieves need to implement  this cactus stack abstraction for so for  the silk runtime system what exact why  exactly do we need this cactus stack  what's wrong was just having the thief  have stock used the victims stack yep  the victim might just free up a bunch of  stuff ends that it's no longer  accessible so it can free some amount of  some amount of stuff in particular  everything up to the function foo but in  fact it can't return from the function  foo because some other well assuming  that the silk RT has leaf frame thing is  implemented the function foo is no  longer in a stack it won't ever reach it  bla bla bla  so it won't return from the function foo  while another worker is working on it  but good observation there's something  else that can go wrong if the thief just  directly uses the victim's stack  well let's take a hint from the slide we  have so far so the example that's going  to be shown is that the thief steals the  continuation of foo and then the thief  is going to call a function Baz  so the thief is using the victim's stack  and then it calls a function Baz what  goes wrong  [Music]  exactly the victim in this picture for  example has some other functions on its  stack below foo so if the thief does any  function calls and is using the same  stack it's gonna scribble all over the  state of in this case spawn bar and bar  which the victim is trying to use and  maintain so the thief will end up  corrupting the victims stack and if you  think about it it's also possible for  the victim to grab the thief's stack  involved they can't they can't share a  stack but they do want to share some  amount of some amount of data on the  stack they do both care about the state  of foo and that needs to be consistent  across all the workers but you at least  need a separate call stack for the thief  we'd rather not do unnecessary work in  order to initialize this call stack  however we really need this call stack  for things that the thief might might  evoke variable is that the local  variables the thief might need or  functions that the thief might call or  spawn okay so how do we implement the  call stack sorry the cactus stuck we  have a victim sack the thief stack and  we have a pretty cute trick in my  opinion so the thief steals this  continuation it's not it's gonna do a  little bit of magic with its stack  pointers what it's gonna do is it's  going to use the rbp it was given which  points out the victim stack and it's  gonna set the stack pointer to point at  its own stack so RB P is over there and  RSP for the thief is pointing to the  beginning of the thief's call stack and  that is basically fine  the thief can access all the state in  the function foo as offsets from RB p  but if the thief needs to do any  function calls we have a calling  convention that involves saving our VP  and updating our SP in order to execute  the call so in particular the thief  calls a function bars  it saves its current value of our VP on  to its own stack okay advances RRSP it  says RB P equal to RSP it pushes the  stack frame for Baz onto the stack and  it advances RSP a little bit further and  just like that the thief is churning  away on its own stack so just with this  magic of our VP pointing there and rst  pointing here we get our cactus stack  everyone followed that anyone  desperately confused by this stack  pointer base pointer nonsense  who thinks is kind of a neat trick all  right cool  everything is a really mundane trick  hopefully no things are the mundane  trick okay there's like half a hand  there that's fine noise I think this is  a neat trick just messing around with  the stack pointers are there any worries  about using our VP and our SP this way  any any concerns that you might think of  from using using these two stack  pointers as described in a past lecture  briefly mentioned was a compiler  optimization for dealing with stacks  yeah  three of them  right there was a compiler optimization  that said in certain cases you don't  need both the base pointer and the stack  pointer you can do all offsets I think  it's actually off the stack pointer and  then the base pointer becomes an  additional general purpose register that  optimization clearly does not work if  you need the base pointer stack pointer  to do this wacky to do this wacky trick  the answer is that the SIL compiler  specifically specifically says if this  function has a continuation that could  be stolen don't do that optimization  it's super illegal it's very bad don't  do the optimization it's not about being  the answer and it costs us a general  purpose register for silk functions not  the biggest loss in the world  all right there's a little bit of time  left so we can talk about synchronizing  computation I'll give you the brief  version of this this part gets fairly  fairly complicated and so I'll give you  a high-level summary of how all this  works  so just to page back in some contexts we  have this scenario where different  processors are executing different parts  of our computation dag and one processor  might encounter a silk sync statement  that it can't execute because some other  processor is busy executing some spawned  sub computation now in this case P 3 is  waiting on P 1 to finish its execution  before the sync can proceed and  synchronization needs to happen really  only on the sub computation that P 1 is  executing P 2 shouldn't play a role in  this so what exactly happens when a  worker reaches the silk sync before all  the spawns sub computations return well  we'd like to worker to become a thief  we'd rather the worker not just sit  there and wait until all this balance of  computation is returned that's a waste  of a perfectly good worker but we also  can't let the workers current function  frame disappear there's a spawn sub  computation that's using that frame that  frame is its parent  it may be accessing state in that frame  and maybe trying to save a return value  to some location in that frame and so  the frame has to has to persist even if  the worker that's working on the frame  goes off and becomes a thief moreover in  the future that's up computation we  believe should return and that worker  must resume the frame and actually  execute past the silk sync finally the  silk sync should only apply to the  nested sub computations underneath this  function not the program in general and  so we don't we don't allow ourselves  synchronization just among all the  workers wholesale we don't say okay  we've hit a sync every worker in the  system must reach some point in the  execution we only care about this nested  synchronization so if we think about  this and we're talking about nested  synchronization for computations under a  function we have this notion of cactus  stack we have this notion of a tree of  function invocations we may immediately  start to think about well what if we  just maintain some state in a tree to  keep track of who needs to synchronize  with whom which computations are waiting  on which other computations to finish  and in fact that's essentially what this  so part-time system does it maintains a  tree of states called full frames and  those full frames store state for the  parallel of sub computations and it  keeps track those full frames keep track  which sub computations are outstanding  and how they relate to each other this  is a high-level picture of a full frame  there are some there lots of details  delighted to be honest but at 30,000  feet a full frame keeps track of a bunch  of information for the parallel  execution I know I'm giving you the  quick version of this including pointers  to parent frames and possibly pointers  to child frames or at least the number  of outstanding child frames the  processors when visit system work on  what are called active full frames in  the diagram those full frames  there are the rounded rectangles  highlight and dark blue other full  frames in the system are what we call  suspended there they're waiting on some  sub computation to return that's what a  full frame tree can look like under some  execution let's see how a full frame  tree can come into being just by working  through an animation so suppose we have  some worker with a bunch of spawn and  called frames on its deck no one else  has no other workers have anything on  their decks and and finally some worker  says it wants to steal sub car it wants  a steal and I'll admit this animation is  crafted slightly just to make the  pictures a little bit nicer it can look  more complicated in practice don't worry  if that was actually worried of yours so  what's gonna so what's gonna happen the  thief is gonna take some frames from the  top of the victim's deck and it's  actually going to steal not just those  frames but the whole full frame  structure along with it the full frame  structure is just represented with this  rounded rectangle in fact it's a  constant size thing but the theme is  gonna take the whole full frame  structure and it's going to give the  victim a brand new full frame and  establish the the child to parent  pointer in that in the victims new full  frame that's kind of weird it's not  obvious why the thief would take the  full frame as a stealing computation at  least not from one step but we can see  why it helps just given one more step so  let's fast forward this picture a little  bit and now we have another worker try  to steal some sub computation some  computation and we have a little bit  more stuff going on so this worker might  randomly select the last worker on the  right steal a computation from the top  of its deck and it's gonna steal the  full frame along with those along with  the the deck frames  and because it stole the full-frame all  pointers to that full-frame from any  child sub computations are still valid  the child self computation on the Left  still points to the correct full-frame  the the full frame that was stolen has  the parent context of that child and so  we need to make sure that pointer is  still it's still good if if it created a  new full frame for itself then you would  have to update the child pointers  somehow and that requires more  synchronization and a more complicated  protocol synchronization is expensive  protocols are complicated this ends up  saving some complexity and then it  creates a frame for the child and we can  see this process unfold just a little  bit further  hello behold after a few steals we end  up with a tree we have to two children  pointing to one parent and one of those  children has its own child  great now suppose that some worker says  oh I encountered a sink can i  synchronize in this case the worker has  an outstanding child computation so I  can't synchronize and so we can't  recycle the full frame we can't recycle  any of the any of the stack for this  child and so instead the worker will  suspend its full frame turn you from  dark blue light blue in our picture and  it goes and becomes a thief  the program has ample parallelism what  do you expect to typically happen when  the program execution reaches a silk  sync we're kind of out of time so I  think I'm just gonna spoil the answer  for this unless anyone has it has a  guest handy so what's the common case  for a silk sync  for the sake of time the common case is  that the executing function has no  outstanding children all the workers on  the system we're busy doing their own  thing  there's no synchronization that's  necessary and so how is the runtime  optimize this case it ends up using some  bits yeah ends up having the full frame  use some bits of an Associated stack  frame in particular flag field  and that's why when we look at the  compiled code for a silk sync we see  some conditions that that evaluate the  flags within the within the local stack  frame that's just an optimization to say  if you don't need a sync don't do any  computation otherwise otherwise some  steals really did occur go ahead and  execute the silk RTS sync routine there  are a bunch of other runtime features if  you take a look at that picture for a  long time you may be dissatisfied with  what that implies about some of the  protocols and there's a lot more code  within the runtime system itself to  implement a variety of other features  such as support for C++ exceptions  reducer hyper objects and a form of ID  ID is called pedigrees we won't talk  about that today I'm actually all out of  time thanks for listening to all the all  this about the soap run type system feel  free to ask any questions after your  class
 

3
00:00:05,769 --> 00:00:08,019

 

4
00:00:08,019 --> 00:00:09,850

 

5
00:00:09,850 --> 00:00:10,930

 

6
00:00:10,930 --> 00:00:13,120

 

7
00:00:13,120 --> 00:00:15,160

 

8
00:00:15,160 --> 00:00:21,790

 

9
00:00:21,790 --> 00:00:26,600

 

10
00:00:26,600 --> 00:00:31,160

 

11
00:00:31,160 --> 00:00:36,830

 

12
00:00:36,830 --> 00:00:41,750

 

13
00:00:41,750 --> 00:00:44,420

 

14
00:00:44,420 --> 00:00:44,430

 

15
00:00:44,430 --> 00:00:45,280


16
00:00:45,280 --> 00:00:49,549

 

17
00:00:49,549 --> 00:00:51,950

 

18
00:00:51,950 --> 00:00:53,689

 

19
00:00:53,689 --> 00:00:55,040

 

20
00:00:55,040 --> 00:00:56,990

 

21
00:00:56,990 --> 00:01:00,590

 

22
00:01:00,590 --> 00:01:03,439

 

23
00:01:03,439 --> 00:01:06,440

 

24
00:01:06,440 --> 00:01:08,719

 

25
00:01:08,719 --> 00:01:11,120

 

26
00:01:11,120 --> 00:01:14,870

 

27
00:01:14,870 --> 00:01:17,899

 

28
00:01:17,899 --> 00:01:21,700

 

29
00:01:21,700 --> 00:01:23,990

 

30
00:01:23,990 --> 00:01:26,960

 

31
00:01:26,960 --> 00:01:29,480

 

32
00:01:29,480 --> 00:01:31,250

 

33
00:01:31,250 --> 00:01:33,650

 

34
00:01:33,650 --> 00:01:37,190

 

35
00:01:37,190 --> 00:01:39,640

 

36
00:01:39,640 --> 00:01:42,740

 

37
00:01:42,740 --> 00:01:45,230

 

38
00:01:45,230 --> 00:01:48,950

 

39
00:01:48,950 --> 00:01:51,290

 

40
00:01:51,290 --> 00:01:52,970

 

41
00:01:52,970 --> 00:01:55,880

 

42
00:01:55,880 --> 00:01:58,220

 

43
00:01:58,220 --> 00:02:00,560

 

44
00:02:00,560 --> 00:02:02,240

 

45
00:02:02,240 --> 00:02:05,780

 

46
00:02:05,780 --> 00:02:07,999

 

47
00:02:07,999 --> 00:02:11,029

 

48
00:02:11,029 --> 00:02:13,369

 

49
00:02:13,369 --> 00:02:14,920

 

50
00:02:14,920 --> 00:02:17,630

 

51
00:02:17,630 --> 00:02:20,559

 

52
00:02:20,559 --> 00:02:23,089

 

53
00:02:23,089 --> 00:02:25,280

 

54
00:02:25,280 --> 00:02:28,339

 

55
00:02:28,339 --> 00:02:32,600

 

56
00:02:32,600 --> 00:02:36,130

 

57
00:02:36,130 --> 00:02:39,170

 

58
00:02:39,170 --> 00:02:42,440

 

59
00:02:42,440 --> 00:02:44,750

 

60
00:02:44,750 --> 00:02:46,970

 

61
00:02:46,970 --> 00:02:50,990

 

62
00:02:50,990 --> 00:02:53,059

 

63
00:02:53,059 --> 00:02:55,430

 

64
00:02:55,430 --> 00:02:57,590

 

65
00:02:57,590 --> 00:02:59,720

 

66
00:02:59,720 --> 00:03:00,610

 

67
00:03:00,610 --> 00:03:02,750

 

68
00:03:02,750 --> 00:03:04,729

 

69
00:03:04,729 --> 00:03:07,400

 

70
00:03:07,400 --> 00:03:10,040

 

71
00:03:10,040 --> 00:03:12,110

 

72
00:03:12,110 --> 00:03:14,539

 

73
00:03:14,539 --> 00:03:17,509

 

74
00:03:17,509 --> 00:03:19,130

 

75
00:03:19,130 --> 00:03:22,130

 

76
00:03:22,130 --> 00:03:24,530

 

77
00:03:24,530 --> 00:03:28,100

 

78
00:03:28,100 --> 00:03:30,289

 

79
00:03:30,289 --> 00:03:32,870

 

80
00:03:32,870 --> 00:03:35,360

 

81
00:03:35,360 --> 00:03:37,490

 

82
00:03:37,490 --> 00:03:39,590

 

83
00:03:39,590 --> 00:03:43,340

 

84
00:03:43,340 --> 00:03:45,170

 

85
00:03:45,170 --> 00:03:47,080

 

86
00:03:47,080 --> 00:03:50,420

 

87
00:03:50,420 --> 00:03:53,030

 

88
00:03:53,030 --> 00:03:56,509

 

89
00:03:56,509 --> 00:04:00,170

 

90
00:04:00,170 --> 00:04:02,569

 

91
00:04:02,569 --> 00:04:06,050

 

92
00:04:06,050 --> 00:04:07,849

 

93
00:04:07,849 --> 00:04:10,789

 

94
00:04:10,789 --> 00:04:13,699

 

95
00:04:13,699 --> 00:04:17,390

 

96
00:04:17,390 --> 00:04:19,880

 

97
00:04:19,880 --> 00:04:22,400

 

98
00:04:22,400 --> 00:04:24,230

 

99
00:04:24,230 --> 00:04:26,840

 

100
00:04:26,840 --> 00:04:29,150

 

101
00:04:29,150 --> 00:04:31,370

 

102
00:04:31,370 --> 00:04:37,430

 

103
00:04:37,430 --> 00:04:39,890

 

104
00:04:39,890 --> 00:04:41,810

 

105
00:04:41,810 --> 00:04:43,460

 

106
00:04:43,460 --> 00:04:43,470

 

107
00:04:43,470 --> 00:04:45,290


108
00:04:45,290 --> 00:04:48,750

 

109
00:04:48,750 --> 00:04:50,700

 

110
00:04:50,700 --> 00:04:54,650

 

111
00:04:54,650 --> 00:04:56,820

 

112
00:04:56,820 --> 00:04:58,560

 

113
00:04:58,560 --> 00:05:00,420

 

114
00:05:00,420 --> 00:05:02,610

 

115
00:05:02,610 --> 00:05:04,860

 

116
00:05:04,860 --> 00:05:07,680

 

117
00:05:07,680 --> 00:05:09,690

 

118
00:05:09,690 --> 00:05:12,420

 

119
00:05:12,420 --> 00:05:14,010

 

120
00:05:14,010 --> 00:05:16,680

 

121
00:05:16,680 --> 00:05:18,090

 

122
00:05:18,090 --> 00:05:19,950

 

123
00:05:19,950 --> 00:05:21,810

 

124
00:05:21,810 --> 00:05:24,660

 

125
00:05:24,660 --> 00:05:26,670

 

126
00:05:26,670 --> 00:05:30,210

 

127
00:05:30,210 --> 00:05:32,490

 

128
00:05:32,490 --> 00:05:34,290

 

129
00:05:34,290 --> 00:05:36,600

 

130
00:05:36,600 --> 00:05:38,550

 

131
00:05:38,550 --> 00:05:41,100

 

132
00:05:41,100 --> 00:05:43,470

 

133
00:05:43,470 --> 00:05:44,820

 

134
00:05:44,820 --> 00:05:47,190

 

135
00:05:47,190 --> 00:05:50,250

 

136
00:05:50,250 --> 00:05:52,920

 

137
00:05:52,920 --> 00:05:55,230

 

138
00:05:55,230 --> 00:05:57,090

 

139
00:05:57,090 --> 00:06:01,440

 

140
00:06:01,440 --> 00:06:04,080

 

141
00:06:04,080 --> 00:06:11,240

 

142
00:06:11,240 --> 00:06:13,050

 

143
00:06:13,050 --> 00:06:16,020

 

144
00:06:16,020 --> 00:06:19,200

 

145
00:06:19,200 --> 00:06:20,790

 

146
00:06:20,790 --> 00:06:23,190

 

147
00:06:23,190 --> 00:06:24,930

 

148
00:06:24,930 --> 00:06:27,660

 

149
00:06:27,660 --> 00:06:28,710

 

150
00:06:28,710 --> 00:06:30,420

 

151
00:06:30,420 --> 00:06:32,160

 

152
00:06:32,160 --> 00:06:34,470

 

153
00:06:34,470 --> 00:06:38,370

 

154
00:06:38,370 --> 00:06:41,010

 

155
00:06:41,010 --> 00:06:43,050

 

156
00:06:43,050 --> 00:06:46,440

 

157
00:06:46,440 --> 00:06:49,020

 

158
00:06:49,020 --> 00:06:53,040

 

159
00:06:53,040 --> 00:06:55,620

 

160
00:06:55,620 --> 00:06:58,930

 

161
00:06:58,930 --> 00:07:00,880

 

162
00:07:00,880 --> 00:07:03,120

 

163
00:07:03,120 --> 00:07:07,180

 

164
00:07:07,180 --> 00:07:09,010

 

165
00:07:09,010 --> 00:07:10,960

 

166
00:07:10,960 --> 00:07:12,880

 

167
00:07:12,880 --> 00:07:15,040

 

168
00:07:15,040 --> 00:07:18,280

 

169
00:07:18,280 --> 00:07:21,940

 

170
00:07:21,940 --> 00:07:26,770

 

171
00:07:26,770 --> 00:07:29,620

 

172
00:07:29,620 --> 00:07:31,600

 

173
00:07:31,600 --> 00:07:34,270

 

174
00:07:34,270 --> 00:07:36,130

 

175
00:07:36,130 --> 00:07:38,290

 

176
00:07:38,290 --> 00:07:42,250

 

177
00:07:42,250 --> 00:07:43,750

 

178
00:07:43,750 --> 00:07:49,320

 

179
00:07:49,320 --> 00:08:04,710

 

180
00:08:04,710 --> 00:08:04,720

 

181
00:08:04,720 --> 00:08:08,860


182
00:08:08,860 --> 00:08:10,870

 

183
00:08:10,870 --> 00:08:14,680

 

184
00:08:14,680 --> 00:08:18,820

 

185
00:08:18,820 --> 00:08:20,920

 

186
00:08:20,920 --> 00:08:24,010

 

187
00:08:24,010 --> 00:08:25,420

 

188
00:08:25,420 --> 00:08:27,909

 

189
00:08:27,909 --> 00:08:29,290

 

190
00:08:29,290 --> 00:08:31,780

 

191
00:08:31,780 --> 00:08:33,399

 

192
00:08:33,399 --> 00:08:35,829

 

193
00:08:35,829 --> 00:08:38,200

 

194
00:08:38,200 --> 00:08:40,540

 

195
00:08:40,540 --> 00:08:42,490

 

196
00:08:42,490 --> 00:08:44,890

 

197
00:08:44,890 --> 00:08:50,470

 

198
00:08:50,470 --> 00:08:52,900

 

199
00:08:52,900 --> 00:08:54,640

 

200
00:08:54,640 --> 00:08:58,630

 

201
00:08:58,630 --> 00:08:59,950

 

202
00:08:59,950 --> 00:09:01,420

 

203
00:09:01,420 --> 00:09:03,730

 

204
00:09:03,730 --> 00:09:06,610

 

205
00:09:06,610 --> 00:09:09,070

 

206
00:09:09,070 --> 00:09:10,840

 

207
00:09:10,840 --> 00:09:12,760

 

208
00:09:12,760 --> 00:09:14,770

 

209
00:09:14,770 --> 00:09:16,780

 

210
00:09:16,780 --> 00:09:22,060

 

211
00:09:22,060 --> 00:09:25,329

 

212
00:09:25,329 --> 00:09:28,090

 

213
00:09:28,090 --> 00:09:29,829

 

214
00:09:29,829 --> 00:09:31,360

 

215
00:09:31,360 --> 00:09:35,230

 

216
00:09:35,230 --> 00:09:39,520

 

217
00:09:39,520 --> 00:09:41,170

 

218
00:09:41,170 --> 00:09:43,930

 

219
00:09:43,930 --> 00:09:46,870

 

220
00:09:46,870 --> 00:09:48,250

 

221
00:09:48,250 --> 00:09:50,590

 

222
00:09:50,590 --> 00:09:54,520

 

223
00:09:54,520 --> 00:09:57,550

 

224
00:09:57,550 --> 00:10:00,520

 

225
00:10:00,520 --> 00:10:03,400

 

226
00:10:03,400 --> 00:10:05,590

 

227
00:10:05,590 --> 00:10:08,140

 

228
00:10:08,140 --> 00:10:11,740

 

229
00:10:11,740 --> 00:10:13,900

 

230
00:10:13,900 --> 00:10:16,060

 

231
00:10:16,060 --> 00:10:17,980

 

232
00:10:17,980 --> 00:10:20,690

 

233
00:10:20,690 --> 00:10:22,670

 

234
00:10:22,670 --> 00:10:28,850

 

235
00:10:28,850 --> 00:10:30,230

 

236
00:10:30,230 --> 00:10:32,030

 

237
00:10:32,030 --> 00:10:35,030

 

238
00:10:35,030 --> 00:10:36,680

 

239
00:10:36,680 --> 00:10:38,930

 

240
00:10:38,930 --> 00:10:41,000

 

241
00:10:41,000 --> 00:10:43,730

 

242
00:10:43,730 --> 00:10:46,100

 

243
00:10:46,100 --> 00:10:48,850

 

244
00:10:48,850 --> 00:10:51,170

 

245
00:10:51,170 --> 00:10:53,660

 

246
00:10:53,660 --> 00:10:55,960

 

247
00:10:55,960 --> 00:10:58,700

 

248
00:10:58,700 --> 00:11:01,250

 

249
00:11:01,250 --> 00:11:03,110

 

250
00:11:03,110 --> 00:11:05,870

 

251
00:11:05,870 --> 00:11:08,840

 

252
00:11:08,840 --> 00:11:10,820

 

253
00:11:10,820 --> 00:11:13,640

 

254
00:11:13,640 --> 00:11:15,680

 

255
00:11:15,680 --> 00:11:18,440

 

256
00:11:18,440 --> 00:11:21,590

 

257
00:11:21,590 --> 00:11:24,500

 

258
00:11:24,500 --> 00:11:27,490

 

259
00:11:27,490 --> 00:11:32,300

 

260
00:11:32,300 --> 00:11:34,940

 

261
00:11:34,940 --> 00:11:36,530

 

262
00:11:36,530 --> 00:11:38,930

 

263
00:11:38,930 --> 00:11:40,550

 

264
00:11:40,550 --> 00:11:44,570

 

265
00:11:44,570 --> 00:11:46,490

 

266
00:11:46,490 --> 00:11:48,200

 

267
00:11:48,200 --> 00:11:50,360

 

268
00:11:50,360 --> 00:11:53,630

 

269
00:11:53,630 --> 00:11:57,050

 

270
00:11:57,050 --> 00:11:58,250

 

271
00:11:58,250 --> 00:12:00,730

 

272
00:12:00,730 --> 00:12:02,930

 

273
00:12:02,930 --> 00:12:04,130

 

274
00:12:04,130 --> 00:12:06,650

 

275
00:12:06,650 --> 00:12:09,320

 

276
00:12:09,320 --> 00:12:12,050

 

277
00:12:12,050 --> 00:12:14,150

 

278
00:12:14,150 --> 00:12:15,820

 

279
00:12:15,820 --> 00:12:21,830

 

280
00:12:21,830 --> 00:12:23,720

 

281
00:12:23,720 --> 00:12:26,090

 

282
00:12:26,090 --> 00:12:29,150

 

283
00:12:29,150 --> 00:12:30,860

 

284
00:12:30,860 --> 00:12:32,510

 

285
00:12:32,510 --> 00:12:34,300

 

286
00:12:34,300 --> 00:12:38,460

 

287
00:12:38,460 --> 00:12:42,340

 

288
00:12:42,340 --> 00:12:44,530

 

289
00:12:44,530 --> 00:12:46,750

 

290
00:12:46,750 --> 00:12:48,880

 

291
00:12:48,880 --> 00:12:51,850

 

292
00:12:51,850 --> 00:12:54,730

 

293
00:12:54,730 --> 00:12:56,860

 

294
00:12:56,860 --> 00:12:58,420

 

295
00:12:58,420 --> 00:13:01,329

 

296
00:13:01,329 --> 00:13:02,769

 

297
00:13:02,769 --> 00:13:05,290

 

298
00:13:05,290 --> 00:13:07,030

 

299
00:13:07,030 --> 00:13:09,640

 

300
00:13:09,640 --> 00:13:12,880

 

301
00:13:12,880 --> 00:13:16,269

 

302
00:13:16,269 --> 00:13:18,550

 

303
00:13:18,550 --> 00:13:21,820

 

304
00:13:21,820 --> 00:13:23,829

 

305
00:13:23,829 --> 00:13:25,180

 

306
00:13:25,180 --> 00:13:29,340

 

307
00:13:29,340 --> 00:13:31,210

 

308
00:13:31,210 --> 00:13:32,890

 

309
00:13:32,890 --> 00:13:34,600

 

310
00:13:34,600 --> 00:13:36,430

 

311
00:13:36,430 --> 00:13:39,910

 

312
00:13:39,910 --> 00:13:42,160

 

313
00:13:42,160 --> 00:13:46,450

 

314
00:13:46,450 --> 00:13:48,340

 

315
00:13:48,340 --> 00:13:49,930

 

316
00:13:49,930 --> 00:13:53,590

 

317
00:13:53,590 --> 00:13:55,660

 

318
00:13:55,660 --> 00:13:59,079

 

319
00:13:59,079 --> 00:14:03,280

 

320
00:14:03,280 --> 00:14:06,460

 

321
00:14:06,460 --> 00:14:09,340

 

322
00:14:09,340 --> 00:14:13,540

 

323
00:14:13,540 --> 00:14:15,460

 

324
00:14:15,460 --> 00:14:17,860

 

325
00:14:17,860 --> 00:14:20,829

 

326
00:14:20,829 --> 00:14:23,560

 

327
00:14:23,560 --> 00:14:26,650

 

328
00:14:26,650 --> 00:14:29,470

 

329
00:14:29,470 --> 00:14:31,060

 

330
00:14:31,060 --> 00:14:33,070

 

331
00:14:33,070 --> 00:14:35,260

 

332
00:14:35,260 --> 00:14:37,480

 

333
00:14:37,480 --> 00:14:39,910

 

334
00:14:39,910 --> 00:14:42,370

 

335
00:14:42,370 --> 00:14:43,560

 

336
00:14:43,560 --> 00:14:45,570

 

337
00:14:45,570 --> 00:14:47,130

 

338
00:14:47,130 --> 00:14:49,770

 

339
00:14:49,770 --> 00:14:52,770

 

340
00:14:52,770 --> 00:14:56,430

 

341
00:14:56,430 --> 00:14:58,800

 

342
00:14:58,800 --> 00:15:01,140

 

343
00:15:01,140 --> 00:15:02,910

 

344
00:15:02,910 --> 00:15:04,260

 

345
00:15:04,260 --> 00:15:07,350

 

346
00:15:07,350 --> 00:15:09,300

 

347
00:15:09,300 --> 00:15:11,610

 

348
00:15:11,610 --> 00:15:13,140

 

349
00:15:13,140 --> 00:15:14,550

 

350
00:15:14,550 --> 00:15:17,160

 

351
00:15:17,160 --> 00:15:18,330

 

352
00:15:18,330 --> 00:15:20,460

 

353
00:15:20,460 --> 00:15:22,500

 

354
00:15:22,500 --> 00:15:25,020

 

355
00:15:25,020 --> 00:15:27,330

 

356
00:15:27,330 --> 00:15:29,610

 

357
00:15:29,610 --> 00:15:33,920

 

358
00:15:33,920 --> 00:15:36,750

 

359
00:15:36,750 --> 00:15:40,170

 

360
00:15:40,170 --> 00:15:42,660

 

361
00:15:42,660 --> 00:15:44,700

 

362
00:15:44,700 --> 00:15:47,130

 

363
00:15:47,130 --> 00:15:49,610

 

364
00:15:49,610 --> 00:16:07,170

 

365
00:16:07,170 --> 00:16:10,640

 

366
00:16:10,640 --> 00:16:13,800

 

367
00:16:13,800 --> 00:16:17,280

 

368
00:16:17,280 --> 00:16:21,630

 

369
00:16:21,630 --> 00:16:25,860

 

370
00:16:25,860 --> 00:16:27,000

 

371
00:16:27,000 --> 00:16:32,160

 

372
00:16:32,160 --> 00:16:32,170

 

373
00:16:32,170 --> 00:16:32,580


374
00:16:32,580 --> 00:16:34,380

 

375
00:16:34,380 --> 00:16:38,760

 

376
00:16:38,760 --> 00:16:40,470

 

377
00:16:40,470 --> 00:16:42,630

 

378
00:16:42,630 --> 00:16:46,860

 

379
00:16:46,860 --> 00:16:50,280

 

380
00:16:50,280 --> 00:16:53,010

 

381
00:16:53,010 --> 00:16:54,900

 

382
00:16:54,900 --> 00:16:58,920

 

383
00:16:58,920 --> 00:17:01,680

 

384
00:17:01,680 --> 00:17:08,570

 

385
00:17:08,570 --> 00:17:12,600

 

386
00:17:12,600 --> 00:17:14,670

 

387
00:17:14,670 --> 00:17:18,120

 

388
00:17:18,120 --> 00:17:19,740

 

389
00:17:19,740 --> 00:17:21,120

 

390
00:17:21,120 --> 00:17:25,290

 

391
00:17:25,290 --> 00:17:26,430

 

392
00:17:26,430 --> 00:17:28,980

 

393
00:17:28,980 --> 00:17:40,280

 

394
00:17:40,280 --> 00:17:42,050

 

395
00:17:42,050 --> 00:17:45,290

 

396
00:17:45,290 --> 00:17:49,160

 

397
00:17:49,160 --> 00:17:51,830

 

398
00:17:51,830 --> 00:17:53,270

 

399
00:17:53,270 --> 00:17:56,270

 

400
00:17:56,270 --> 00:18:00,310

 

401
00:18:00,310 --> 00:18:03,830

 

402
00:18:03,830 --> 00:18:05,750

 

403
00:18:05,750 --> 00:18:07,340

 

404
00:18:07,340 --> 00:18:09,710

 

405
00:18:09,710 --> 00:18:13,370

 

406
00:18:13,370 --> 00:18:15,050

 

407
00:18:15,050 --> 00:18:16,460

 

408
00:18:16,460 --> 00:18:20,030

 

409
00:18:20,030 --> 00:18:23,450

 

410
00:18:23,450 --> 00:18:25,910

 

411
00:18:25,910 --> 00:18:27,680

 

412
00:18:27,680 --> 00:18:30,980

 

413
00:18:30,980 --> 00:18:33,800

 

414
00:18:33,800 --> 00:18:36,080

 

415
00:18:36,080 --> 00:18:38,990

 

416
00:18:38,990 --> 00:18:42,620

 

417
00:18:42,620 --> 00:18:45,640

 

418
00:18:45,640 --> 00:18:48,620

 

419
00:18:48,620 --> 00:18:50,270

 

420
00:18:50,270 --> 00:18:53,450

 

421
00:18:53,450 --> 00:18:56,240

 

422
00:18:56,240 --> 00:19:00,320

 

423
00:19:00,320 --> 00:19:02,960

 

424
00:19:02,960 --> 00:19:06,620

 

425
00:19:06,620 --> 00:19:08,810

 

426
00:19:08,810 --> 00:19:10,550

 

427
00:19:10,550 --> 00:19:14,390

 

428
00:19:14,390 --> 00:19:16,940

 

429
00:19:16,940 --> 00:19:20,150

 

430
00:19:20,150 --> 00:19:22,250

 

431
00:19:22,250 --> 00:19:24,710

 

432
00:19:24,710 --> 00:19:26,660

 

433
00:19:26,660 --> 00:19:29,630

 

434
00:19:29,630 --> 00:19:32,300

 

435
00:19:32,300 --> 00:19:34,010

 

436
00:19:34,010 --> 00:19:38,210

 

437
00:19:38,210 --> 00:19:40,670

 

438
00:19:40,670 --> 00:19:43,370

 

439
00:19:43,370 --> 00:19:45,410

 

440
00:19:45,410 --> 00:19:47,900

 

441
00:19:47,900 --> 00:19:49,430

 

442
00:19:49,430 --> 00:19:51,650

 

443
00:19:51,650 --> 00:19:53,740

 

444
00:19:53,740 --> 00:19:55,840

 

445
00:19:55,840 --> 00:19:58,390

 

446
00:19:58,390 --> 00:20:00,940

 

447
00:20:00,940 --> 00:20:03,220

 

448
00:20:03,220 --> 00:20:05,530

 

449
00:20:05,530 --> 00:20:07,690

 

450
00:20:07,690 --> 00:20:09,220

 

451
00:20:09,220 --> 00:20:13,930

 

452
00:20:13,930 --> 00:20:15,910

 

453
00:20:15,910 --> 00:20:19,840

 

454
00:20:19,840 --> 00:20:23,830

 

455
00:20:23,830 --> 00:20:27,190

 

456
00:20:27,190 --> 00:20:28,990

 

457
00:20:28,990 --> 00:20:32,230

 

458
00:20:32,230 --> 00:20:34,510

 

459
00:20:34,510 --> 00:20:38,440

 

460
00:20:38,440 --> 00:20:39,790

 

461
00:20:39,790 --> 00:20:42,370

 

462
00:20:42,370 --> 00:20:44,260

 

463
00:20:44,260 --> 00:20:46,060

 

464
00:20:46,060 --> 00:20:51,760

 

465
00:20:51,760 --> 00:20:53,650

 

466
00:20:53,650 --> 00:20:55,570

 

467
00:20:55,570 --> 00:20:57,910

 

468
00:20:57,910 --> 00:21:00,400

 

469
00:21:00,400 --> 00:21:02,890

 

470
00:21:02,890 --> 00:21:04,120

 

471
00:21:04,120 --> 00:21:06,070

 

472
00:21:06,070 --> 00:21:08,320

 

473
00:21:08,320 --> 00:21:10,540

 

474
00:21:10,540 --> 00:21:12,880

 

475
00:21:12,880 --> 00:21:16,900

 

476
00:21:16,900 --> 00:21:19,000

 

477
00:21:19,000 --> 00:21:20,470

 

478
00:21:20,470 --> 00:21:22,900

 

479
00:21:22,900 --> 00:21:25,180

 

480
00:21:25,180 --> 00:21:27,580

 

481
00:21:27,580 --> 00:21:30,460

 

482
00:21:30,460 --> 00:21:33,040

 

483
00:21:33,040 --> 00:21:36,490

 

484
00:21:36,490 --> 00:21:40,810

 

485
00:21:40,810 --> 00:21:43,530

 

486
00:21:43,530 --> 00:21:46,000

 

487
00:21:46,000 --> 00:21:49,240

 

488
00:21:49,240 --> 00:21:52,470

 

489
00:21:52,470 --> 00:21:54,340

 

490
00:21:54,340 --> 00:21:56,320

 

491
00:21:56,320 --> 00:21:57,490

 

492
00:21:57,490 --> 00:22:01,120

 

493
00:22:01,120 --> 00:22:02,800

 

494
00:22:02,800 --> 00:22:05,560

 

495
00:22:05,560 --> 00:22:07,270

 

496
00:22:07,270 --> 00:22:11,820

 

497
00:22:11,820 --> 00:22:13,450

 

498
00:22:13,450 --> 00:22:15,430

 

499
00:22:15,430 --> 00:22:17,770

 

500
00:22:17,770 --> 00:22:21,340

 

501
00:22:21,340 --> 00:22:24,280

 

502
00:22:24,280 --> 00:22:26,050

 

503
00:22:26,050 --> 00:22:27,760

 

504
00:22:27,760 --> 00:22:30,270

 

505
00:22:30,270 --> 00:22:33,040

 

506
00:22:33,040 --> 00:22:35,080

 

507
00:22:35,080 --> 00:22:38,680

 

508
00:22:38,680 --> 00:22:41,260

 

509
00:22:41,260 --> 00:22:43,720

 

510
00:22:43,720 --> 00:22:46,450

 

511
00:22:46,450 --> 00:22:48,820

 

512
00:22:48,820 --> 00:22:51,870

 

513
00:22:51,870 --> 00:22:54,100

 

514
00:22:54,100 --> 00:22:57,010

 

515
00:22:57,010 --> 00:22:58,270

 

516
00:22:58,270 --> 00:23:01,470

 

517
00:23:01,470 --> 00:23:04,180

 

518
00:23:04,180 --> 00:23:06,190

 

519
00:23:06,190 --> 00:23:07,600

 

520
00:23:07,600 --> 00:23:13,300

 

521
00:23:13,300 --> 00:23:15,010

 

522
00:23:15,010 --> 00:23:17,440

 

523
00:23:17,440 --> 00:23:21,910

 

524
00:23:21,910 --> 00:23:24,070

 

525
00:23:24,070 --> 00:23:25,720

 

526
00:23:25,720 --> 00:23:27,700

 

527
00:23:27,700 --> 00:23:29,350

 

528
00:23:29,350 --> 00:23:30,940

 

529
00:23:30,940 --> 00:23:37,300

 

530
00:23:37,300 --> 00:23:39,700

 

531
00:23:39,700 --> 00:23:41,770

 

532
00:23:41,770 --> 00:23:42,910

 

533
00:23:42,910 --> 00:23:46,090

 

534
00:23:46,090 --> 00:23:48,580

 

535
00:23:48,580 --> 00:23:50,680

 

536
00:23:50,680 --> 00:23:54,010

 

537
00:23:54,010 --> 00:23:55,750

 

538
00:23:55,750 --> 00:23:58,650

 

539
00:23:58,650 --> 00:24:01,330

 

540
00:24:01,330 --> 00:24:03,220

 

541
00:24:03,220 --> 00:24:06,370

 

542
00:24:06,370 --> 00:24:09,610

 

543
00:24:09,610 --> 00:24:12,270

 

544
00:24:12,270 --> 00:24:15,160

 

545
00:24:15,160 --> 00:24:18,550

 

546
00:24:18,550 --> 00:24:20,000

 

547
00:24:20,000 --> 00:24:22,820

 

548
00:24:22,820 --> 00:24:25,010

 

549
00:24:25,010 --> 00:24:27,110

 

550
00:24:27,110 --> 00:24:30,920

 

551
00:24:30,920 --> 00:24:34,600

 

552
00:24:34,600 --> 00:24:38,120

 

553
00:24:38,120 --> 00:24:40,130

 

554
00:24:40,130 --> 00:24:41,990

 

555
00:24:41,990 --> 00:24:44,570

 

556
00:24:44,570 --> 00:24:46,520

 

557
00:24:46,520 --> 00:24:48,760

 

558
00:24:48,760 --> 00:24:51,770

 

559
00:24:51,770 --> 00:24:54,260

 

560
00:24:54,260 --> 00:24:56,450

 

561
00:24:56,450 --> 00:24:58,790

 

562
00:24:58,790 --> 00:25:00,470

 

563
00:25:00,470 --> 00:25:03,920

 

564
00:25:03,920 --> 00:25:06,680

 

565
00:25:06,680 --> 00:25:08,960

 

566
00:25:08,960 --> 00:25:11,060

 

567
00:25:11,060 --> 00:25:13,850

 

568
00:25:13,850 --> 00:25:18,340

 

569
00:25:18,340 --> 00:25:22,210

 

570
00:25:22,210 --> 00:25:25,100

 

571
00:25:25,100 --> 00:25:27,470

 

572
00:25:27,470 --> 00:25:29,840

 

573
00:25:29,840 --> 00:25:32,300

 

574
00:25:32,300 --> 00:25:34,400

 

575
00:25:34,400 --> 00:25:37,400

 

576
00:25:37,400 --> 00:25:39,260

 

577
00:25:39,260 --> 00:25:43,010

 

578
00:25:43,010 --> 00:25:45,140

 

579
00:25:45,140 --> 00:25:46,820

 

580
00:25:46,820 --> 00:25:49,130

 

581
00:25:49,130 --> 00:25:51,710

 

582
00:25:51,710 --> 00:25:53,980

 

583
00:25:53,980 --> 00:25:56,630

 

584
00:25:56,630 --> 00:26:03,410

 

585
00:26:03,410 --> 00:26:06,010

 

586
00:26:06,010 --> 00:26:08,060

 

587
00:26:08,060 --> 00:26:10,310

 

588
00:26:10,310 --> 00:26:12,920

 

589
00:26:12,920 --> 00:26:15,260

 

590
00:26:15,260 --> 00:26:18,260

 

591
00:26:18,260 --> 00:26:21,530

 

592
00:26:21,530 --> 00:26:23,750

 

593
00:26:23,750 --> 00:26:25,970

 

594
00:26:25,970 --> 00:26:29,420

 

595
00:26:29,420 --> 00:26:31,010

 

596
00:26:31,010 --> 00:26:33,530

 

597
00:26:33,530 --> 00:26:35,450

 

598
00:26:35,450 --> 00:26:37,130

 

599
00:26:37,130 --> 00:26:40,190

 

600
00:26:40,190 --> 00:26:42,799

 

601
00:26:42,799 --> 00:26:45,580

 

602
00:26:45,580 --> 00:26:47,690

 

603
00:26:47,690 --> 00:26:49,400

 

604
00:26:49,400 --> 00:26:50,540

 

605
00:26:50,540 --> 00:26:52,190

 

606
00:26:52,190 --> 00:26:54,169

 

607
00:26:54,169 --> 00:26:57,110

 

608
00:26:57,110 --> 00:27:00,710

 

609
00:27:00,710 --> 00:27:03,710

 

610
00:27:03,710 --> 00:27:07,010

 

611
00:27:07,010 --> 00:27:10,940

 

612
00:27:10,940 --> 00:27:13,940

 

613
00:27:13,940 --> 00:27:15,680

 

614
00:27:15,680 --> 00:27:21,460

 

615
00:27:21,460 --> 00:27:24,470

 

616
00:27:24,470 --> 00:27:26,600

 

617
00:27:26,600 --> 00:27:28,430

 

618
00:27:28,430 --> 00:27:32,000

 

619
00:27:32,000 --> 00:27:34,640

 

620
00:27:34,640 --> 00:27:37,280

 

621
00:27:37,280 --> 00:27:40,280

 

622
00:27:40,280 --> 00:27:43,160

 

623
00:27:43,160 --> 00:27:45,110

 

624
00:27:45,110 --> 00:27:47,990

 

625
00:27:47,990 --> 00:27:50,570

 

626
00:27:50,570 --> 00:27:53,480

 

627
00:27:53,480 --> 00:27:56,270

 

628
00:27:56,270 --> 00:27:58,010

 

629
00:27:58,010 --> 00:28:04,460

 

630
00:28:04,460 --> 00:28:06,650

 

631
00:28:06,650 --> 00:28:08,200

 

632
00:28:08,200 --> 00:28:12,720

 

633
00:28:12,720 --> 00:28:18,330

 

634
00:28:18,330 --> 00:28:20,300

 

635
00:28:20,300 --> 00:28:22,950

 

636
00:28:22,950 --> 00:28:24,930

 

637
00:28:24,930 --> 00:28:26,970

 

638
00:28:26,970 --> 00:28:28,080

 

639
00:28:28,080 --> 00:28:30,090

 

640
00:28:30,090 --> 00:28:32,610

 

641
00:28:32,610 --> 00:28:35,040

 

642
00:28:35,040 --> 00:28:37,530

 

643
00:28:37,530 --> 00:28:40,320

 

644
00:28:40,320 --> 00:28:42,180

 

645
00:28:42,180 --> 00:28:43,680

 

646
00:28:43,680 --> 00:28:45,500

 

647
00:28:45,500 --> 00:28:47,730

 

648
00:28:47,730 --> 00:28:50,100

 

649
00:28:50,100 --> 00:28:52,950

 

650
00:28:52,950 --> 00:28:57,690

 

651
00:28:57,690 --> 00:29:00,770

 

652
00:29:00,770 --> 00:29:03,330

 

653
00:29:03,330 --> 00:29:05,550

 

654
00:29:05,550 --> 00:29:07,200

 

655
00:29:07,200 --> 00:29:09,960

 

656
00:29:09,960 --> 00:29:11,820

 

657
00:29:11,820 --> 00:29:14,970

 

658
00:29:14,970 --> 00:29:18,780

 

659
00:29:18,780 --> 00:29:19,890

 

660
00:29:19,890 --> 00:29:22,470

 

661
00:29:22,470 --> 00:29:27,390

 

662
00:29:27,390 --> 00:29:29,340

 

663
00:29:29,340 --> 00:29:34,020

 

664
00:29:34,020 --> 00:29:37,110

 

665
00:29:37,110 --> 00:29:38,940

 

666
00:29:38,940 --> 00:29:40,650

 

667
00:29:40,650 --> 00:29:43,500

 

668
00:29:43,500 --> 00:29:46,530

 

669
00:29:46,530 --> 00:29:47,670

 

670
00:29:47,670 --> 00:29:51,360

 

671
00:29:51,360 --> 00:29:54,080

 

672
00:29:54,080 --> 00:29:56,940

 

673
00:29:56,940 --> 00:29:57,540

 

674
00:29:57,540 --> 00:30:01,890

 

675
00:30:01,890 --> 00:30:01,900

 

676
00:30:01,900 --> 00:30:08,750


677
00:30:08,750 --> 00:30:10,640

 

678
00:30:10,640 --> 00:30:12,650

 

679
00:30:12,650 --> 00:30:16,130

 

680
00:30:16,130 --> 00:30:17,600

 

681
00:30:17,600 --> 00:30:20,600

 

682
00:30:20,600 --> 00:30:23,690

 

683
00:30:23,690 --> 00:30:26,090

 

684
00:30:26,090 --> 00:30:29,750

 

685
00:30:29,750 --> 00:30:31,880

 

686
00:30:31,880 --> 00:30:35,750

 

687
00:30:35,750 --> 00:30:37,580

 

688
00:30:37,580 --> 00:30:39,620

 

689
00:30:39,620 --> 00:30:42,220

 

690
00:30:42,220 --> 00:30:44,659

 

691
00:30:44,659 --> 00:30:47,299

 

692
00:30:47,299 --> 00:30:49,310

 

693
00:30:49,310 --> 00:30:51,799

 

694
00:30:51,799 --> 00:30:53,210

 

695
00:30:53,210 --> 00:30:56,120

 

696
00:30:56,120 --> 00:30:58,640

 

697
00:30:58,640 --> 00:31:00,440

 

698
00:31:00,440 --> 00:31:03,620

 

699
00:31:03,620 --> 00:31:05,060

 

700
00:31:05,060 --> 00:31:06,770

 

701
00:31:06,770 --> 00:31:07,880

 

702
00:31:07,880 --> 00:31:11,180

 

703
00:31:11,180 --> 00:31:13,850

 

704
00:31:13,850 --> 00:31:16,100

 

705
00:31:16,100 --> 00:31:17,840

 

706
00:31:17,840 --> 00:31:20,930

 

707
00:31:20,930 --> 00:31:25,400

 

708
00:31:25,400 --> 00:31:27,260

 

709
00:31:27,260 --> 00:31:30,200

 

710
00:31:30,200 --> 00:31:32,240

 

711
00:31:32,240 --> 00:31:35,030

 

712
00:31:35,030 --> 00:31:38,390

 

713
00:31:38,390 --> 00:31:40,820

 

714
00:31:40,820 --> 00:31:42,830

 

715
00:31:42,830 --> 00:31:45,500

 

716
00:31:45,500 --> 00:31:47,090

 

717
00:31:47,090 --> 00:31:49,430

 

718
00:31:49,430 --> 00:31:51,320

 

719
00:31:51,320 --> 00:31:53,120

 

720
00:31:53,120 --> 00:31:56,450

 

721
00:31:56,450 --> 00:31:58,159

 

722
00:31:58,159 --> 00:32:01,990

 

723
00:32:01,990 --> 00:32:04,480

 

724
00:32:04,480 --> 00:32:08,720

 

725
00:32:08,720 --> 00:32:14,080

 

726
00:32:14,080 --> 00:32:17,450

 

727
00:32:17,450 --> 00:32:20,600

 

728
00:32:20,600 --> 00:32:24,620

 

729
00:32:24,620 --> 00:32:26,540

 

730
00:32:26,540 --> 00:32:29,450

 

731
00:32:29,450 --> 00:32:31,640

 

732
00:32:31,640 --> 00:32:33,680

 

733
00:32:33,680 --> 00:32:35,000

 

734
00:32:35,000 --> 00:32:37,040

 

735
00:32:37,040 --> 00:32:41,870

 

736
00:32:41,870 --> 00:32:44,060

 

737
00:32:44,060 --> 00:32:47,030

 

738
00:32:47,030 --> 00:32:49,310

 

739
00:32:49,310 --> 00:32:50,450

 

740
00:32:50,450 --> 00:32:53,450

 

741
00:32:53,450 --> 00:32:56,750

 

742
00:32:56,750 --> 00:32:58,160

 

743
00:32:58,160 --> 00:33:00,550

 

744
00:33:00,550 --> 00:33:03,470

 

745
00:33:03,470 --> 00:33:06,110

 

746
00:33:06,110 --> 00:33:07,970

 

747
00:33:07,970 --> 00:33:09,770

 

748
00:33:09,770 --> 00:33:12,380

 

749
00:33:12,380 --> 00:33:13,580

 

750
00:33:13,580 --> 00:33:15,290

 

751
00:33:15,290 --> 00:33:17,720

 

752
00:33:17,720 --> 00:33:20,120

 

753
00:33:20,120 --> 00:33:22,430

 

754
00:33:22,430 --> 00:33:24,710

 

755
00:33:24,710 --> 00:33:28,000

 

756
00:33:28,000 --> 00:33:30,950

 

757
00:33:30,950 --> 00:33:36,230

 

758
00:33:36,230 --> 00:33:37,610

 

759
00:33:37,610 --> 00:33:39,290

 

760
00:33:39,290 --> 00:33:41,630

 

761
00:33:41,630 --> 00:33:43,340

 

762
00:33:43,340 --> 00:33:46,100

 

763
00:33:46,100 --> 00:33:48,290

 

764
00:33:48,290 --> 00:33:50,240

 

765
00:33:50,240 --> 00:33:53,090

 

766
00:33:53,090 --> 00:34:02,060

 

767
00:34:02,060 --> 00:34:04,470

 

768
00:34:04,470 --> 00:34:06,029

 

769
00:34:06,029 --> 00:34:09,359

 

770
00:34:09,359 --> 00:34:11,480

 

771
00:34:11,480 --> 00:34:15,030

 

772
00:34:15,030 --> 00:34:26,040

 

773
00:34:26,040 --> 00:34:29,220

 

774
00:34:29,220 --> 00:34:31,740

 

775
00:34:31,740 --> 00:34:34,379

 

776
00:34:34,379 --> 00:34:36,720

 

777
00:34:36,720 --> 00:34:39,060

 

778
00:34:39,060 --> 00:34:43,260

 

779
00:34:43,260 --> 00:34:47,580

 

780
00:34:47,580 --> 00:34:50,339

 

781
00:34:50,339 --> 00:34:51,720

 

782
00:34:51,720 --> 00:34:55,379

 

783
00:34:55,379 --> 00:34:58,589

 

784
00:34:58,589 --> 00:35:01,589

 

785
00:35:01,589 --> 00:35:03,240

 

786
00:35:03,240 --> 00:35:04,920

 

787
00:35:04,920 --> 00:35:08,460

 

788
00:35:08,460 --> 00:35:10,170

 

789
00:35:10,170 --> 00:35:12,150

 

790
00:35:12,150 --> 00:35:15,000

 

791
00:35:15,000 --> 00:35:17,210

 

792
00:35:17,210 --> 00:35:27,070

 

793
00:35:27,070 --> 00:35:30,000

 

794
00:35:30,000 --> 00:35:32,080

 

795
00:35:32,080 --> 00:35:33,250

 

796
00:35:33,250 --> 00:35:34,740

 

797
00:35:34,740 --> 00:35:36,940

 

798
00:35:36,940 --> 00:35:39,630

 

799
00:35:39,630 --> 00:35:42,100

 

800
00:35:42,100 --> 00:35:44,770

 

801
00:35:44,770 --> 00:35:47,790

 

802
00:35:47,790 --> 00:35:50,500

 

803
00:35:50,500 --> 00:35:53,020

 

804
00:35:53,020 --> 00:35:55,390

 

805
00:35:55,390 --> 00:35:58,900

 

806
00:35:58,900 --> 00:36:01,980

 

807
00:36:01,980 --> 00:36:04,450

 

808
00:36:04,450 --> 00:36:07,120

 

809
00:36:07,120 --> 00:36:09,070

 

810
00:36:09,070 --> 00:36:11,290

 

811
00:36:11,290 --> 00:36:15,100

 

812
00:36:15,100 --> 00:36:17,590

 

813
00:36:17,590 --> 00:36:19,780

 

814
00:36:19,780 --> 00:36:22,420

 

815
00:36:22,420 --> 00:36:24,580

 

816
00:36:24,580 --> 00:36:26,680

 

817
00:36:26,680 --> 00:36:29,710

 

818
00:36:29,710 --> 00:36:31,780

 

819
00:36:31,780 --> 00:36:36,010

 

820
00:36:36,010 --> 00:36:38,770

 

821
00:36:38,770 --> 00:36:41,860

 

822
00:36:41,860 --> 00:36:43,180

 

823
00:36:43,180 --> 00:36:44,980

 

824
00:36:44,980 --> 00:36:46,720

 

825
00:36:46,720 --> 00:36:49,450

 

826
00:36:49,450 --> 00:36:52,660

 

827
00:36:52,660 --> 00:36:55,990

 

828
00:36:55,990 --> 00:36:58,300

 

829
00:36:58,300 --> 00:37:00,940

 

830
00:37:00,940 --> 00:37:02,920

 

831
00:37:02,920 --> 00:37:05,920

 

832
00:37:05,920 --> 00:37:08,710

 

833
00:37:08,710 --> 00:37:11,470

 

834
00:37:11,470 --> 00:37:14,230

 

835
00:37:14,230 --> 00:37:18,130

 

836
00:37:18,130 --> 00:37:20,830

 

837
00:37:20,830 --> 00:37:22,660

 

838
00:37:22,660 --> 00:37:25,690

 

839
00:37:25,690 --> 00:37:28,990

 

840
00:37:28,990 --> 00:37:30,490

 

841
00:37:30,490 --> 00:37:32,890

 

842
00:37:32,890 --> 00:37:34,690

 

843
00:37:34,690 --> 00:37:36,850

 

844
00:37:36,850 --> 00:37:39,040

 

845
00:37:39,040 --> 00:37:40,359

 

846
00:37:40,359 --> 00:37:44,589

 

847
00:37:44,589 --> 00:37:47,589

 

848
00:37:47,589 --> 00:37:50,859

 

849
00:37:50,859 --> 00:37:53,319

 

850
00:37:53,319 --> 00:37:54,910

 

851
00:37:54,910 --> 00:37:57,069

 

852
00:37:57,069 --> 00:37:59,890

 

853
00:37:59,890 --> 00:38:01,299

 

854
00:38:01,299 --> 00:38:03,789

 

855
00:38:03,789 --> 00:38:06,130

 

856
00:38:06,130 --> 00:38:09,400

 

857
00:38:09,400 --> 00:38:14,559

 

858
00:38:14,559 --> 00:38:17,609

 

859
00:38:17,609 --> 00:38:20,400

 

860
00:38:20,400 --> 00:38:23,170

 

861
00:38:23,170 --> 00:38:24,759

 

862
00:38:24,759 --> 00:38:26,650

 

863
00:38:26,650 --> 00:38:29,200

 

864
00:38:29,200 --> 00:38:33,729

 

865
00:38:33,729 --> 00:38:36,039

 

866
00:38:36,039 --> 00:38:38,499

 

867
00:38:38,499 --> 00:38:42,309

 

868
00:38:42,309 --> 00:38:43,749

 

869
00:38:43,749 --> 00:38:45,969

 

870
00:38:45,969 --> 00:38:48,700

 

871
00:38:48,700 --> 00:38:50,140

 

872
00:38:50,140 --> 00:38:51,940

 

873
00:38:51,940 --> 00:38:56,680

 

874
00:38:56,680 --> 00:38:58,959

 

875
00:38:58,959 --> 00:39:02,349

 

876
00:39:02,349 --> 00:39:04,209

 

877
00:39:04,209 --> 00:39:08,170

 

878
00:39:08,170 --> 00:39:11,349

 

879
00:39:11,349 --> 00:39:12,849

 

880
00:39:12,849 --> 00:39:17,410

 

881
00:39:17,410 --> 00:39:20,170

 

882
00:39:20,170 --> 00:39:21,819

 

883
00:39:21,819 --> 00:39:23,739

 

884
00:39:23,739 --> 00:39:26,170

 

885
00:39:26,170 --> 00:39:28,809

 

886
00:39:28,809 --> 00:39:31,450

 

887
00:39:31,450 --> 00:39:34,059

 

888
00:39:34,059 --> 00:39:36,130

 

889
00:39:36,130 --> 00:39:45,339

 

890
00:39:45,339 --> 00:39:50,299

 

891
00:39:50,299 --> 00:39:53,929

 

892
00:39:53,929 --> 00:39:55,370

 

893
00:39:55,370 --> 00:39:55,939

 

894
00:39:55,939 --> 00:39:57,859

 

895
00:39:57,859 --> 00:39:57,869

 

896
00:39:57,869 --> 00:40:08,980


897
00:40:08,980 --> 00:40:12,430

 

898
00:40:12,430 --> 00:40:14,859

 

899
00:40:14,859 --> 00:40:17,589

 

900
00:40:17,589 --> 00:40:19,150

 

901
00:40:19,150 --> 00:40:29,950

 

902
00:40:29,950 --> 00:40:31,510

 

903
00:40:31,510 --> 00:40:33,570

 

904
00:40:33,570 --> 00:40:37,270

 

905
00:40:37,270 --> 00:40:49,730

 

906
00:40:49,730 --> 00:40:52,430

 

907
00:40:52,430 --> 00:40:58,520

 

908
00:40:58,520 --> 00:41:01,460

 

909
00:41:01,460 --> 00:41:04,940

 

910
00:41:04,940 --> 00:41:08,839

 

911
00:41:08,839 --> 00:41:10,940

 

912
00:41:10,940 --> 00:41:13,430

 

913
00:41:13,430 --> 00:41:15,980

 

914
00:41:15,980 --> 00:41:18,770

 

915
00:41:18,770 --> 00:41:20,599

 

916
00:41:20,599 --> 00:41:22,730

 

917
00:41:22,730 --> 00:41:24,410

 

918
00:41:24,410 --> 00:41:26,570

 

919
00:41:26,570 --> 00:41:28,190

 

920
00:41:28,190 --> 00:41:29,450

 

921
00:41:29,450 --> 00:41:31,730

 

922
00:41:31,730 --> 00:41:37,520

 

923
00:41:37,520 --> 00:41:43,070

 

924
00:41:43,070 --> 00:41:45,589

 

925
00:41:45,589 --> 00:41:47,900

 

926
00:41:47,900 --> 00:41:49,339

 

927
00:41:49,339 --> 00:41:53,870

 

928
00:41:53,870 --> 00:41:55,640

 

929
00:41:55,640 --> 00:41:58,940

 

930
00:41:58,940 --> 00:42:01,070

 

931
00:42:01,070 --> 00:42:03,170

 

932
00:42:03,170 --> 00:42:05,990

 

933
00:42:05,990 --> 00:42:09,260

 

934
00:42:09,260 --> 00:42:11,230

 

935
00:42:11,230 --> 00:42:13,940

 

936
00:42:13,940 --> 00:42:17,060

 

937
00:42:17,060 --> 00:42:19,550

 

938
00:42:19,550 --> 00:42:22,250

 

939
00:42:22,250 --> 00:42:25,640

 

940
00:42:25,640 --> 00:42:27,200

 

941
00:42:27,200 --> 00:42:28,480

 

942
00:42:28,480 --> 00:42:30,320

 

943
00:42:30,320 --> 00:42:33,530

 

944
00:42:33,530 --> 00:42:36,079

 

945
00:42:36,079 --> 00:42:38,870

 

946
00:42:38,870 --> 00:42:40,730

 

947
00:42:40,730 --> 00:42:44,180

 

948
00:42:44,180 --> 00:42:46,370

 

949
00:42:46,370 --> 00:42:49,849

 

950
00:42:49,849 --> 00:42:51,230

 

951
00:42:51,230 --> 00:42:54,050

 

952
00:42:54,050 --> 00:43:01,550

 

953
00:43:01,550 --> 00:43:05,570

 

954
00:43:05,570 --> 00:43:05,580

 

955
00:43:05,580 --> 00:43:07,650


956
00:43:07,650 --> 00:43:09,809

 

957
00:43:09,809 --> 00:43:18,170

 

958
00:43:18,170 --> 00:43:29,720

 

959
00:43:29,720 --> 00:43:32,660

 

960
00:43:32,660 --> 00:43:36,290

 

961
00:43:36,290 --> 00:43:41,480

 

962
00:43:41,480 --> 00:43:44,510

 

963
00:43:44,510 --> 00:43:59,950

 

964
00:43:59,950 --> 00:44:04,520

 

965
00:44:04,520 --> 00:44:05,570

 

966
00:44:05,570 --> 00:44:07,790

 

967
00:44:07,790 --> 00:44:10,310

 

968
00:44:10,310 --> 00:44:12,050

 

969
00:44:12,050 --> 00:44:14,630

 

970
00:44:14,630 --> 00:44:16,850

 

971
00:44:16,850 --> 00:44:18,560

 

972
00:44:18,560 --> 00:44:18,570

 

973
00:44:18,570 --> 00:44:21,320


974
00:44:21,320 --> 00:44:23,480

 

975
00:44:23,480 --> 00:44:28,460

 

976
00:44:28,460 --> 00:44:31,850

 

977
00:44:31,850 --> 00:44:33,710

 

978
00:44:33,710 --> 00:44:36,740

 

979
00:44:36,740 --> 00:44:39,920

 

980
00:44:39,920 --> 00:44:43,410

 

981
00:44:43,410 --> 00:44:43,420

 

982
00:44:43,420 --> 00:44:46,390


983
00:44:46,390 --> 00:44:50,620

 

984
00:44:50,620 --> 00:44:53,230

 

985
00:44:53,230 --> 00:45:05,300

 

986
00:45:05,300 --> 00:45:09,800

 

987
00:45:09,800 --> 00:45:11,630

 

988
00:45:11,630 --> 00:45:16,610

 

989
00:45:16,610 --> 00:45:19,220

 

990
00:45:19,220 --> 00:45:21,980

 

991
00:45:21,980 --> 00:45:23,720

 

992
00:45:23,720 --> 00:45:27,710

 

993
00:45:27,710 --> 00:45:29,060

 

994
00:45:29,060 --> 00:45:33,200

 

995
00:45:33,200 --> 00:45:36,290

 

996
00:45:36,290 --> 00:45:38,360

 

997
00:45:38,360 --> 00:45:42,770

 

998
00:45:42,770 --> 00:45:46,580

 

999
00:45:46,580 --> 00:45:55,390

 

1000
00:45:55,390 --> 00:45:58,049

 

1001
00:45:58,049 --> 00:46:03,519

 

1002
00:46:03,519 --> 00:46:05,079

 

1003
00:46:05,079 --> 00:46:08,319

 

1004
00:46:08,319 --> 00:46:11,009

 

1005
00:46:11,009 --> 00:46:13,329

 

1006
00:46:13,329 --> 00:46:16,269

 

1007
00:46:16,269 --> 00:46:18,849

 

1008
00:46:18,849 --> 00:46:21,430

 

1009
00:46:21,430 --> 00:46:24,279

 

1010
00:46:24,279 --> 00:46:27,519

 

1011
00:46:27,519 --> 00:46:30,819

 

1012
00:46:30,819 --> 00:46:33,999

 

1013
00:46:33,999 --> 00:46:38,739

 

1014
00:46:38,739 --> 00:46:43,420

 

1015
00:46:43,420 --> 00:46:45,630

 

1016
00:46:45,630 --> 00:46:48,009

 

1017
00:46:48,009 --> 00:46:51,819

 

1018
00:46:51,819 --> 00:46:56,529

 

1019
00:46:56,529 --> 00:46:58,599

 

1020
00:46:58,599 --> 00:47:00,970

 

1021
00:47:00,970 --> 00:47:04,029

 

1022
00:47:04,029 --> 00:47:08,920

 

1023
00:47:08,920 --> 00:47:11,019

 

1024
00:47:11,019 --> 00:47:13,089

 

1025
00:47:13,089 --> 00:47:13,779

 

1026
00:47:13,779 --> 00:47:17,259

 

1027
00:47:17,259 --> 00:47:19,239

 

1028
00:47:19,239 --> 00:47:21,789

 

1029
00:47:21,789 --> 00:47:21,799

 

1030
00:47:21,799 --> 00:47:31,030


1031
00:47:31,030 --> 00:47:32,680

 

1032
00:47:32,680 --> 00:47:41,420

 

1033
00:47:41,420 --> 00:47:43,730

 

1034
00:47:43,730 --> 00:47:46,160

 

1035
00:47:46,160 --> 00:47:48,230

 

1036
00:47:48,230 --> 00:47:52,010

 

1037
00:47:52,010 --> 00:47:53,570

 

1038
00:47:53,570 --> 00:47:55,550

 

1039
00:47:55,550 --> 00:47:57,050

 

1040
00:47:57,050 --> 00:48:00,860

 

1041
00:48:00,860 --> 00:48:03,230

 

1042
00:48:03,230 --> 00:48:06,710

 

1043
00:48:06,710 --> 00:48:08,900

 

1044
00:48:08,900 --> 00:48:10,240

 

1045
00:48:10,240 --> 00:48:12,290

 

1046
00:48:12,290 --> 00:48:14,360

 

1047
00:48:14,360 --> 00:48:16,790

 

1048
00:48:16,790 --> 00:48:18,380

 

1049
00:48:18,380 --> 00:48:23,180

 

1050
00:48:23,180 --> 00:48:25,310

 

1051
00:48:25,310 --> 00:48:28,490

 

1052
00:48:28,490 --> 00:48:31,040

 

1053
00:48:31,040 --> 00:48:34,400

 

1054
00:48:34,400 --> 00:48:36,770

 

1055
00:48:36,770 --> 00:48:47,819

 

1056
00:48:47,819 --> 00:48:50,019

 

1057
00:48:50,019 --> 00:48:51,730

 

1058
00:48:51,730 --> 00:48:58,420

 

1059
00:48:58,420 --> 00:49:00,640

 

1060
00:49:00,640 --> 00:49:04,410

 

1061
00:49:04,410 --> 00:49:07,059

 

1062
00:49:07,059 --> 00:49:09,579

 

1063
00:49:09,579 --> 00:49:11,259

 

1064
00:49:11,259 --> 00:49:13,299

 

1065
00:49:13,299 --> 00:49:17,710

 

1066
00:49:17,710 --> 00:49:19,299

 

1067
00:49:19,299 --> 00:49:21,700

 

1068
00:49:21,700 --> 00:49:24,730

 

1069
00:49:24,730 --> 00:49:28,029

 

1070
00:49:28,029 --> 00:49:30,190

 

1071
00:49:30,190 --> 00:49:37,610

 

1072
00:49:37,610 --> 00:49:37,620

 

1073
00:49:37,620 --> 00:49:39,850


1074
00:49:39,850 --> 00:49:52,390

 

1075
00:49:52,390 --> 00:49:56,049

 

1076
00:49:56,049 --> 00:49:57,640

 

1077
00:49:57,640 --> 00:50:00,700

 

1078
00:50:00,700 --> 00:50:02,289

 

1079
00:50:02,289 --> 00:50:04,870

 

1080
00:50:04,870 --> 00:50:06,579

 

1081
00:50:06,579 --> 00:50:10,269

 

1082
00:50:10,269 --> 00:50:14,289

 

1083
00:50:14,289 --> 00:50:17,069

 

1084
00:50:17,069 --> 00:50:20,140

 

1085
00:50:20,140 --> 00:50:22,319

 

1086
00:50:22,319 --> 00:50:25,960

 

1087
00:50:25,960 --> 00:50:28,239

 

1088
00:50:28,239 --> 00:50:31,049

 

1089
00:50:31,049 --> 00:50:32,890

 

1090
00:50:32,890 --> 00:50:32,900

 

1091
00:50:32,900 --> 00:50:33,519


1092
00:50:33,519 --> 00:50:35,920

 

1093
00:50:35,920 --> 00:50:46,990

 

1094
00:50:46,990 --> 00:50:55,810

 

1095
00:50:55,810 --> 00:50:57,010

 

1096
00:50:57,010 --> 00:51:00,940

 

1097
00:51:00,940 --> 00:51:05,680

 

1098
00:51:05,680 --> 00:51:08,140

 

1099
00:51:08,140 --> 00:51:12,630

 

1100
00:51:12,630 --> 00:51:15,100

 

1101
00:51:15,100 --> 00:51:23,630

 

1102
00:51:23,630 --> 00:51:28,840

 

1103
00:51:28,840 --> 00:51:44,740

 

1104
00:51:44,740 --> 00:51:48,220

 

1105
00:51:48,220 --> 00:51:49,390

 

1106
00:51:49,390 --> 00:51:53,710

 

1107
00:51:53,710 --> 00:51:55,349

 

1108
00:51:55,349 --> 00:51:59,770

 

1109
00:51:59,770 --> 00:52:02,740

 

1110
00:52:02,740 --> 00:52:05,530

 

1111
00:52:05,530 --> 00:52:09,160

 

1112
00:52:09,160 --> 00:52:11,800

 

1113
00:52:11,800 --> 00:52:14,530

 

1114
00:52:14,530 --> 00:52:17,910

 

1115
00:52:17,910 --> 00:52:20,710

 

1116
00:52:20,710 --> 00:52:30,040

 

1117
00:52:30,040 --> 00:52:32,109

 

1118
00:52:32,109 --> 00:52:38,440

 

1119
00:52:38,440 --> 00:52:41,440

 

1120
00:52:41,440 --> 00:52:44,160

 

1121
00:52:44,160 --> 00:52:47,200

 

1122
00:52:47,200 --> 00:52:49,180

 

1123
00:52:49,180 --> 00:52:51,400

 

1124
00:52:51,400 --> 00:52:54,240

 

1125
00:52:54,240 --> 00:52:56,560

 

1126
00:52:56,560 --> 00:53:00,700

 

1127
00:53:00,700 --> 00:53:02,260

 

1128
00:53:02,260 --> 00:53:05,010

 

1129
00:53:05,010 --> 00:53:08,440

 

1130
00:53:08,440 --> 00:53:10,270

 

1131
00:53:10,270 --> 00:53:12,640

 

1132
00:53:12,640 --> 00:53:14,830

 

1133
00:53:14,830 --> 00:53:17,080

 

1134
00:53:17,080 --> 00:53:18,730

 

1135
00:53:18,730 --> 00:53:22,210

 

1136
00:53:22,210 --> 00:53:23,980

 

1137
00:53:23,980 --> 00:53:26,650

 

1138
00:53:26,650 --> 00:53:29,170

 

1139
00:53:29,170 --> 00:53:30,490

 

1140
00:53:30,490 --> 00:53:30,500

 

1141
00:53:30,500 --> 00:53:31,470


1142
00:53:31,470 --> 00:53:56,820

 

1143
00:53:56,820 --> 00:53:59,430

 

1144
00:53:59,430 --> 00:54:09,410

 

1145
00:54:09,410 --> 00:54:09,420

 

1146
00:54:09,420 --> 00:54:11,710


1147
00:54:11,710 --> 00:54:17,420

 

1148
00:54:17,420 --> 00:54:17,430

 

1149
00:54:17,430 --> 00:54:21,550


1150
00:54:21,550 --> 00:54:27,170

 

1151
00:54:27,170 --> 00:54:28,510

 

1152
00:54:28,510 --> 00:54:32,390

 

1153
00:54:32,390 --> 00:54:34,280

 

1154
00:54:34,280 --> 00:54:37,790

 

1155
00:54:37,790 --> 00:54:39,380

 

1156
00:54:39,380 --> 00:54:41,480

 

1157
00:54:41,480 --> 00:54:43,790

 

1158
00:54:43,790 --> 00:54:47,180

 

1159
00:54:47,180 --> 00:54:51,260

 

1160
00:54:51,260 --> 00:54:52,760

 

1161
00:54:52,760 --> 00:54:55,220

 

1162
00:54:55,220 --> 00:54:58,670

 

1163
00:54:58,670 --> 00:55:00,920

 

1164
00:55:00,920 --> 00:55:03,860

 

1165
00:55:03,860 --> 00:55:05,990

 

1166
00:55:05,990 --> 00:55:14,000

 

1167
00:55:14,000 --> 00:55:15,670

 

1168
00:55:15,670 --> 00:55:19,580

 

1169
00:55:19,580 --> 00:55:22,070

 

1170
00:55:22,070 --> 00:55:25,130

 

1171
00:55:25,130 --> 00:55:27,290

 

1172
00:55:27,290 --> 00:55:29,660

 

1173
00:55:29,660 --> 00:55:34,460

 

1174
00:55:34,460 --> 00:55:35,960

 

1175
00:55:35,960 --> 00:55:36,980

 

1176
00:55:36,980 --> 00:55:39,350

 

1177
00:55:39,350 --> 00:55:41,000

 

1178
00:55:41,000 --> 00:55:43,160

 

1179
00:55:43,160 --> 00:55:45,710

 

1180
00:55:45,710 --> 00:55:48,800

 

1181
00:55:48,800 --> 00:55:52,450

 

1182
00:55:52,450 --> 00:55:54,140

 

1183
00:55:54,140 --> 00:55:55,280

 

1184
00:55:55,280 --> 00:55:57,200

 

1185
00:55:57,200 --> 00:56:01,130

 

1186
00:56:01,130 --> 00:56:04,010

 

1187
00:56:04,010 --> 00:56:08,240

 

1188
00:56:08,240 --> 00:56:11,050

 

1189
00:56:11,050 --> 00:56:15,680

 

1190
00:56:15,680 --> 00:56:19,100

 

1191
00:56:19,100 --> 00:56:21,410

 

1192
00:56:21,410 --> 00:56:23,270

 

1193
00:56:23,270 --> 00:56:25,580

 

1194
00:56:25,580 --> 00:56:27,950

 

1195
00:56:27,950 --> 00:56:29,510

 

1196
00:56:29,510 --> 00:56:32,450

 

1197
00:56:32,450 --> 00:56:33,650

 

1198
00:56:33,650 --> 00:56:36,740

 

1199
00:56:36,740 --> 00:56:38,840

 

1200
00:56:38,840 --> 00:56:43,580

 

1201
00:56:43,580 --> 00:56:45,380

 

1202
00:56:45,380 --> 00:56:49,550

 

1203
00:56:49,550 --> 00:56:52,460

 

1204
00:56:52,460 --> 00:56:55,730

 

1205
00:56:55,730 --> 00:56:58,850

 

1206
00:56:58,850 --> 00:57:00,410

 

1207
00:57:00,410 --> 00:57:03,470

 

1208
00:57:03,470 --> 00:57:06,830

 

1209
00:57:06,830 --> 00:57:08,900

 

1210
00:57:08,900 --> 00:57:11,390

 

1211
00:57:11,390 --> 00:57:15,170

 

1212
00:57:15,170 --> 00:57:18,500

 

1213
00:57:18,500 --> 00:57:21,230

 

1214
00:57:21,230 --> 00:57:23,950

 

1215
00:57:23,950 --> 00:57:27,050

 

1216
00:57:27,050 --> 00:57:30,110

 

1217
00:57:30,110 --> 00:57:33,170

 

1218
00:57:33,170 --> 00:57:36,830

 

1219
00:57:36,830 --> 00:57:40,280

 

1220
00:57:40,280 --> 00:57:47,420

 

1221
00:57:47,420 --> 00:57:50,000

 

1222
00:57:50,000 --> 00:57:52,660

 

1223
00:57:52,660 --> 00:57:55,340

 

1224
00:57:55,340 --> 00:57:57,590

 

1225
00:57:57,590 --> 00:57:59,630

 

1226
00:57:59,630 --> 00:58:02,090

 

1227
00:58:02,090 --> 00:58:04,790

 

1228
00:58:04,790 --> 00:58:07,160

 

1229
00:58:07,160 --> 00:58:09,440

 

1230
00:58:09,440 --> 00:58:13,010

 

1231
00:58:13,010 --> 00:58:14,720

 

1232
00:58:14,720 --> 00:58:17,300

 

1233
00:58:17,300 --> 00:58:19,610

 

1234
00:58:19,610 --> 00:58:22,250

 

1235
00:58:22,250 --> 00:58:25,370

 

1236
00:58:25,370 --> 00:58:27,050

 

1237
00:58:27,050 --> 00:58:27,800

 

1238
00:58:27,800 --> 00:58:30,470

 

1239
00:58:30,470 --> 00:58:32,420

 

1240
00:58:32,420 --> 00:58:35,180

 

1241
00:58:35,180 --> 00:58:37,280

 

1242
00:58:37,280 --> 00:58:41,480

 

1243
00:58:41,480 --> 00:58:44,420

 

1244
00:58:44,420 --> 00:58:46,110

 

1245
00:58:46,110 --> 00:58:49,880

 

1246
00:58:49,880 --> 00:58:52,800

 

1247
00:58:52,800 --> 00:58:54,540

 

1248
00:58:54,540 --> 00:58:57,650

 

1249
00:58:57,650 --> 00:59:01,080

 

1250
00:59:01,080 --> 00:59:03,390

 

1251
00:59:03,390 --> 00:59:07,530

 

1252
00:59:07,530 --> 00:59:09,750

 

1253
00:59:09,750 --> 00:59:12,150

 

1254
00:59:12,150 --> 00:59:14,700

 

1255
00:59:14,700 --> 00:59:16,650

 

1256
00:59:16,650 --> 00:59:18,930

 

1257
00:59:18,930 --> 00:59:21,480

 

1258
00:59:21,480 --> 00:59:24,930

 

1259
00:59:24,930 --> 00:59:31,710

 

1260
00:59:31,710 --> 00:59:34,380

 

1261
00:59:34,380 --> 00:59:36,690

 

1262
00:59:36,690 --> 00:59:38,610

 

1263
00:59:38,610 --> 00:59:41,820

 

1264
00:59:41,820 --> 00:59:42,960

 

1265
00:59:42,960 --> 00:59:44,520

 

1266
00:59:44,520 --> 00:59:46,620

 

1267
00:59:46,620 --> 00:59:48,510

 

1268
00:59:48,510 --> 00:59:50,370

 

1269
00:59:50,370 --> 00:59:52,260

 

1270
00:59:52,260 --> 00:59:54,420

 

1271
00:59:54,420 --> 00:59:58,470

 

1272
00:59:58,470 --> 01:00:02,150

 

1273
01:00:02,150 --> 01:00:05,400

 

1274
01:00:05,400 --> 01:00:15,540

 

1275
01:00:15,540 --> 01:00:23,460

 

1276
01:00:23,460 --> 01:00:26,250

 

1277
01:00:26,250 --> 01:00:30,120

 

1278
01:00:30,120 --> 01:00:37,860

 

1279
01:00:37,860 --> 01:00:45,750

 

1280
01:00:45,750 --> 01:00:50,310

 

1281
01:00:50,310 --> 01:00:52,170

 

1282
01:00:52,170 --> 01:00:54,270

 

1283
01:00:54,270 --> 01:00:59,460

 

1284
01:00:59,460 --> 01:01:02,730

 

1285
01:01:02,730 --> 01:01:04,500

 

1286
01:01:04,500 --> 01:01:06,330

 

1287
01:01:06,330 --> 01:01:08,280

 

1288
01:01:08,280 --> 01:01:10,020

 

1289
01:01:10,020 --> 01:01:13,170

 

1290
01:01:13,170 --> 01:01:16,320

 

1291
01:01:16,320 --> 01:01:21,270

 

1292
01:01:21,270 --> 01:01:24,450

 

1293
01:01:24,450 --> 01:01:27,510

 

1294
01:01:27,510 --> 01:01:30,600

 

1295
01:01:30,600 --> 01:01:31,920

 

1296
01:01:31,920 --> 01:01:33,840

 

1297
01:01:33,840 --> 01:01:36,510

 

1298
01:01:36,510 --> 01:01:40,140

 

1299
01:01:40,140 --> 01:01:44,460

 

1300
01:01:44,460 --> 01:01:47,550

 

1301
01:01:47,550 --> 01:01:49,650

 

1302
01:01:49,650 --> 01:01:53,760

 

1303
01:01:53,760 --> 01:01:55,350

 

1304
01:01:55,350 --> 01:02:01,230

 

1305
01:02:01,230 --> 01:02:03,930

 

1306
01:02:03,930 --> 01:02:06,560

 

1307
01:02:06,560 --> 01:02:09,120

 

1308
01:02:09,120 --> 01:02:11,760

 

1309
01:02:11,760 --> 01:02:15,420

 

1310
01:02:15,420 --> 01:02:17,040

 

1311
01:02:17,040 --> 01:02:19,710

 

1312
01:02:19,710 --> 01:02:21,990

 

1313
01:02:21,990 --> 01:02:24,750

 

1314
01:02:24,750 --> 01:02:27,120

 

1315
01:02:27,120 --> 01:02:27,130

 

1316
01:02:27,130 --> 01:02:28,170


1317
01:02:28,170 --> 01:02:30,750

 

1318
01:02:30,750 --> 01:02:36,329

 

1319
01:02:36,329 --> 01:02:37,650

 

1320
01:02:37,650 --> 01:02:43,470

 

1321
01:02:43,470 --> 01:02:46,829

 

1322
01:02:46,829 --> 01:02:53,130

 

1323
01:02:53,130 --> 01:02:55,650

 

1324
01:02:55,650 --> 01:02:57,960

 

1325
01:02:57,960 --> 01:03:10,380

 

1326
01:03:10,380 --> 01:03:12,900

 

1327
01:03:12,900 --> 01:03:14,279

 

1328
01:03:14,279 --> 01:03:17,569

 

1329
01:03:17,569 --> 01:03:21,059

 

1330
01:03:21,059 --> 01:03:23,160

 

1331
01:03:23,160 --> 01:03:25,829

 

1332
01:03:25,829 --> 01:03:25,839

 

1333
01:03:25,839 --> 01:03:26,370


1334
01:03:26,370 --> 01:03:30,359

 

1335
01:03:30,359 --> 01:03:32,250

 

1336
01:03:32,250 --> 01:03:40,190

 

1337
01:03:40,190 --> 01:03:42,420

 

1338
01:03:42,420 --> 01:03:44,279

 

1339
01:03:44,279 --> 01:03:47,670

 

1340
01:03:47,670 --> 01:03:50,220

 

1341
01:03:50,220 --> 01:03:54,059

 

1342
01:03:54,059 --> 01:03:57,809

 

1343
01:03:57,809 --> 01:03:59,130

 

1344
01:03:59,130 --> 01:04:00,690

 

1345
01:04:00,690 --> 01:04:02,069

 

1346
01:04:02,069 --> 01:04:09,029

 

1347
01:04:09,029 --> 01:04:11,490

 

1348
01:04:11,490 --> 01:04:13,710

 

1349
01:04:13,710 --> 01:04:15,330

 

1350
01:04:15,330 --> 01:04:17,580

 

1351
01:04:17,580 --> 01:04:20,150

 

1352
01:04:20,150 --> 01:04:24,359

 

1353
01:04:24,359 --> 01:04:29,130

 

1354
01:04:29,130 --> 01:04:31,380

 

1355
01:04:31,380 --> 01:04:33,810

 

1356
01:04:33,810 --> 01:04:37,260

 

1357
01:04:37,260 --> 01:04:38,730

 

1358
01:04:38,730 --> 01:04:41,790

 

1359
01:04:41,790 --> 01:04:42,930

 

1360
01:04:42,930 --> 01:04:44,849

 

1361
01:04:44,849 --> 01:04:49,200

 

1362
01:04:49,200 --> 01:04:50,670

 

1363
01:04:50,670 --> 01:04:50,680

 

1364
01:04:50,680 --> 01:05:01,510


1365
01:05:01,510 --> 01:05:04,569

 

1366
01:05:04,569 --> 01:05:09,609

 

1367
01:05:09,609 --> 01:05:13,390

 

1368
01:05:13,390 --> 01:05:20,999

 

1369
01:05:20,999 --> 01:05:24,999

 

1370
01:05:24,999 --> 01:05:30,730

 

1371
01:05:30,730 --> 01:05:32,589

 

1372
01:05:32,589 --> 01:05:36,940

 

1373
01:05:36,940 --> 01:05:39,009

 

1374
01:05:39,009 --> 01:05:40,599

 

1375
01:05:40,599 --> 01:05:42,970

 

1376
01:05:42,970 --> 01:05:45,460

 

1377
01:05:45,460 --> 01:05:47,470

 

1378
01:05:47,470 --> 01:05:54,730

 

1379
01:05:54,730 --> 01:06:02,230

 

1380
01:06:02,230 --> 01:06:07,390

 

1381
01:06:07,390 --> 01:06:09,130

 

1382
01:06:09,130 --> 01:06:12,700

 

1383
01:06:12,700 --> 01:06:17,020

 

1384
01:06:17,020 --> 01:06:19,210

 

1385
01:06:19,210 --> 01:06:22,180

 

1386
01:06:22,180 --> 01:06:40,180

 

1387
01:06:40,180 --> 01:06:42,490

 

1388
01:06:42,490 --> 01:06:43,990

 

1389
01:06:43,990 --> 01:06:47,260

 

1390
01:06:47,260 --> 01:06:49,359

 

1391
01:06:49,359 --> 01:06:53,529

 

1392
01:06:53,529 --> 01:06:55,210

 

1393
01:06:55,210 --> 01:06:58,630

 

1394
01:06:58,630 --> 01:07:01,120

 

1395
01:07:01,120 --> 01:07:03,609

 

1396
01:07:03,609 --> 01:07:05,470

 

1397
01:07:05,470 --> 01:07:06,160

 

1398
01:07:06,160 --> 01:07:08,579

 

1399
01:07:08,579 --> 01:07:12,270

 

1400
01:07:12,270 --> 01:07:15,430

 

1401
01:07:15,430 --> 01:07:18,270

 

1402
01:07:18,270 --> 01:07:30,530

 

1403
01:07:30,530 --> 01:07:32,270

 

1404
01:07:32,270 --> 01:07:34,490

 

1405
01:07:34,490 --> 01:07:36,770

 

1406
01:07:36,770 --> 01:07:39,230

 

1407
01:07:39,230 --> 01:07:43,300

 

1408
01:07:43,300 --> 01:07:46,250

 

1409
01:07:46,250 --> 01:07:49,040

 

1410
01:07:49,040 --> 01:07:55,990

 

1411
01:07:55,990 --> 01:07:56,000

 

1412
01:07:56,000 --> 01:08:05,230


1413
01:08:05,230 --> 01:08:08,960

 

1414
01:08:08,960 --> 01:08:11,540

 

1415
01:08:11,540 --> 01:08:15,560

 

1416
01:08:15,560 --> 01:08:17,960

 

1417
01:08:17,960 --> 01:08:21,110

 

1418
01:08:21,110 --> 01:08:23,510

 

1419
01:08:23,510 --> 01:08:26,240

 

1420
01:08:26,240 --> 01:08:29,030

 

1421
01:08:29,030 --> 01:08:31,640

 

1422
01:08:31,640 --> 01:08:32,960

 

1423
01:08:32,960 --> 01:08:34,490

 

1424
01:08:34,490 --> 01:08:36,200

 

1425
01:08:36,200 --> 01:08:39,079

 

1426
01:08:39,079 --> 01:08:41,720

 

1427
01:08:41,720 --> 01:08:43,970

 

1428
01:08:43,970 --> 01:08:45,560

 

1429
01:08:45,560 --> 01:08:49,280

 

1430
01:08:49,280 --> 01:08:52,480

 

1431
01:08:52,480 --> 01:08:55,550

 

1432
01:08:55,550 --> 01:08:57,200

 

1433
01:08:57,200 --> 01:09:00,680

 

1434
01:09:00,680 --> 01:09:03,620

 

1435
01:09:03,620 --> 01:09:05,630

 

1436
01:09:05,630 --> 01:09:07,250

 

1437
01:09:07,250 --> 01:09:08,960

 

1438
01:09:08,960 --> 01:09:12,349

 

1439
01:09:12,349 --> 01:09:15,050

 

1440
01:09:15,050 --> 01:09:17,570

 

1441
01:09:17,570 --> 01:09:20,390

 

1442
01:09:20,390 --> 01:09:23,360

 

1443
01:09:23,360 --> 01:09:25,610

 

1444
01:09:25,610 --> 01:09:27,680

 

1445
01:09:27,680 --> 01:09:30,160

 

1446
01:09:30,160 --> 01:09:33,560

 

1447
01:09:33,560 --> 01:09:35,900

 

1448
01:09:35,900 --> 01:09:38,210

 

1449
01:09:38,210 --> 01:09:42,250

 

1450
01:09:42,250 --> 01:09:45,170

 

1451
01:09:45,170 --> 01:09:47,770

 

1452
01:09:47,770 --> 01:09:50,750

 

1453
01:09:50,750 --> 01:09:53,329

 

1454
01:09:53,329 --> 01:09:56,020

 

1455
01:09:56,020 --> 01:09:58,130

 

1456
01:09:58,130 --> 01:10:00,740

 

1457
01:10:00,740 --> 01:10:03,800

 

1458
01:10:03,800 --> 01:10:06,920

 

1459
01:10:06,920 --> 01:10:09,770

 

1460
01:10:09,770 --> 01:10:11,600

 

1461
01:10:11,600 --> 01:10:14,120

 

1462
01:10:14,120 --> 01:10:17,540

 

1463
01:10:17,540 --> 01:10:20,600

 

1464
01:10:20,600 --> 01:10:22,459

 

1465
01:10:22,459 --> 01:10:25,490

 

1466
01:10:25,490 --> 01:10:28,850

 

1467
01:10:28,850 --> 01:10:31,580

 

1468
01:10:31,580 --> 01:10:34,010

 

1469
01:10:34,010 --> 01:10:39,010

 

1470
01:10:39,010 --> 01:10:47,330

 

1471
01:10:47,330 --> 01:10:49,250

 

1472
01:10:49,250 --> 01:10:53,500

 

1473
01:10:53,500 --> 01:10:58,310

 

1474
01:10:58,310 --> 01:10:59,950

 

1475
01:10:59,950 --> 01:11:01,910

 

1476
01:11:01,910 --> 01:11:03,410

 

1477
01:11:03,410 --> 01:11:07,669

 

1478
01:11:07,669 --> 01:11:10,549

 

1479
01:11:10,549 --> 01:11:12,500

 

1480
01:11:12,500 --> 01:11:14,180

 

1481
01:11:14,180 --> 01:11:16,669

 

1482
01:11:16,669 --> 01:11:21,250

 

1483
01:11:21,250 --> 01:11:24,410

 

1484
01:11:24,410 --> 01:11:29,319

 

1485
01:11:29,319 --> 01:11:32,180

 

1486
01:11:32,180 --> 01:11:34,870

 

1487
01:11:34,870 --> 01:11:34,880

 

1488
01:11:34,880 --> 01:11:40,100


1489
01:11:40,100 --> 01:11:44,660

 

1490
01:11:44,660 --> 01:11:47,280

 

1491
01:11:47,280 --> 01:11:50,130

 

1492
01:11:50,130 --> 01:11:52,020

 

1493
01:11:52,020 --> 01:11:54,600

 

1494
01:11:54,600 --> 01:11:55,950

 

1495
01:11:55,950 --> 01:11:57,240

 

1496
01:11:57,240 --> 01:12:00,470

 

1497
01:12:00,470 --> 01:12:03,600

 

1498
01:12:03,600 --> 01:12:05,550

 

1499
01:12:05,550 --> 01:12:10,220

 

1500
01:12:10,220 --> 01:12:13,350

 

1501
01:12:13,350 --> 01:12:16,620

 

1502
01:12:16,620 --> 01:12:18,270

 

1503
01:12:18,270 --> 01:12:20,600

 

1504
01:12:20,600 --> 01:12:24,630

 

1505
01:12:24,630 --> 01:12:27,360

 

1506
01:12:27,360 --> 01:12:29,460

 

1507
01:12:29,460 --> 01:12:31,620

 

1508
01:12:31,620 --> 01:12:33,620

 

1509
01:12:33,620 --> 01:12:36,690

 

1510
01:12:36,690 --> 01:12:39,120

 

1511
01:12:39,120 --> 01:12:42,780

 

1512
01:12:42,780 --> 01:12:44,510

 

1513
01:12:44,510 --> 01:12:46,890

 

1514
01:12:46,890 --> 01:12:49,830

 

1515
01:12:49,830 --> 01:12:49,840

 

1516
01:12:49,840 --> 01:12:50,540


1517
01:12:50,540 --> 01:12:54,900

 

1518
01:12:54,900 --> 01:12:56,700

 

1519
01:12:56,700 --> 01:12:58,680

 

1520
01:12:58,680 --> 01:13:02,130

 

1521
01:13:02,130 --> 01:13:03,810

 

1522
01:13:03,810 --> 01:13:06,390

 

1523
01:13:06,390 --> 01:13:09,030

 

1524
01:13:09,030 --> 01:13:13,230

 

1525
01:13:13,230 --> 01:13:15,450

 

1526
01:13:15,450 --> 01:13:18,200

 

1527
01:13:18,200 --> 01:13:20,520

 

1528
01:13:20,520 --> 01:13:23,130

 

1529
01:13:23,130 --> 01:13:25,920

 

1530
01:13:25,920 --> 01:13:30,600

 

1531
01:13:30,600 --> 01:13:32,670

 

1532
01:13:32,670 --> 01:13:35,180

 

1533
01:13:35,180 --> 01:13:37,380

 

1534
01:13:37,380 --> 01:13:39,270

 

1535
01:13:39,270 --> 01:13:42,180

 

1536
01:13:42,180 --> 01:13:43,410

 

1537
01:13:43,410 --> 01:13:47,550

 

1538
01:13:47,550 --> 01:13:49,860

 

1539
01:13:49,860 --> 01:13:52,230

 

1540
01:13:52,230 --> 01:13:54,570

 

1541
01:13:54,570 --> 01:13:55,760

 

1542
01:13:55,760 --> 01:13:57,740

 

1543
01:13:57,740 --> 01:13:59,930

 

1544
01:13:59,930 --> 01:14:03,470

 

1545
01:14:03,470 --> 01:14:06,800

 

1546
01:14:06,800 --> 01:14:08,600

 

1547
01:14:08,600 --> 01:14:12,530

 

1548
01:14:12,530 --> 01:14:14,840

 

1549
01:14:14,840 --> 01:14:18,560

 

1550
01:14:18,560 --> 01:14:20,870

 

1551
01:14:20,870 --> 01:14:25,790

 

1552
01:14:25,790 --> 01:14:27,110

 

1553
01:14:27,110 --> 01:14:29,300

 

1554
01:14:29,300 --> 01:14:31,520

 

1555
01:14:31,520 --> 01:14:33,880

 

1556
01:14:33,880 --> 01:14:36,560

 

1557
01:14:36,560 --> 01:14:38,600

 

1558
01:14:38,600 --> 01:14:40,790

 

1559
01:14:40,790 --> 01:14:43,610

 

1560
01:14:43,610 --> 01:14:46,510

 

1561
01:14:46,510 --> 01:14:50,690

 

1562
01:14:50,690 --> 01:14:52,310

 

1563
01:14:52,310 --> 01:14:54,800

 

1564
01:14:54,800 --> 01:14:57,830

 

1565
01:14:57,830 --> 01:15:00,350

 

1566
01:15:00,350 --> 01:15:04,100

 

1567
01:15:04,100 --> 01:15:06,050

 

1568
01:15:06,050 --> 01:15:09,260

 

1569
01:15:09,260 --> 01:15:11,360

 

1570
01:15:11,360 --> 01:15:14,300

 

1571
01:15:14,300 --> 01:15:16,070

 

1572
01:15:16,070 --> 01:15:18,470

 

1573
01:15:18,470 --> 01:15:20,240

 

1574
01:15:20,240 --> 01:15:24,590

 

1575
01:15:24,590 --> 01:15:26,960

 

1576
01:15:26,960 --> 01:15:28,490

 

1577
01:15:28,490 --> 01:15:30,680

 

1578
01:15:30,680 --> 01:15:33,230

 

1579
01:15:33,230 --> 01:15:36,920

 

1580
01:15:36,920 --> 01:15:38,570

 

1581
01:15:38,570 --> 01:15:40,550

 

1582
01:15:40,550 --> 01:15:44,690

 

1583
01:15:44,690 --> 01:15:47,330

 

1584
01:15:47,330 --> 01:15:48,950

 

1585
01:15:48,950 --> 01:15:50,600

 

1586
01:15:50,600 --> 01:15:53,870

 

1587
01:15:53,870 --> 01:15:56,720

 

1588
01:15:56,720 --> 01:15:58,760

 

1589
01:15:58,760 --> 01:16:02,120

 

1590
01:16:02,120 --> 01:16:03,920

 

1591
01:16:03,920 --> 01:16:05,810

 

1592
01:16:05,810 --> 01:16:07,880

 

1593
01:16:07,880 --> 01:16:09,730

 

1594
01:16:09,730 --> 01:16:12,710

 

1595
01:16:12,710 --> 01:16:14,330

 

1596
01:16:14,330 --> 01:16:17,300

 

1597
01:16:17,300 --> 01:16:21,170

 

1598
01:16:21,170 --> 01:16:23,660

 

1599
01:16:23,660 --> 01:16:25,730

 

1600
01:16:25,730 --> 01:16:28,640

 

1601
01:16:28,640 --> 01:16:31,970

 

1602
01:16:31,970 --> 01:16:34,070

 

1603
01:16:34,070 --> 01:16:36,110

 

1604
01:16:36,110 --> 01:16:37,640

 

1605
01:16:37,640 --> 01:16:40,730

 

1606
01:16:40,730 --> 01:16:43,370

 

1607
01:16:43,370 --> 01:16:47,600

 

1608
01:16:47,600 --> 01:16:50,990

 

1609
01:16:50,990 --> 01:16:54,860

 

1610
01:16:54,860 --> 01:16:57,010

 

1611
01:16:57,010 --> 01:17:00,560

 

1612
01:17:00,560 --> 01:17:02,060

 

1613
01:17:02,060 --> 01:17:05,510

 

1614
01:17:05,510 --> 01:17:07,640

 

1615
01:17:07,640 --> 01:17:08,930

 

1616
01:17:08,930 --> 01:17:10,520

 

1617
01:17:10,520 --> 01:17:13,190

 

1618
01:17:13,190 --> 01:17:14,780

 

1619
01:17:14,780 --> 01:17:16,310

 

1620
01:17:16,310 --> 01:17:19,790

 

1621
01:17:19,790 --> 01:17:20,810

 

1622
01:17:20,810 --> 01:17:25,160

 

1623
01:17:25,160 --> 01:17:27,290

 

1624
01:17:27,290 --> 01:17:29,750

 

1625
01:17:29,750 --> 01:17:34,100

 

1626
01:17:34,100 --> 01:17:37,460

 

1627
01:17:37,460 --> 01:17:39,680

 

1628
01:17:39,680 --> 01:17:43,760

 

1629
01:17:43,760 --> 01:17:46,010

 

1630
01:17:46,010 --> 01:17:48,770

 

1631
01:17:48,770 --> 01:17:50,090

 

1632
01:17:50,090 --> 01:17:53,000

 

1633
01:17:53,000 --> 01:17:55,370

 

1634
01:17:55,370 --> 01:17:57,050

 

1635
01:17:57,050 --> 01:18:00,590

 

1636
01:18:00,590 --> 01:18:02,300

 

1637
01:18:02,300 --> 01:18:03,980

 

1638
01:18:03,980 --> 01:18:06,620

 

1639
01:18:06,620 --> 01:18:10,070

 

1640
01:18:10,070 --> 01:18:13,710

 

1641
01:18:13,710 --> 01:18:17,490

 

1642
01:18:17,490 --> 01:18:19,800

 

1643
01:18:19,800 --> 01:18:23,120

 

1644
01:18:23,120 --> 01:18:25,700

 

1645
01:18:25,700 --> 01:18:29,540

 

1646
01:18:29,540 --> 01:18:32,220

 

1647
01:18:32,220 --> 01:18:34,260

 

1648
01:18:34,260 --> 01:18:36,270

 

1649
01:18:36,270 --> 01:18:40,590

 

1650
01:18:40,590 --> 01:18:42,780

 

1651
01:18:42,780 --> 01:18:44,540

 

1652
01:18:44,540 --> 01:18:46,410

 

1653
01:18:46,410 --> 01:18:48,900

 

1654
01:18:48,900 --> 01:18:51,870

 

1655
01:18:51,870 --> 01:18:54,420

 

1656
01:18:54,420 --> 01:18:58,200

 

1657
01:18:58,200 --> 01:19:02,040

 

1658
01:19:02,040 --> 01:19:03,540

 

1659
01:19:03,540 --> 01:19:06,380

 

1660
01:19:06,380 --> 01:19:09,210

 

1661
01:19:09,210 --> 01:19:12,930

 

1662
01:19:12,930 --> 01:19:14,940

 

1663
01:19:14,940 --> 01:19:17,660

 

1664
01:19:17,660 --> 01:19:21,240

 

1665
01:19:21,240 --> 01:19:22,890

 

1666
01:19:22,890 --> 01:19:25,410

 

1667
01:19:25,410 --> 01:19:27,270

 

1668
01:19:27,270 --> 01:19:30,930

 

1669
01:19:30,930 --> 01:19:32,970

 

1670
01:19:32,970 --> 01:19:35,280

 

1671
01:19:35,280 --> 01:19:37,740

 

1672
01:19:37,740 --> 01:19:40,080

 

1673
01:19:40,080 --> 01:19:43,050

 

1674
01:19:43,050 --> 01:19:47,620

 

1675
01:19:47,620 --> 01:19:50,779

 

1676
01:19:50,779 --> 01:19:51,950

 

1677
01:19:51,950 --> 01:19:53,839

 

1678
01:19:53,839 --> 01:19:56,180

 

1679
01:19:56,180 --> 01:19:57,799

 

1680
01:19:57,799 --> 01:19:59,959

 

1681
01:19:59,959 --> 01:20:07,220

 

1682
01:20:07,220 --> 01:20:16,440

 

1683
01:20:16,440 --> 01:20:19,660

 

1684
01:20:19,660 --> 01:20:21,100

 

1685
01:20:21,100 --> 01:20:23,350

 

1686
01:20:23,350 --> 01:20:24,610

 

1687
01:20:24,610 --> 01:20:24,620

 

1688
01:20:24,620 --> 01:20:25,270


1689
01:20:25,270 --> 01:20:27,580

 

1690
01:20:27,580 --> 01:20:30,250

 

1691
01:20:30,250 --> 01:20:33,820

 

1692
01:20:33,820 --> 01:20:36,370

 

1693
01:20:36,370 --> 01:20:38,500

 

1694
01:20:38,500 --> 01:20:41,560

 

1695
01:20:41,560 --> 01:20:44,920

 

1696
01:20:44,920 --> 01:20:46,870

 

1697
01:20:46,870 --> 01:20:49,540

 

1698
01:20:49,540 --> 01:20:52,120

 

1699
01:20:52,120 --> 01:20:54,940

 

1700
01:20:54,940 --> 01:20:56,860

 

1701
01:20:56,860 --> 01:21:01,240

 

1702
01:21:01,240 --> 01:21:03,640

 

1703
01:21:03,640 --> 01:21:07,540

 

1704
01:21:07,540 --> 01:21:09,100

 

1705
01:21:09,100 --> 01:21:10,390

 

1706
01:21:10,390 --> 01:21:13,350

 

1707
01:21:13,350 --> 01:21:15,280

 

1708
01:21:15,280 --> 01:21:17,800

 

1709
01:21:17,800 --> 01:21:19,720

 

1710
01:21:19,720 --> 01:21:20,950

 

1711
01:21:20,950 --> 01:21:23,020

 

1712
01:21:23,020 --> 01:21:27,280

 

1713
01:21:27,280 --> 01:21:30,220

 

1714
01:21:30,220 --> 01:21:32,800

 

1715
01:21:32,800 --> 01:21:36,520

 

1716
01:21:36,520 --> 01:21:38,530

 

1717
01:21:38,530 --> 01:21:39,760

 

1718
01:21:39,760 --> 01:21:39,770

 

1719
01:21:39,770 --> 01:21:41,980