1
00:00:01,069 --> 00:00:03,189

the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  ok let's get started so today we're  gonna talk about measurement and timing  and I want to start out by just showing  you a study that one of my students did  actually at that point he was a former  student where is timing a code for  sorting so here's the code this isn't  exactly his code but it's in the same  spirit so that you get the idea  and so let's just run through this code  and take a look to see what it's doing  it's pretty it's pretty straightforward  so we have a we're going to use the the  time th header file to get access to the  clock get time routine which is going to  be what we use to get timing  measurements and then we have a sorting  routine that we're going to time that  I'm not showing you okay and there is  also a fill routine which is going to  fill up the fill up the array with  numbers okay with random numbers so we  have something to sort and the clock get  time uses a struct that is defined here  and so I'm defining two timing strux a  start and an end  okay so this is just absolute  boilerplate setting up for taking timing  measurements okay and in this case  basically the high order part of the  struct tells the seconds the low order  part tells the nanoseconds and then  we're gonna loop over arrays of  increasing length okay and then what  we're going to do is fill them up I know  I forgot the fill and then we're gonna  measure how much time what the time is  just before we sort then we're gonna  sort and then we're gonna measure the  time after sorting okay and then we  compute the difference and figure out  what the elapsed time is print that out  and then we do it again for a little bit  larger array  okay so is that clear what's going on so  we're just sorting a bunch of numbers  that we're sorting some bigger ones  sorting some bigger ones so we can see  what the growth of the sorting routine  should be okay  that people have a pretty good  understanding of what the code does okay  so so what do we expect to see what's  this curve gonna look like what are some  properties of it  yep  alright so micro is then login but  certainly gonna grow right right it  should go should be up into the right in  fact one rule if you ever get into  marketing is that all graphs must go up  into the right if they're going down  into the right then your your company's  in trouble ok so it shouldn't be going  up to the right and it should follow for  example n low again if it's an N log n  sort which is what this one was I think  he was in this case timing emerged sort  I should go up into the right and should  follow n log in or whatever ok so let's  see what actually happened when he took  the measurements and these are actually  his this is actually his data from gosh  this must have been been 20 years ago or  something here's what it looked like  ok so the the the blue X's there are the  run times okay and then through that  we've plotted two curves one which is  the best fit to order n log n growth and  the best fit to order n growth you  notice that for even though we're going  up to 4 million here there's not that  much difference between n log N and  order n ok you can see it mostly down in  the tails definitely the N log n follows  a little bit better but but really log  in is is is pretty pretty small already  but wow those measured times okay so if  you look they're points way up here  right really slow and basically it's  going  it starts out it goes you know slow a  little bit and then it gets a little bit  worse and then a little bit worse and a  little bit worse notice also that the  bumps are getting closer and closer  together what is going on why like I  don't know about you but I thought the  data would follow you know the  green dots reasonably closely okay but  you can see it doesn't okay it's always  good to have a model for what you think  is going on because then when you  because some people will just take  numbers they'll say here's my numbers  for my for my that I've measured and if  you don't actually have a model for what  those numbers mean you're probably  fooling yourself you're more likely to  have made some sort of error or there's  something going on that you're not  observing or whatever if you don't  actually you know have a model for what  you think it should be going on so  what's going on here who can suggest  that hypothesis for for for what is  going on so he took these numbers on his  laptop by the way  what do you suppose is happening here  some ideas yeah  maybe it doesn't fit in the cash what  would you expect to happen if things  didn't fit in the cash  yeah you sort of think that would go  along right it would sort of go along  and then it would jump right so  interesting you know issue but that  doesn't seem to be what's happening  there it's going op things going back  down it's going up and going back down  right rollercoaster what other ideas are  there good idea good idea let's think a  little bit about what's going on in the  machine what are some other good ideas  or bad ideas let's eliminate some yeah  they're not powers of two these are not  powers of two right because they're  getting closer and closer as we get  bigger and bigger yeah so you're right  it doesn't have to do it's not  correlated with powers of two weird  because sometimes things are alignment  issues and we'll talk more about that  you know come up we want to talk about  caching after the quiz everybody knows  there's a quiz next time especially all  of you who aren't here okay so what else  what else might be going on in the  machine here okay because this is  reality this is what happens when you  take measurements so we're being very  nice to we're giving you AWS run we have  done everything we can to make sure  those numbers come out clean and  beautiful and untouched you know there  they are that is quality quality  measurements we're taking for you okay  but if you had to do it yourself that's  what this lecture in part is about yeah  [Music]  so you think that that there may be  something having to do with the the cash  but I'm going through each time and I'm  refilling the array each time so so  they're kind of starting from a clean  slate similar to clean slate each time  what else is going on in the machine  here yeah yeah there could be totally  unrelated stuff running right URI you're  not you know when you're running you  might have demons you might have all  kinds of things and so forth so he  thought of that okay and he shut down  everything he possibly could and this is  what he got still okay so but that's a  great idea because often there's some  external things going on in this case  it's called multi-tenancy there's more  than one thing using the computer at a  time okay good idea  but happens not to be the one it could  be precision with the timing yeah  sometimes there can be issues there but  this was not this was not a precision  issue this was not a pretty he could  have used a really dumb timer and gotten  something very similar to this what else  is going on on your machine yeah  [Music]  yeah maybes checking for updates that's  once again some some external things no  it wasn't checking for updates wasn't  checking for updates okay what is going  on here what is going on let's have some  more ideas what other things might  disrupt measurements yeah  yeah so so this could be but wow this is  so that would sort of if it was like  quicksort  this was actually merge sort he whose  timing so there's no randomization but  but even that if it were quicksort it  sort of be at random that things would  tend to take longer right rather than  following this crazy pattern what is  causing that crazy pattern ya know  because the random fill is done outside  the timer okay so we we filled and then  we you know each time through the loop  we fill and then we start the timer and  then do it new we take the measurements  so forth yeah maybe it's a Liva it's not  allocating memory but that's an  interesting idea because sometimes you  have things going on where you think  things are happening right away but the  system is being clever and and delaying  it and so you end up paying for it at  some later time and that could possibly  create something turns out not to be  what's going on here  okay well so what's happening here is  that the machine is changing the clock  frequency why is the machine changing  the clock frequency your laptop's change  the clock rate the systems that we have  they changed a lot clock frequency why  did they change it because the laptop is  getting hot so what are we doing we're  running something computational right  and the smaller ones okay we get a lot  of those done until it starts heating up  and so it slows down the system clock to  save power okay and then what happens it  slows it down a little bit cools off a  little bit starts to speed up again okay  and then we run longer and why are these  things getting closer and closer  together  yeah  yeah it takes longer and longer to run  the sorts so you're going to see the  effect you know closer in in an interval  here even if it happened to be equal in  time okay even if it's equal in time  we're doing bigger and bigger problems  so it's going this is this is nuts right  like you know we want to take  measurements we want to know whether the  software is faster what are you supposed  to do so here if you just took a  measurement and said look this is the  time that it takes me to run this code  you would be hugely missing the boat if  you were taking one of those high  measurements you compare a to B right  you run a first then you run B B is  slower oh well that's because when you  ran a it heated up the processor so the  processor slowed it down okay so this  particular architectural feature is  called devfs dynamic frequency and  voltage scaling it's a technique to  reduce power by adjusting the clock  frequency and so apply voltage to  transistors okay so the idea is that it  if the chip ends up getting too hot or  in the case of laptops often if you want  to conserve battery it chooses to slow  down the clock okay and the second thing  it can do is reduce the voltage if the  frequency is reduced okay so when you're  actually running slower you can actually  get the same reliability of switching  with a lower voltage okay at a higher  voltage sorry at a higher clock  frequencies you need enough voltage to  make sure those electrons are scooting  across the transistor junctions fast  enough so the the basic power law that  the electrical engineers use anybody  here an electrical engineer  okay there's good stuff in EEE let me  tell you okay so those of you are too  embarrassed to raise your hands okay I  support EE okay so power goes has CV  squared F okay where C is what's called  the dynamic capacitance there's actually  another term which is the static  capacitance which doesn't have to deal  with frequency or whatever but but for  the for the dynamic power its CV squared  F it's the dynamic capacitance it's  roughly the area of the circuitry times  how many bits are moving so if if bits  don't move they don't consume power for  for dynamic power okay and then V is the  supply voltage and then F is the clock  frequency so if you can reduce the  frequency and voltage you get a cubic  reduction in power and also in heat so  you can see why you know who thinks  their battery doesn't last long enough  yeah okay right wouldn't be nice if it  lasted a month okay so you can see why  they're motivated to play this game okay  to save the battery or to run things as  hot as they can but if it gets too hot  we'll just back off okay but for  performance measurement this is this is  basically a nightmare it wreaks havoc so  the topic of today's lecture is how can  one reliably measure the performance of  software when we have stuff like this  going on in our system now the good  thing about something like devfs is  there's actually ways of shutting it off  and taking measurements okay but you  know it's you probably pay some  performance for doing that okay because  they can basically who's ever worked on  an overclocked system okay some some of  you game players right you know so part  of the game that the idea  they're when they're overclocking it is  let's let's keep you know get things  cool and so forth and there's lots of  games like this they're played in  overclock systems so so I'm going to  talk about sort of three topics today  one is about quiescing systems which is  making them quiet enough that we can  take good measurements getting rid of  some of the noise and the second I'm  going to talk about some tools for  measuring software performance and then  we'll talk a bit about performance  modeling okay so I hope this is a little  motivational boy you know it's like you  think you've seen it all sometimes okay  and then somebody presents you with a  puzzle and it's like what's going on  okay and that takes a little detective  work to figure out that that's actually  what's going on  so we'll start first about quiescing  systems who knows who  genichi Taguchi was anybody here know  he's very famous because he's the one  who made our automobile industry produce  reliable cars okay very very famous  fellow he did a lot of different things  I'm gonna point out one of the things  that he observed and it's it's in the  context of suppose you were an Olympic  pistol coach and you've got two shooters  a and B and you look at their targets  and here's a z' target so we hit the  bullseye good and if you add that up  he's got a pretty decent score okay then  you look at B well B didn't even hit the  bullseye and his score is like really  lousy who do you want on your team a or  b who do you want  yeah cuz it's easy to tell them what to  do you adjust the sight you teach them  to you know shoot down a little bit  lower to the left  okay there's it's easy to diagnose what  his problem is okay that's all you have  to do what do you do for a it's not  clear what you do okay  and so what what he did in the realm of  quality control was say look when we're  producing products before we try to fix  the reliability or whatever is of the  product increase the quality of the  product  let's first reliably produce whatever  we're producing so we get the same thing  every time because then we can go in and  see what the problem is and we can  adjust for it okay and so the idea is go  after the variance first okay go after  the spread try to get the spread as  small as possible try to figure out how  it is that you can isolate you know  produce the same thing even if it's not  very good okay and then go and try to  make change because if you try to make  changes while you have high variance you  don't even know if you're making  progress or not because it could be in  the noise of your measurements okay  now in the context of performance  engineering if you can reduce  variability you can do things like  compensate for systematic and random  measurement errors and you can also do  things like not run as many trials to  discover whether one program is better  than another there are a lot of  advantages to being able to have really  quiet things it's kind of obvious that  you should want to have reliable  measurements okay but but there's  actually as I say a theory behind it for  why it is that you want to have quiet  measurements and and what you need to  focus on now in our computer systems  there are lots of sources of variability  and some people came up with some of  them here let me mention just a few of  them here so there are things like  demons and background  jobs things that are running on your  your system that are helping the system  to do whatever it's doing okay many of  them are unnecessary and those can be  running so for example maybe you set up  a cron job you know to you know do  something for you every night and that  happens to be just when you happen to be  be making some measurements well that's  not really good because you're now  sharing the resource ok interrupts ok  something comes in I had one time where  we were measuring stuff on the  connection machine cm5 which in its day  was the world's most powerful computer  1993 it was the top of the list of the  supercomputers and now this computer is  more powerful it cost 43 million dollars  or something ok I mean Moore's law has  really made a difference and we were we  were measuring the performance of  something and we kept getting these  anomalous results and eventually we  tracked it down it took us a lot of work  but we tracked it down the graduate  student who was running it while he was  running it he was moving the mouse  around and when you moved the mouse it  was interrupting 200 times a second to  deal with the interrupts that was  interfering with our measurements ok  true story true story okay so so  interrupts can have a so it's like oh ok  and we you know we disconnected the  network even ok so that we wouldn't get  external things and we're still getting  this thing what is going on ok and  eventually we tracked it down to to the  fact that he was just sitting there I  Lehman so it's like started up and then  hands off there's like nobody move ok  code and data alignment where the code  wise affects its performance if a code  goes across 2 cache lines versus  completely within one cache line that  can have a difference depending upon  what scan  flick ting in the cache okay or if you  go across page boundaries it can have a  very big difference for example in the  translation lookaside buffer or TLB you  may be going for different entries there  may be different different things there  so code alignment can make a difference  in the in the what you're doing  thread placement okay which well you  know if you have a multi-core machine  which core is it running it turns out  the system likes to use core zero for a  lot of its stuff so if you're going to  take reliable measurements don't run on  core zero okay runtime scheduler okay  the fact that for example we have a  randomized scheduler which means that  there's random numbers going on so  you're going to try to take measurements  in the midst of all this randomness a  hyper threading hyper-threading is where  they take one core it's also called  symmetric  it's called symmetric multi threading or  no simultaneous multi-threading that's  what it's called simultaneous  multi-threading or hyper threading is  what Intel calls it what that is is they  have one functional unit okay and then  they run two instruction streams through  it at the same time each with its own  set of registers but using the same  functional units and they get it  basically a 20% speed up or something  from having it looks it looks from the  software point of view like you've got  two processors but really it only gives  you one point two processors okay so  it's one of these things that by the way  if you start counting up how many  processors and saying work over the  number of processors you say well how  many processors do I have well you may  have just a hyper threaded processor  rather than a real processor okay so one  thing for example in the in the cloud  system that you folks are using we turn  off hyper threading okay so that we can  get more reliable measurements and so we  can look at the measurements as a  function of the number of processors  multi-tenancy and this is particularly  important in the cloud if you're in the  cloud there are other people using the  system they can  end up using lots of resources that  maybe you want like cash okay and they  can end up using network traffic that  may affect you and such I'm actually  quite amazed at how well Amazon does in  AWS in making it so that that stuff  doesn't affect you very much  okay our numbers show that they're in  fact they're definitely the leader right  now in in having you know repeatable  measurements you know compared to all  the cloud providers we talked about  devfs there's also another one called  turbo boost  so turbo boost looks to see how many how  many jobs are actually running on the  multi-core if there's only one job  running on the multi-core increases the  clock frequency for that job or if  there's just a few as soon as another  one comes in it slows things back down  so tries to give a boost when you're in  executing serial codes that says well  I'm not I'm not  you know generating heat from all the  cores I just have to generate it from  the one that I've got I just have to you  know I can afford to have one of them  run hotter network traffic and there's  by the way a bunch of other ones we're  going to talk about a few of these but  first let me tell you what the impact of  quiescing a system is so this is joint  work that that I did in my group with  Tim kayler so we wrote a silk program to  count the primes in an interval and we  ran on a see for instance 18 core  similar what you're running with we had  2-way hyper threading on turbo boost was  on we had 18 silk workers with a hundred  runs each about one second okay and what  I've plotted here is how much is the  percent that that each run was we  basically ran a hundred runs and then I  sorted them from smallest run to largest  and I normalize them to whatever the  minimum one was so that each each thing  here is how percentage above the minimum  so you can see that the slowest run was  almost 25 percent you know slower than  the fastest  okay so you can see sort of what the  impact is so 25% if you're trying to  produce a code by getting a 3%  improvement you know 30 times you can't  you know there's so much noise here it's  very hard for you to figure out that  you're 3% faster okay so if you quiesce  the system okay same thing we turn  hyper-threading off turn to burst off  and we quieted all the demons and and  and so forth then out of a hundred runs  we got essentially the same value every  single time except for three times and  notice that the scale here is changed  the scale here is now point less than  0.8 percent less than 1% slower so this  says that hey if I took a couple of  measurements I'm very likely to hit  something that is that is the you know  the the real running time which tends to  be for this what the what the minimum is  here okay because all the other stuff is  noise is just adding in okay makes sense  so it is possible to quiesce a system  it's not that hard okay it just takes a  small matter of you know programming and  systems work so here are some some tips  on how you can if you wanted to do this  for your laptop for example make sure no  other jobs are running shutdown demons  and cron jobs disconnect the network  don't fiddle with the mouse okay for  serial jobs don't run on core 0 where  interrupt handlers are usually run to  him - threading off turn off DFS F devfs  turn off turbo boost use task set to pin  workers to course of tasks as utility  that says I want you to run this thread  on this core and don't  let the operating system bounce it  around okay so normally operating system  Maps threads two cores the workers the  silk workers two cores in any way that  it feels like this says no I want you to  have it exactly here so that when I run  something else is exactly the same thing  and so forth and we've already done a  lot of this for you for AWS run okay by  the way there is no way of getting a  completely deterministic result out of  running on modern hardware does anybody  know why I mean if I have a serial  program and it's to terminus stick  serial program and I set it up and I  reboot the system so it's got exactly  the same content etc I still can get  non-deterministic results does anybody  know why yeah  no you can turn that off so you can make  sure that so they actually do randomized  address space for security reasons but  when you run under the debugger and so  for they tend to turn that off so that  you can get repeatability for debugging  purposes yeah no those are generally  well to a first-order these so now those  are deterministic but there is something  in the hardware which is  non-deterministic does anybody know what  is happen to know can guess what it is  something the hardware isn't  non-deterministic yeah disk access would  be but if I'm just running something in  core I'm not using disk I'm just going  to run it turns out non-deterministic  even though I'm just running everything  inside with you no ordinary DRAM memory  and so forth yeah no the out of order  execution is all deterministic there's  no randomization in there okay yeah  branch prediction is all is all  deterministic algorithms all  deterministic yeah  system clock signals very regular very  regular if you've turned off you know  gvfs and stuff like that yep  so the Linux schedule is in fact a  deterministic algorithm and if you're  just running on one core the scheduler  never comes into it okay so there is one  source of non-determinism and that is  memory errors so there's a chance that  one of your memory when you're accessing  the DRAM that an alpha particle collided  with one of the bits and flipped it and  there's Hardware in there to do error  correction but it takes an extra cycle  to do it so if it if it reads the memory  location discovers that there is a that  there is a an error in what it read then  it performs the correction and then you  get it and that's something it's  completely non deterministic because  it's alpha particles coming from outer  space and you know space aliens you know  messing with your system yeah no  actually it there's actually was a and  now most cache is covered by error  correction but the most likely if you  look the the biggest memory is the DRAM  so that's the most likely source of  these things but yeah there's so so in  any case I want to now just talk about a  few things which just some examples of  things that that might come up for you  so I'm these are ones that I'd mentioned  but I just want to go through them a  little bit more depth so one of the  things is code alignment can make a  difference okay so what happens is you  know you have your program and you make  a change that you think is improving  your program but what happens let's say  is it causes you know an extra bite to  be put into the code okay you know so  maybe you know the compiler is really  smart you made some little change then  everything that follows it gets shifted  down and so the cache alignment issues  can be completely different  something can go across a page boundary  that didn't use to go across the page  boundary and that can have a big impact  on your performance this is like yikes  right this is like yikes how am I  supposed to you know it's like maybe we  just pack up on performance engineering  right this is okay so everybody gets  what the issue is there so you know in  this case I inserted one bite  well everything after that it's all  linear in memory would would change  here's another one that's even more  insidious okay if you change the order  in which the dot o files appear on the  linker command line that can actually  have a bigger effect than going between  - OH - and - o3 okay you know you when  you compile you can buy all this order  versus this order okay you get can have  a actually quite a big big difference  yike okay so what do you do well first  of all one of the things I'm glad to say  is that the compiler people have  recognized this and it's the situation  is not as dire as it was years ago what  they do now very often is do a lot of  alignment already so for example it's  common for compilers to produce to start  every function on the first word of a  cache line okay that way when things get  slid down you might pop from one cache  line to the next okay for where it  starts but you're not going to you know  affect the you know where you lie on the  cache line which can make a difference  by the way in branch predictors and  things like that  okay and so that helps that really  quiets a lot of things and in fact they  give you some directives so LLVM has  these switches as far as I could tell  the first one which is a line all  functions I think I was unable to test  this in advance but I suspect that it's  actually already doing  this and this is actually in no ah'd  because it's already aligning all  functions okay I don't know that for a  fact but you can give the switch anyway  which will help which will help if if if  I'm lying okay  so that it forces the alignment all  functions so all functions start on a  cache line and that way if you change  one function it's unlikely to check it  won't change the cache alignment of  another function it'll only change the  cache alignment of the function that  you're messing with you can also ask it  to align all blocks in the function so  remember that in LLVM we have these  basic blocks these pieces of serial code  with with you know with links you know  the the basic blocks so what you what it  will do is force every one of those to  be on a boundary but of course what that  means is now you've got a jump between  these codes even if it were going to be  the next instruction or put in a bunch  of no-ops so that can substantially  increase the size of your that can I  substantially increase the size of your  binary and it can slow you down but on  the other hand you'll get very reliable  results from then on because every block  is now cache aligned okay  probably more practical is too long a  line is to force the alignment of all  blocks that have no fall through  predecessors that is you don't have to  add no so this basically reduces it to  the ones that are usually causing the  trouble  so a line of code is more likely to  avoid performance anomalies but it can  also sometimes be slower and so one of  the questions as well which you know  what which matters to you and your  particular particular thing okay  here's one that I love so the example  that I gave before of the order of  linking we have that as a reading  assignment by the way on the on the web  that paper and here's another result  from from this paper by Mike Mitka wits  and  and his colleagues the if they have a  wonderful paper called producing wrong  data without doing anything obviously  wrong okay so one of the examples they  give me swords the ordering of things  saying that look that actually had an  impact comparative between Oh two and O  three the program's name can affect its  speed and here's why the executables  name ends up in an environment variable  okay so when you start up a program it  puts it in environment bill the  environment variables it turns out end  up on the call stack okay so you can  find out from the program what's the  name of the program I was invoked with  okay the length of the name affects the  stack alignment if you have a longer  name they put longer you know a longer  piece there and so if the data happens  to go across page boundaries some  critical piece of data goes on to pages  rather than one that could have a big  impact on your performance okay and  there are other date alignment problems  that can can arise okay so yeah this is  kind of nasty stuff so as I say what  we've done it we've given you a very  quest system we have done everything we  can to make it so that you're you know  that when you submit stuff in this class  its measured but as we get along in the  semester and you're doing more  sophisticated stuff some of this is  going to fall increasingly to you to do  things that make it robust okay okay  let's move on we talked about quiescing  now let's talk a little about about  tools for measuring software performance  so I did a little thinking and I figured  that there's actually like about five  different ways I know of measuring the  performance of a program so one is you  can measure the program externally you  can run the time command if you say time  and then give a shell command it will  run the shell command and then tell you  at the end how long it took okay you can  the program okay that means to actually  put timing calls into the program so you  can use things like clock get time which  I recommend and we'll talk a little bit  about these other to get time of day and  already TSC and you can either do this  by hand where you say I'm going to time  something myself or it turns out you can  have compiler support for doing taking  timing measurements but what that  involves is changing the program in  order to put these timing calls in and  of course you have the effect that if  your perturbing the you know if you're  putting these timing calls in you can be  changing the timing so you've got that  problem to worry about as well  one of the ones is you can interrupt the  program one of the cheapest ways you can  do is just basically take gdb so you  start running your program gdb  you know you run it under gdb and then  you type control C and then you look  where is the program and you do that a  few times you say hey the same routine  is always where the code is running oh  that must be where it's spending all its  time okay that's actually been put into  a thing called poor-man's profiler and  is and then you can actually this is  essentially the method that Jeep refuses  okay we'll talk about that in a little  bit you can exploit hardware and  operating system support so they're a  bunch of hardware counters that the  operating system and hardware support so  that for example perfuses and so you  folks are familiar with the perf toolset  those are basically you know using  hardware and operating system support  and you can also simulate the program  you can run it as a simulation and then  you really can go in and do anything you  want to understand but of course it's  much much slower and you hope that your  simulator is modeling everything that's  of relevance to you maybe it's not not  only something that turns out to be  relevant for you so I'm going to talk it  give some examples of each of these five  things okay so let's start with  the time command so it can measure a  elapsed time user time and system time  for an entire program does anybody know  what these three terms mean elapsed time  user time and system time sure what we  call wall clock time good it's in the  kernel working on your stuff as opposed  to somebody else's okay so that's that's  exactly right so when you run the time  command you get some numbers like this  where it says here's the real time  here's the user time here's the system  time now you might think that the user  time in the system time should add up to  your total time uh doesn't work that way  and part of the reason is that it made  the processor may actually not be  working on your code you may be context  switched out and something else is in  okay so so in any case we have those  those three types of things wall clock  time the amount of processor time spent  in the user mode code within your  process and the system's time which is  the time spent in the kernel but within  your process okay for example satisfying  system calls and such now the the the  timing call that I recommend you use and  that was used in the example that I gave  is clock get time in particularly there  are a bunch of options to that and the  one that I strongly recommend is clock  monotonic okay and it takes about  eighty-three nanoseconds to to actually  read what the time is that's about two  orders of magnitude faster than a system  call and  one of the things about it is this is  such a funny thing to have to say is it  guarantees never to run backwards okay  turns out the other timers can run  backwards you can take measurements and  discover their negative okay this one  does not run backwards part of it is  because some of the other timers do  things like oh you know there's this  National Standards thing that  periodically your your computer goes out  to find out what the real time is and it  resets its clock to be consistent with  whatever the global clock is okay and  that will cause the clock to be updated  in a non-standard way okay and where  we're suddenly you lost some time or  gained some time okay so this is really  I you know the best nation is the only  unfortunate thing about this is that it  is non-deterministic how long it takes  and let me explain a little bit what's  going on in this so what happens is it  takes a measurement it has to take two  measurements to figure out what the  elapsed time is to find out what the  actual time is okay can't just take one  measurement because it may have been  swapped out and the the colonel helps  support in userspace  something that says here's the total  amount of time you've spent up until you  started your time slice so when you read  that you have to read those two values  well how do you know that you don't have  an atom isset Eve I elation you read one  of the values you got switched out you  get switched back in now you have a new  value for the other one so the way it  does it is it reads this register so it  reads what the operating system has kept  is its cumulative it reads the the the  clock and then it reads that register  again and if those two things differ it  knows there's been a switch in there if  they're the same knows there isn't and  the the number that it can take is  reliable so in that kind of case it will  actually take  two measurements in or more than one  measurement I mean could you could do it  again and it would you know you could  have another context switch and you  could do it again have another but this  thing is generally pretty fast so um and  on my laptop it takes about 83  nanoseconds to to run there's there's a  lot of people say well why don't I just  read the cycle counter that's actually  cheaper it runs in about 32 nanoseconds  and that you can do with the are DTSC  you know read the time stamp counter  instruction and you can do it yourself  by using a you know a built in assembly  and what it does is it returns to how  many clock cycles since the boot and it  rhymes in about 32 nanoseconds but why  not use this well one thing is that RTI  DTSC may give different answers on  different cores on the same machine so  the cycle counters on a on a processor  by processor basis okay sometimes TSC  runs backwards okay as I mentioned okay  and also the counter may not progress at  a constant speed so remember that the  time you know between you know the the  system is possibly slowing and speeding  up the counters and so forth and  converting clock cycles for that reason  two seconds can be very tricky so I  recommend you stay away from this faster  counter this faster timer okay the other  one is don't use get time of day that's  the one most people know that gives you  you know microsecond precision it's not  actually microsecond accurate but it  gives you microsecond precision because  it has similar problems whereas this  this particular the you know the clock  get time monotonic has been very well  engineered in my opinion to give good  reliable numbers at a reasonable  cost okay any questions about that about  taking measurements and what to use this  stuff by the way over time it's going to  change okay you know people are going to  come up with better ways or worse ways  or they'll say we're not going to  support that anymore or what-have-you  and then if you're out there as an  engineer you're going to be on your own  okay hopefully you know what some of the  issues here are and you're prepared to  be on your own yeah  hmm  so when it does that it aggregates it  and the operating system has to do it  those numbers actually are very are  relatively coarse grained you cannot  time something that's very short with  time for example with it with the time  command okay in general my experience is  you should even with something like you  can get very fairly fine grained  measurements with with the clock get  time you can get fairly good  measurements there but unless you are  aggregating unless you're running code  that's running around a second certainly  at least a tenth of a second okay if  you're not running things that are that  long you run the risk that you've got  really really bad  you know you get unlucky let me point  that out in in this particular example  so so here's the interrupting strategy  which we talked about briefly this is  where I just control C and at random  intervals and you look at the stack and  say who needs a fancy profiler in fact  there are large companies who use this  for debugging  they are big big coats okay Facebook  comes to mind okay they actually use  this technique it is a really easy  worthwhile technique for figuring out  where time is being spent now there are  some other things the the poor man's  profiler and people have built things  like G prof and so forth to increment to  automate this strategy so you get this  information because then it  automatically AK units the looks at the  stack what's being executed puts that  into a call graph and so forth and  figures out you know where the time is  going but neither of those programs is  accurate if you don't obtain enough  samples and just to give you an example  G profs ampuls only 100 times per second  so if you're gonna use G prof for for  timing something that's only a second  long you only got one  samples how many samples is a hundred  that's not actually a real lot okay and  so you can get wildly inaccurate numbers  from interrupting okay but on the other  hand for a quick type of you know I use  G prof all the time right it's it's  quick we do control seeing all the time  it's really quick I don't have to  install anything I just I just you know  take a look and it gives me a first cut  at at what I want to do okay it all  depends you know you don't need all this  surgically precise tools all the time  sometimes a really dumb tool is adequate  for the job and a lot quicker to deal  with okay hardware counters so one of  the nice things that's happened in  recent years is that there's become  available library called live p FM 4  which is virtualizing all the hardware  counters so that you have access to them  with to all of these types of events on  a per process basis okay so normally  there's hardware counters but then you  switch to some other you know there's  context switching going on to some other  process then what happens to your  calendars they have to be saved they  have to be you know updated so anyway  live p FM 4 does all of that kind of  virtualization to make it so that the  counter you can view it as if it's your  own counter ok and proofs that for  example employees that there are a lot  of esoteric hardware counters and as I  say good luck and figuring out what they  all measure because they often are not  well documented okay a few of the  important ones are well documented but  but most of them are very poorly  documented exactly what it does a really  good example was I we had somebody who  was looking at cache misses okay to  figure out how much bandwidth so last  level cache ok l3 cache misses to count  how much data was being trans  heard from from DRAM okay and they were  getting curious numbers that didn't seem  to measure up okay and it's like wait a  minute have a miss moves from moose from  from DRAM into the onto the chip why is  that not counting how much stuff is  being moved if I count up how many cache  misses times how many bytes in a cache  line which is what machines we're using  how many how many bytes in a cache line  on the machines we're using okay  sixty-four okay gotcha oh you guys are  okay 64 4 but not every machine has that  but anyway so why was this not measuring  how much stuff is being moved I'll give  you a hint it used to measure how much  stuff was being moved but then those  architects they are such pesky clever  people and they put in a great feature  prefetching okay they put in prefetching  they there's things that fetch it and it  doesn't update that counter okay so if  you want you have to count how many  prefetch incidents you have as well okay  and so you can often cobble this  together but good luck figuring out what  some of these two okay also watch out  you may think that the tools let you  measure a lot of counters if you want  but if you read the fine print it turns  out that if you do more than four or  five it starts essentially time sharing  the available counting bandwidth that it  has and it's not it's actually just  doing something statistical rather than  actually counting them so you can't  count more than like four or five  actually four or five I think is a high  number  but somebody I know Wells who knows this  stuff said said four or five is probably  what it is today okay so that's hardware  counters so hardware counter is a good  good technique next one is simulators so  things like cash grin usually rush run  much slower than real time but what's  great about simulators is you can get  repeatable numbers out of them okay you  know you run the code you run it again  if you've setup everything right you can  get and you can see what's going on  inside the downside is that they don't  necessarily it's slower and it doesn't  necessarily model everything going on in  the cache but for things like cache  misses this is a great tool okay to just  figure out what's the fundamental cache  miss and we'll talk about that when we  talk about caches in the next couple  weeks and if you want a particular  statistic in principle you can go in and  if it's an open source simulator like  cache grinned is you can collect it  without perturbing the simulation so any  question about you know these ways of  collecting measurements there are a  whole bunch of ways of doing they all  have pros and cons it all can be useful  in a given context they all have some  some flaws okay a really good strategy  I'll talk about this later is  triangulation I never take one  measurement and believe it I always want  to take at least two measurements okay  in different ways and make sure they're  telling me the same story  okay triangulation if there's a  discrepancy then I want to know what's  causing the discrepancy but I never  trust one number and I never trust any  numbers without having a model for what  I think is is coming up and in fact  that's what we're gonna talk about next  is performance modeling so any questions  about measurements and such so isn't it  good we have AWS run I mean right it's  like number comes back it's the number  and it's actually a pretty good number  okay we worked very hard to make that a  pretty good number  okay so performance modeling so yeah we  cover a lot of stuff in this class as I  think some of you have started to notice  and but really performance software  performance engineering is pretty simple  process okay you you take a program that  you want to make go fast program a you  make a change to program a to produce a  hopefully faster program a prime you  measure the performance of program a  prime if a prime beats a then you set a  equals a prime and if a is still not  fast enough you repeat the process okay  right but that's what we that's  basically what you're doing right it's  pretty pretty simple okay  and as should be apparent if you can't  measure performance reliably it's hard  to make many small changes that add up  because it's hard to tell whether a  beats  a prime sorry whether a prime beats a  right it's hard to tell okay and so as a  consequence what we want to do is want  to model performance so that we're in a  position to draw accurate conclusions so  we want to do things like drive the the  variability of measurement down to zero  and we want to do things like figure out  ways of using statistics okay to give us  more accurate picture of what's going on  then maybe what is appearance and that's  basically what we're going to talk about  for a little bit part of what I'm going  to do is going to talk about statistics  how many people have had a statistics  class or a machine learning class or  something read ill dealt with statistics  beyond probability I mean I mean real  statistics right you know sample sample  means and things like that so a few of  you okay the basics of what you need to  know you can find from Wikipedia or math  world or someplace like that  and I'm not gonna try to teach you  statistics 101 but I point you in some  directions and give you some pointers so  some tools you can do okay so here's a  puzzle suppose you measure the  performance of a deterministic program a  hundred times on a computer with some  interfering background noise what  statistic best represents the raw  performance of the software is that the  mean of the arithmetic mean of those  runs  is it the geometric mean of those runs  is it the median of the runs is that the  maximum of the runs is that the minimum  of the runs okay this is where it's  helpful if we had those you know those  clickers or whatever they have right you  know but we don't so so I'll ask people  to vote now I want everybody to vote  once again it doesn't matter if you're  right or wrong you can be right for the  wrong reasons you can be wrong but have  the idea right but it's fun when  everybody participates certainly more  fun for me when I see hands go up then  when I see people sitting there looking  bored okay okay how many people think  arithmetic mean would be a good way of  measuring the raw performance arithmetic  that's a that's the most common  statistic that we ever gather okay  what about geometric mean okay what  about median okay good what about  maximum one for maximum what about  minimum okay so it turns out actually  these are all good measures to take and  depends upon what you're doing with them  but turns out minimum is does the best  at noise rejection and that's because  you expect any measurements higher than  the minimum if it's a deterministic  program and so forth that's going to be  due to noise so if you're really  interested in knowing how long  fundamentally you're  Co takes on the underlying hardware when  there's other things going on taking the  minimum rejects it now you might say the  median also rejects noise but it doesn't  because because if the you know if you  view your your program as being it's  running time plus noise then the median  is going to give you some number that is  in the midst of the noise right it's  going to have some complement so minimum  is the only one that really rejects all  of them but they're all useful measures  in different contexts and there are ways  that you can use the mean and some of  these other ones as we'll talk about in  a minute to get information about making  decisions okay because the thing that  we're after is not necessarily always  the raw performance of the software  sometimes we're interested in whether a  beats B that's a different question than  how fast does this sort of fundamentally  go now  so there are a lot of different types of  summary statistics and there's lots of  reasons to pick different ones so for  example if you're interested in serving  as many requests as possible in a web  server for example you're going to be  looking at something like the CPU  utilization and taking the the  arithmetic mean to try to understand how  how those things add up if all tasks  have to be completed within 10  milliseconds then you're going to look  at you're looking at the total and  you're going to add it up and you're  going to be interested in making sure  that that each one is is small and  that's also what the mean does and  you're gonna be looking at wall clock  time if you wanted to ensure that most  requests are satisfied within a hundred  milliseconds you might be looking at the  90th percentile behavior okay and you'll  say yes I won't make every one but I  want 90% of the time I want to get it  there and I'll be using something like  wall clock time okay  there may be in in a lot of web  companies there's a thing called a  service level  Raymond this is what they should give  you for your telephone but they don't  tells you what kind of service you can  expect and if they don't meet that  service requirement then they haven't  lived up to it instead we buy these  phones and we get the service and we  just get whatever they decide to give us  right but if you're a big company you  insist that you get some kind of service  out of the people that your you're using  and so there that's typically some  weighted combination and you're using  sort of multiple things you might want  to fit into a machine with a hundred  megabytes of memory some sort of  embedded machine or whatever then you're  going to be interested in maximum of the  memory use so it's not all the  performances not all just time you might  want the least cost possible and you're  looking at things like energy use etc  here are the fastest biggest best  solutions you can see average comes up a  lot you know as one of the ways so I  wanted to though cover one particular  example which I find is the most common  place I see a misuse of summary  statistics and that's for when I'm  summarizing ratios so suppose I have two  programs a and B and I run the four  trials normally you'd run a lot more but  I wanted to fit on a slide okay and  program a you know on whatever trial one  was took took nine seconds say program  two on trial two it took eight two and  ten and program about three to twenty  and - okay so I can compute the mean for  each of those so the mean of the program  a is eight point two five and the mean  of program B is four point seven five  over those four benchmarks okay I can  also take a look at how much is a  winning sorry its be winning over a okay  and so if I take the ratios I then get  three four  one tenth and five for a mean of three  point one three  it's actually three point one two five  but I'm only keeping things to two to  two digits okay and so if I was asked to  summarize this I could perhaps conclude  that program three is more than three  times better than program a based on  these statistics okay  but there's a bug in that reasoning can  anybody see the bug this is wrong  [Music]  it doesn't make sense to take the  arithmetic mean of a bunch of ratios why  is that yeah one thing here is that you  can see is that notice that the mean of  the ratio is not the same as the ratio  of the mean that should be some  suspicious why am i comparing should I  not be comparing the ratio of the mean  or sort of a comparing the mean of the  ratios  okay so so that's not particularly good  another thing is to suppose I take a  look at the ratio B over a and I take  the arithmetic mean then what I discover  is that a is better by a factor of  almost three than B so clearly taking  the the average of the of the of the  means is you know is assuming the  average of the ratios there's something  wrong with that right and in particular  as I say the ratio of the means is not  the mean of the ratios okay and then  your intuition is spot-on  okay I suppose instead I compute the  geometric mean so geometric mean is  basically like taking the average of the  logs the arithmetic mean of the logs so  you're basically taking the product and  taking the nth root of the product okay  and I've computed that for these things  and now I've taken the arithmetic mean  of the run times because that makes  sense that's kind of how an average over  these things how long did things take  and now I look at a over B and B over a  and I get the same the same thing and  it's in fact the case that the ratio of  the means is the mean of the ratios okay  so when you look at ratios there's  another place that comes up where you  look at rates and I've seen people look  at rates and it turns out when you're  looking at rates often it's the harmonic  mean that you want in order to preserve  these good mathematical properties and  make sure that  like it's really bad if I you know if I  do this thing and I say look how much  better a is than B and somebody if I  flip the ratio B would be better than a  right that's really suspect if if if the  identity of whether you're a or B and  which one was the numerator which in the  denominator you know had an impact right  that would be that would be like really  suspect okay so this is the kind of  thing when you're thinking about how  you're reporting things and so forth you  want to be careful about when you're  aggregating things to make sure that you  have the basic mathematical properties  met okay and what's nice is like you  know one divided by one point five seven  here is in fact zero point six four okay  so it didn't matter which way I took the  ratio I got the same answer okay one of  them is a so B is better that by a  factor of about one point six something  like that it was a little bit better  it's not three times better and it for  sure isn't the case that a is better  than B by a factor of three okay good  any questions about that this is kind of  good okay suppose I want to compare two  programs a and B which is faster and I  have a slightly noisy computer on which  to measure their performance what's a  good strategy  what's a good strategy for comparing  them figuring out whether a is better  than B or B better than a sure  so you could take sorry so you're saying  I'm gonna run multiple runs that's a  great idea okay we're gonna do multiple  runs and what am i doing for each of  these runs the low order statistic of  the like for example like minimum were  or 10% or something really low okay so I  take that I have one number for for  program a I have one number for program  B I ran them in times what else well am  i comparing am i comparing the am i  comparing the the two minimums I've got  there or am i comparing each one  I say oh I see you're saying take all  the measurements that go below your your  10% cheapest ones 10% cheapest ones and  then compare those okay that might give  you  so you're substantially doing noise  reduction is what you're saying and then  other than that you're comparing by  comparing means okay that's an  interesting methodology you probably get  something reasonable  I would think but you probably couldn't  prove anything about it but that's an  interesting idea yeah that's that's an  interesting idea what else yeah  good so you're saying do a bunch of  head-to-head runs and mark just who wins  over those things and why is that a why  is that a good so which so one wins more  than the other or the other ones more  than the one what what good is that  yeah so this is actually a very good  strategy okay and actually has some  statistical muscle behind it okay so  what you can do is do in head-to-head  comparisons between a and B okay so in  both these examples is like yeah we  better run it a few times okay and  suppose that a wins more frequency more  frequently okay so now what we want to  do in the statistics where they call the  null hypothesis is the that B beats a so  even though we see a B's be the null  hypothesis that were wrong and that a  beats the B beats a and what we then can  calculate is what they call the p-value  which is the probability we'd observe  that be that a beats be more often than  we did okay so for example for each of  these two imagine that in in the worst  case let's just say that they were equal  in performance and all we're seeing is  the noise then I would expect that I  would get about an even number of things  and so the further that I deviate from  them being even especially once I you  know that distribution is essentially a  Bernoulli distribution the further I get  out or a t-distribution if you have  small numbers as I get further out from  away from the mean by what the variance  is of just flipping coins in that case I  can calculate what's the probability  that I'm seeing something that would be  that extreme and that gives me a good  reason to reject the null hypothesis if  it turns out that deviates by Allah  okay so this is very standard stuff in  the social sciences who's had a course  in this kind of stuff and testing null  hypothesis by ology and so forth and you  remember anything from it or just  vaguely yeah yeah okay that's fine you  know one of the things I have found in  life is that you know I don't remember  anything I learned in college to speak  of right okay what I had to do is  relearn it but I relearned it a lot  faster than when I was in college you  know it and so that's part of this is  also so when you see it again okay now  you know how to go about learning this  so so I would once again you can look  this up on on like Wikipedia or whatever  if you want to use this methodology it's  a it's a good one for being able to  determine even in the presence of noise  what it does do is it doesn't tell you  what the raw performance was but it says  in that environment which one is  actually faster and that may actually be  a more relevant question because you're  not always going to have this completely  quiet system you may have a lot of noise  going on a system you'd like to know  which one is going to behave better in  what that actual system is and so this  methodology is pretty good  note that with a lot of noise we need  lots of trials the last thing that I  want to talk about which I won't is  fitting to a model and this is the issue  of sometimes you measure things but  you're interested in a derived  statistics such as such as in this case  gathering some time accounting and  instructions counting cast measures  counting time and asking okay what's the  you know what can i estimate is the  instruction time and what is the cache  time and to do that you do a least  squares approximation and there's once  again some statistics behind that I'm  not going to go over it here but you can  look in the in the notes there are a  bunch of issues with modeling which is  that you can over fit very easily if you  if you add more basis functions you will  fit the data better  and so how do you know if you're  overfitting answer is if you remove a  base to function it doesn't affect the  quality very much is the model  predictive I'm actually going to just  let you guys look at these because it's  it's pretty good is that I think they're  pretty self-explanatory  giant of science this is Lord Kelvin  what is Kelvin famous for besides Kelvin  okay  he was the Guru of measurement okay and  he said to measure is to know and that's  a good one and he said also if you  cannot measurement it you cannot improve  it  so both very apt sayings from the same  guy so there's a reason he's got a big  forehead I guess so anyway that's it  thanks very much and good luck on the  quiz on Tuesday  you

2
00:00:03,189 --> 00:00:05,769
the following content is provided under  a Creative Commons license your support  will help MIT OpenCourseWare continue to  offer high quality educational resources  for free  to make a donation or to view additional  materials from hundreds of MIT courses  visit MIT opencourseware at ocw.mit.edu  ok let's get started so today we're  gonna talk about measurement and timing  and I want to start out by just showing  you a study that one of my students did  actually at that point he was a former  student where is timing a code for  sorting so here's the code this isn't  exactly his code but it's in the same  spirit so that you get the idea  and so let's just run through this code  and take a look to see what it's doing  it's pretty it's pretty straightforward  so we have a we're going to use the the  time th header file to get access to the  clock get time routine which is going to  be what we use to get timing  measurements and then we have a sorting  routine that we're going to time that  I'm not showing you okay and there is  also a fill routine which is going to  fill up the fill up the array with  numbers okay with random numbers so we  have something to sort and the clock get  time uses a struct that is defined here  and so I'm defining two timing strux a  start and an end  okay so this is just absolute  boilerplate setting up for taking timing  measurements okay and in this case  basically the high order part of the  struct tells the seconds the low order  part tells the nanoseconds and then  we're gonna loop over arrays of  increasing length okay and then what  we're going to do is fill them up I know  I forgot the fill and then we're gonna  measure how much time what the time is  just before we sort then we're gonna  sort and then we're gonna measure the  time after sorting okay and then we  compute the difference and figure out  what the elapsed time is print that out  and then we do it again for a little bit  larger array  okay so is that clear what's going on so  we're just sorting a bunch of numbers  that we're sorting some bigger ones  sorting some bigger ones so we can see  what the growth of the sorting routine  should be okay  that people have a pretty good  understanding of what the code does okay  so so what do we expect to see what's  this curve gonna look like what are some  properties of it  yep  alright so micro is then login but  certainly gonna grow right right it  should go should be up into the right in  fact one rule if you ever get into  marketing is that all graphs must go up  into the right if they're going down  into the right then your your company's  in trouble ok so it shouldn't be going  up to the right and it should follow for  example n low again if it's an N log n  sort which is what this one was I think  he was in this case timing emerged sort  I should go up into the right and should  follow n log in or whatever ok so let's  see what actually happened when he took  the measurements and these are actually  his this is actually his data from gosh  this must have been been 20 years ago or  something here's what it looked like  ok so the the the blue X's there are the  run times okay and then through that  we've plotted two curves one which is  the best fit to order n log n growth and  the best fit to order n growth you  notice that for even though we're going  up to 4 million here there's not that  much difference between n log N and  order n ok you can see it mostly down in  the tails definitely the N log n follows  a little bit better but but really log  in is is is pretty pretty small already  but wow those measured times okay so if  you look they're points way up here  right really slow and basically it's  going  it starts out it goes you know slow a  little bit and then it gets a little bit  worse and then a little bit worse and a  little bit worse notice also that the  bumps are getting closer and closer  together what is going on why like I  don't know about you but I thought the  data would follow you know the  green dots reasonably closely okay but  you can see it doesn't okay it's always  good to have a model for what you think  is going on because then when you  because some people will just take  numbers they'll say here's my numbers  for my for my that I've measured and if  you don't actually have a model for what  those numbers mean you're probably  fooling yourself you're more likely to  have made some sort of error or there's  something going on that you're not  observing or whatever if you don't  actually you know have a model for what  you think it should be going on so  what's going on here who can suggest  that hypothesis for for for what is  going on so he took these numbers on his  laptop by the way  what do you suppose is happening here  some ideas yeah  maybe it doesn't fit in the cash what  would you expect to happen if things  didn't fit in the cash  yeah you sort of think that would go  along right it would sort of go along  and then it would jump right so  interesting you know issue but that  doesn't seem to be what's happening  there it's going op things going back  down it's going up and going back down  right rollercoaster what other ideas are  there good idea good idea let's think a  little bit about what's going on in the  machine what are some other good ideas  or bad ideas let's eliminate some yeah  they're not powers of two these are not  powers of two right because they're  getting closer and closer as we get  bigger and bigger yeah so you're right  it doesn't have to do it's not  correlated with powers of two weird  because sometimes things are alignment  issues and we'll talk more about that  you know come up we want to talk about  caching after the quiz everybody knows  there's a quiz next time especially all  of you who aren't here okay so what else  what else might be going on in the  machine here okay because this is  reality this is what happens when you  take measurements so we're being very  nice to we're giving you AWS run we have  done everything we can to make sure  those numbers come out clean and  beautiful and untouched you know there  they are that is quality quality  measurements we're taking for you okay  but if you had to do it yourself that's  what this lecture in part is about yeah  [Music]  so you think that that there may be  something having to do with the the cash  but I'm going through each time and I'm  refilling the array each time so so  they're kind of starting from a clean  slate similar to clean slate each time  what else is going on in the machine  here yeah yeah there could be totally  unrelated stuff running right URI you're  not you know when you're running you  might have demons you might have all  kinds of things and so forth so he  thought of that okay and he shut down  everything he possibly could and this is  what he got still okay so but that's a  great idea because often there's some  external things going on in this case  it's called multi-tenancy there's more  than one thing using the computer at a  time okay good idea  but happens not to be the one it could  be precision with the timing yeah  sometimes there can be issues there but  this was not this was not a precision  issue this was not a pretty he could  have used a really dumb timer and gotten  something very similar to this what else  is going on on your machine yeah  [Music]  yeah maybes checking for updates that's  once again some some external things no  it wasn't checking for updates wasn't  checking for updates okay what is going  on here what is going on let's have some  more ideas what other things might  disrupt measurements yeah  yeah so so this could be but wow this is  so that would sort of if it was like  quicksort  this was actually merge sort he whose  timing so there's no randomization but  but even that if it were quicksort it  sort of be at random that things would  tend to take longer right rather than  following this crazy pattern what is  causing that crazy pattern ya know  because the random fill is done outside  the timer okay so we we filled and then  we you know each time through the loop  we fill and then we start the timer and  then do it new we take the measurements  so forth yeah maybe it's a Liva it's not  allocating memory but that's an  interesting idea because sometimes you  have things going on where you think  things are happening right away but the  system is being clever and and delaying  it and so you end up paying for it at  some later time and that could possibly  create something turns out not to be  what's going on here  okay well so what's happening here is  that the machine is changing the clock  frequency why is the machine changing  the clock frequency your laptop's change  the clock rate the systems that we have  they changed a lot clock frequency why  did they change it because the laptop is  getting hot so what are we doing we're  running something computational right  and the smaller ones okay we get a lot  of those done until it starts heating up  and so it slows down the system clock to  save power okay and then what happens it  slows it down a little bit cools off a  little bit starts to speed up again okay  and then we run longer and why are these  things getting closer and closer  together  yeah  yeah it takes longer and longer to run  the sorts so you're going to see the  effect you know closer in in an interval  here even if it happened to be equal in  time okay even if it's equal in time  we're doing bigger and bigger problems  so it's going this is this is nuts right  like you know we want to take  measurements we want to know whether the  software is faster what are you supposed  to do so here if you just took a  measurement and said look this is the  time that it takes me to run this code  you would be hugely missing the boat if  you were taking one of those high  measurements you compare a to B right  you run a first then you run B B is  slower oh well that's because when you  ran a it heated up the processor so the  processor slowed it down okay so this  particular architectural feature is  called devfs dynamic frequency and  voltage scaling it's a technique to  reduce power by adjusting the clock  frequency and so apply voltage to  transistors okay so the idea is that it  if the chip ends up getting too hot or  in the case of laptops often if you want  to conserve battery it chooses to slow  down the clock okay and the second thing  it can do is reduce the voltage if the  frequency is reduced okay so when you're  actually running slower you can actually  get the same reliability of switching  with a lower voltage okay at a higher  voltage sorry at a higher clock  frequencies you need enough voltage to  make sure those electrons are scooting  across the transistor junctions fast  enough so the the basic power law that  the electrical engineers use anybody  here an electrical engineer  okay there's good stuff in EEE let me  tell you okay so those of you are too  embarrassed to raise your hands okay I  support EE okay so power goes has CV  squared F okay where C is what's called  the dynamic capacitance there's actually  another term which is the static  capacitance which doesn't have to deal  with frequency or whatever but but for  the for the dynamic power its CV squared  F it's the dynamic capacitance it's  roughly the area of the circuitry times  how many bits are moving so if if bits  don't move they don't consume power for  for dynamic power okay and then V is the  supply voltage and then F is the clock  frequency so if you can reduce the  frequency and voltage you get a cubic  reduction in power and also in heat so  you can see why you know who thinks  their battery doesn't last long enough  yeah okay right wouldn't be nice if it  lasted a month okay so you can see why  they're motivated to play this game okay  to save the battery or to run things as  hot as they can but if it gets too hot  we'll just back off okay but for  performance measurement this is this is  basically a nightmare it wreaks havoc so  the topic of today's lecture is how can  one reliably measure the performance of  software when we have stuff like this  going on in our system now the good  thing about something like devfs is  there's actually ways of shutting it off  and taking measurements okay but you  know it's you probably pay some  performance for doing that okay because  they can basically who's ever worked on  an overclocked system okay some some of  you game players right you know so part  of the game that the idea  they're when they're overclocking it is  let's let's keep you know get things  cool and so forth and there's lots of  games like this they're played in  overclock systems so so I'm going to  talk about sort of three topics today  one is about quiescing systems which is  making them quiet enough that we can  take good measurements getting rid of  some of the noise and the second I'm  going to talk about some tools for  measuring software performance and then  we'll talk a bit about performance  modeling okay so I hope this is a little  motivational boy you know it's like you  think you've seen it all sometimes okay  and then somebody presents you with a  puzzle and it's like what's going on  okay and that takes a little detective  work to figure out that that's actually  what's going on  so we'll start first about quiescing  systems who knows who  genichi Taguchi was anybody here know  he's very famous because he's the one  who made our automobile industry produce  reliable cars okay very very famous  fellow he did a lot of different things  I'm gonna point out one of the things  that he observed and it's it's in the  context of suppose you were an Olympic  pistol coach and you've got two shooters  a and B and you look at their targets  and here's a z' target so we hit the  bullseye good and if you add that up  he's got a pretty decent score okay then  you look at B well B didn't even hit the  bullseye and his score is like really  lousy who do you want on your team a or  b who do you want  yeah cuz it's easy to tell them what to  do you adjust the sight you teach them  to you know shoot down a little bit  lower to the left  okay there's it's easy to diagnose what  his problem is okay that's all you have  to do what do you do for a it's not  clear what you do okay  and so what what he did in the realm of  quality control was say look when we're  producing products before we try to fix  the reliability or whatever is of the  product increase the quality of the  product  let's first reliably produce whatever  we're producing so we get the same thing  every time because then we can go in and  see what the problem is and we can  adjust for it okay and so the idea is go  after the variance first okay go after  the spread try to get the spread as  small as possible try to figure out how  it is that you can isolate you know  produce the same thing even if it's not  very good okay and then go and try to  make change because if you try to make  changes while you have high variance you  don't even know if you're making  progress or not because it could be in  the noise of your measurements okay  now in the context of performance  engineering if you can reduce  variability you can do things like  compensate for systematic and random  measurement errors and you can also do  things like not run as many trials to  discover whether one program is better  than another there are a lot of  advantages to being able to have really  quiet things it's kind of obvious that  you should want to have reliable  measurements okay but but there's  actually as I say a theory behind it for  why it is that you want to have quiet  measurements and and what you need to  focus on now in our computer systems  there are lots of sources of variability  and some people came up with some of  them here let me mention just a few of  them here so there are things like  demons and background  jobs things that are running on your  your system that are helping the system  to do whatever it's doing okay many of  them are unnecessary and those can be  running so for example maybe you set up  a cron job you know to you know do  something for you every night and that  happens to be just when you happen to be  be making some measurements well that's  not really good because you're now  sharing the resource ok interrupts ok  something comes in I had one time where  we were measuring stuff on the  connection machine cm5 which in its day  was the world's most powerful computer  1993 it was the top of the list of the  supercomputers and now this computer is  more powerful it cost 43 million dollars  or something ok I mean Moore's law has  really made a difference and we were we  were measuring the performance of  something and we kept getting these  anomalous results and eventually we  tracked it down it took us a lot of work  but we tracked it down the graduate  student who was running it while he was  running it he was moving the mouse  around and when you moved the mouse it  was interrupting 200 times a second to  deal with the interrupts that was  interfering with our measurements ok  true story true story okay so so  interrupts can have a so it's like oh ok  and we you know we disconnected the  network even ok so that we wouldn't get  external things and we're still getting  this thing what is going on ok and  eventually we tracked it down to to the  fact that he was just sitting there I  Lehman so it's like started up and then  hands off there's like nobody move ok  code and data alignment where the code  wise affects its performance if a code  goes across 2 cache lines versus  completely within one cache line that  can have a difference depending upon  what scan  flick ting in the cache okay or if you  go across page boundaries it can have a  very big difference for example in the  translation lookaside buffer or TLB you  may be going for different entries there  may be different different things there  so code alignment can make a difference  in the in the what you're doing  thread placement okay which well you  know if you have a multi-core machine  which core is it running it turns out  the system likes to use core zero for a  lot of its stuff so if you're going to  take reliable measurements don't run on  core zero okay runtime scheduler okay  the fact that for example we have a  randomized scheduler which means that  there's random numbers going on so  you're going to try to take measurements  in the midst of all this randomness a  hyper threading hyper-threading is where  they take one core it's also called  symmetric  it's called symmetric multi threading or  no simultaneous multi-threading that's  what it's called simultaneous  multi-threading or hyper threading is  what Intel calls it what that is is they  have one functional unit okay and then  they run two instruction streams through  it at the same time each with its own  set of registers but using the same  functional units and they get it  basically a 20% speed up or something  from having it looks it looks from the  software point of view like you've got  two processors but really it only gives  you one point two processors okay so  it's one of these things that by the way  if you start counting up how many  processors and saying work over the  number of processors you say well how  many processors do I have well you may  have just a hyper threaded processor  rather than a real processor okay so one  thing for example in the in the cloud  system that you folks are using we turn  off hyper threading okay so that we can  get more reliable measurements and so we  can look at the measurements as a  function of the number of processors  multi-tenancy and this is particularly  important in the cloud if you're in the  cloud there are other people using the  system they can  end up using lots of resources that  maybe you want like cash okay and they  can end up using network traffic that  may affect you and such I'm actually  quite amazed at how well Amazon does in  AWS in making it so that that stuff  doesn't affect you very much  okay our numbers show that they're in  fact they're definitely the leader right  now in in having you know repeatable  measurements you know compared to all  the cloud providers we talked about  devfs there's also another one called  turbo boost  so turbo boost looks to see how many how  many jobs are actually running on the  multi-core if there's only one job  running on the multi-core increases the  clock frequency for that job or if  there's just a few as soon as another  one comes in it slows things back down  so tries to give a boost when you're in  executing serial codes that says well  I'm not I'm not  you know generating heat from all the  cores I just have to generate it from  the one that I've got I just have to you  know I can afford to have one of them  run hotter network traffic and there's  by the way a bunch of other ones we're  going to talk about a few of these but  first let me tell you what the impact of  quiescing a system is so this is joint  work that that I did in my group with  Tim kayler so we wrote a silk program to  count the primes in an interval and we  ran on a see for instance 18 core  similar what you're running with we had  2-way hyper threading on turbo boost was  on we had 18 silk workers with a hundred  runs each about one second okay and what  I've plotted here is how much is the  percent that that each run was we  basically ran a hundred runs and then I  sorted them from smallest run to largest  and I normalize them to whatever the  minimum one was so that each each thing  here is how percentage above the minimum  so you can see that the slowest run was  almost 25 percent you know slower than  the fastest  okay so you can see sort of what the  impact is so 25% if you're trying to  produce a code by getting a 3%  improvement you know 30 times you can't  you know there's so much noise here it's  very hard for you to figure out that  you're 3% faster okay so if you quiesce  the system okay same thing we turn  hyper-threading off turn to burst off  and we quieted all the demons and and  and so forth then out of a hundred runs  we got essentially the same value every  single time except for three times and  notice that the scale here is changed  the scale here is now point less than  0.8 percent less than 1% slower so this  says that hey if I took a couple of  measurements I'm very likely to hit  something that is that is the you know  the the real running time which tends to  be for this what the what the minimum is  here okay because all the other stuff is  noise is just adding in okay makes sense  so it is possible to quiesce a system  it's not that hard okay it just takes a  small matter of you know programming and  systems work so here are some some tips  on how you can if you wanted to do this  for your laptop for example make sure no  other jobs are running shutdown demons  and cron jobs disconnect the network  don't fiddle with the mouse okay for  serial jobs don't run on core 0 where  interrupt handlers are usually run to  him - threading off turn off DFS F devfs  turn off turbo boost use task set to pin  workers to course of tasks as utility  that says I want you to run this thread  on this core and don't  let the operating system bounce it  around okay so normally operating system  Maps threads two cores the workers the  silk workers two cores in any way that  it feels like this says no I want you to  have it exactly here so that when I run  something else is exactly the same thing  and so forth and we've already done a  lot of this for you for AWS run okay by  the way there is no way of getting a  completely deterministic result out of  running on modern hardware does anybody  know why I mean if I have a serial  program and it's to terminus stick  serial program and I set it up and I  reboot the system so it's got exactly  the same content etc I still can get  non-deterministic results does anybody  know why yeah  no you can turn that off so you can make  sure that so they actually do randomized  address space for security reasons but  when you run under the debugger and so  for they tend to turn that off so that  you can get repeatability for debugging  purposes yeah no those are generally  well to a first-order these so now those  are deterministic but there is something  in the hardware which is  non-deterministic does anybody know what  is happen to know can guess what it is  something the hardware isn't  non-deterministic yeah disk access would  be but if I'm just running something in  core I'm not using disk I'm just going  to run it turns out non-deterministic  even though I'm just running everything  inside with you no ordinary DRAM memory  and so forth yeah no the out of order  execution is all deterministic there's  no randomization in there okay yeah  branch prediction is all is all  deterministic algorithms all  deterministic yeah  system clock signals very regular very  regular if you've turned off you know  gvfs and stuff like that yep  so the Linux schedule is in fact a  deterministic algorithm and if you're  just running on one core the scheduler  never comes into it okay so there is one  source of non-determinism and that is  memory errors so there's a chance that  one of your memory when you're accessing  the DRAM that an alpha particle collided  with one of the bits and flipped it and  there's Hardware in there to do error  correction but it takes an extra cycle  to do it so if it if it reads the memory  location discovers that there is a that  there is a an error in what it read then  it performs the correction and then you  get it and that's something it's  completely non deterministic because  it's alpha particles coming from outer  space and you know space aliens you know  messing with your system yeah no  actually it there's actually was a and  now most cache is covered by error  correction but the most likely if you  look the the biggest memory is the DRAM  so that's the most likely source of  these things but yeah there's so so in  any case I want to now just talk about a  few things which just some examples of  things that that might come up for you  so I'm these are ones that I'd mentioned  but I just want to go through them a  little bit more depth so one of the  things is code alignment can make a  difference okay so what happens is you  know you have your program and you make  a change that you think is improving  your program but what happens let's say  is it causes you know an extra bite to  be put into the code okay you know so  maybe you know the compiler is really  smart you made some little change then  everything that follows it gets shifted  down and so the cache alignment issues  can be completely different  something can go across a page boundary  that didn't use to go across the page  boundary and that can have a big impact  on your performance this is like yikes  right this is like yikes how am I  supposed to you know it's like maybe we  just pack up on performance engineering  right this is okay so everybody gets  what the issue is there so you know in  this case I inserted one bite  well everything after that it's all  linear in memory would would change  here's another one that's even more  insidious okay if you change the order  in which the dot o files appear on the  linker command line that can actually  have a bigger effect than going between  - OH - and - o3 okay you know you when  you compile you can buy all this order  versus this order okay you get can have  a actually quite a big big difference  yike okay so what do you do well first  of all one of the things I'm glad to say  is that the compiler people have  recognized this and it's the situation  is not as dire as it was years ago what  they do now very often is do a lot of  alignment already so for example it's  common for compilers to produce to start  every function on the first word of a  cache line okay that way when things get  slid down you might pop from one cache  line to the next okay for where it  starts but you're not going to you know  affect the you know where you lie on the  cache line which can make a difference  by the way in branch predictors and  things like that  okay and so that helps that really  quiets a lot of things and in fact they  give you some directives so LLVM has  these switches as far as I could tell  the first one which is a line all  functions I think I was unable to test  this in advance but I suspect that it's  actually already doing  this and this is actually in no ah'd  because it's already aligning all  functions okay I don't know that for a  fact but you can give the switch anyway  which will help which will help if if if  I'm lying okay  so that it forces the alignment all  functions so all functions start on a  cache line and that way if you change  one function it's unlikely to check it  won't change the cache alignment of  another function it'll only change the  cache alignment of the function that  you're messing with you can also ask it  to align all blocks in the function so  remember that in LLVM we have these  basic blocks these pieces of serial code  with with you know with links you know  the the basic blocks so what you what it  will do is force every one of those to  be on a boundary but of course what that  means is now you've got a jump between  these codes even if it were going to be  the next instruction or put in a bunch  of no-ops so that can substantially  increase the size of your that can I  substantially increase the size of your  binary and it can slow you down but on  the other hand you'll get very reliable  results from then on because every block  is now cache aligned okay  probably more practical is too long a  line is to force the alignment of all  blocks that have no fall through  predecessors that is you don't have to  add no so this basically reduces it to  the ones that are usually causing the  trouble  so a line of code is more likely to  avoid performance anomalies but it can  also sometimes be slower and so one of  the questions as well which you know  what which matters to you and your  particular particular thing okay  here's one that I love so the example  that I gave before of the order of  linking we have that as a reading  assignment by the way on the on the web  that paper and here's another result  from from this paper by Mike Mitka wits  and  and his colleagues the if they have a  wonderful paper called producing wrong  data without doing anything obviously  wrong okay so one of the examples they  give me swords the ordering of things  saying that look that actually had an  impact comparative between Oh two and O  three the program's name can affect its  speed and here's why the executables  name ends up in an environment variable  okay so when you start up a program it  puts it in environment bill the  environment variables it turns out end  up on the call stack okay so you can  find out from the program what's the  name of the program I was invoked with  okay the length of the name affects the  stack alignment if you have a longer  name they put longer you know a longer  piece there and so if the data happens  to go across page boundaries some  critical piece of data goes on to pages  rather than one that could have a big  impact on your performance okay and  there are other date alignment problems  that can can arise okay so yeah this is  kind of nasty stuff so as I say what  we've done it we've given you a very  quest system we have done everything we  can to make it so that you're you know  that when you submit stuff in this class  its measured but as we get along in the  semester and you're doing more  sophisticated stuff some of this is  going to fall increasingly to you to do  things that make it robust okay okay  let's move on we talked about quiescing  now let's talk a little about about  tools for measuring software performance  so I did a little thinking and I figured  that there's actually like about five  different ways I know of measuring the  performance of a program so one is you  can measure the program externally you  can run the time command if you say time  and then give a shell command it will  run the shell command and then tell you  at the end how long it took okay you can  the program okay that means to actually  put timing calls into the program so you  can use things like clock get time which  I recommend and we'll talk a little bit  about these other to get time of day and  already TSC and you can either do this  by hand where you say I'm going to time  something myself or it turns out you can  have compiler support for doing taking  timing measurements but what that  involves is changing the program in  order to put these timing calls in and  of course you have the effect that if  your perturbing the you know if you're  putting these timing calls in you can be  changing the timing so you've got that  problem to worry about as well  one of the ones is you can interrupt the  program one of the cheapest ways you can  do is just basically take gdb so you  start running your program gdb  you know you run it under gdb and then  you type control C and then you look  where is the program and you do that a  few times you say hey the same routine  is always where the code is running oh  that must be where it's spending all its  time okay that's actually been put into  a thing called poor-man's profiler and  is and then you can actually this is  essentially the method that Jeep refuses  okay we'll talk about that in a little  bit you can exploit hardware and  operating system support so they're a  bunch of hardware counters that the  operating system and hardware support so  that for example perfuses and so you  folks are familiar with the perf toolset  those are basically you know using  hardware and operating system support  and you can also simulate the program  you can run it as a simulation and then  you really can go in and do anything you  want to understand but of course it's  much much slower and you hope that your  simulator is modeling everything that's  of relevance to you maybe it's not not  only something that turns out to be  relevant for you so I'm going to talk it  give some examples of each of these five  things okay so let's start with  the time command so it can measure a  elapsed time user time and system time  for an entire program does anybody know  what these three terms mean elapsed time  user time and system time sure what we  call wall clock time good it's in the  kernel working on your stuff as opposed  to somebody else's okay so that's that's  exactly right so when you run the time  command you get some numbers like this  where it says here's the real time  here's the user time here's the system  time now you might think that the user  time in the system time should add up to  your total time uh doesn't work that way  and part of the reason is that it made  the processor may actually not be  working on your code you may be context  switched out and something else is in  okay so so in any case we have those  those three types of things wall clock  time the amount of processor time spent  in the user mode code within your  process and the system's time which is  the time spent in the kernel but within  your process okay for example satisfying  system calls and such now the the the  timing call that I recommend you use and  that was used in the example that I gave  is clock get time in particularly there  are a bunch of options to that and the  one that I strongly recommend is clock  monotonic okay and it takes about  eighty-three nanoseconds to to actually  read what the time is that's about two  orders of magnitude faster than a system  call and  one of the things about it is this is  such a funny thing to have to say is it  guarantees never to run backwards okay  turns out the other timers can run  backwards you can take measurements and  discover their negative okay this one  does not run backwards part of it is  because some of the other timers do  things like oh you know there's this  National Standards thing that  periodically your your computer goes out  to find out what the real time is and it  resets its clock to be consistent with  whatever the global clock is okay and  that will cause the clock to be updated  in a non-standard way okay and where  we're suddenly you lost some time or  gained some time okay so this is really  I you know the best nation is the only  unfortunate thing about this is that it  is non-deterministic how long it takes  and let me explain a little bit what's  going on in this so what happens is it  takes a measurement it has to take two  measurements to figure out what the  elapsed time is to find out what the  actual time is okay can't just take one  measurement because it may have been  swapped out and the the colonel helps  support in userspace  something that says here's the total  amount of time you've spent up until you  started your time slice so when you read  that you have to read those two values  well how do you know that you don't have  an atom isset Eve I elation you read one  of the values you got switched out you  get switched back in now you have a new  value for the other one so the way it  does it is it reads this register so it  reads what the operating system has kept  is its cumulative it reads the the the  clock and then it reads that register  again and if those two things differ it  knows there's been a switch in there if  they're the same knows there isn't and  the the number that it can take is  reliable so in that kind of case it will  actually take  two measurements in or more than one  measurement I mean could you could do it  again and it would you know you could  have another context switch and you  could do it again have another but this  thing is generally pretty fast so um and  on my laptop it takes about 83  nanoseconds to to run there's there's a  lot of people say well why don't I just  read the cycle counter that's actually  cheaper it runs in about 32 nanoseconds  and that you can do with the are DTSC  you know read the time stamp counter  instruction and you can do it yourself  by using a you know a built in assembly  and what it does is it returns to how  many clock cycles since the boot and it  rhymes in about 32 nanoseconds but why  not use this well one thing is that RTI  DTSC may give different answers on  different cores on the same machine so  the cycle counters on a on a processor  by processor basis okay sometimes TSC  runs backwards okay as I mentioned okay  and also the counter may not progress at  a constant speed so remember that the  time you know between you know the the  system is possibly slowing and speeding  up the counters and so forth and  converting clock cycles for that reason  two seconds can be very tricky so I  recommend you stay away from this faster  counter this faster timer okay the other  one is don't use get time of day that's  the one most people know that gives you  you know microsecond precision it's not  actually microsecond accurate but it  gives you microsecond precision because  it has similar problems whereas this  this particular the you know the clock  get time monotonic has been very well  engineered in my opinion to give good  reliable numbers at a reasonable  cost okay any questions about that about  taking measurements and what to use this  stuff by the way over time it's going to  change okay you know people are going to  come up with better ways or worse ways  or they'll say we're not going to  support that anymore or what-have-you  and then if you're out there as an  engineer you're going to be on your own  okay hopefully you know what some of the  issues here are and you're prepared to  be on your own yeah  hmm  so when it does that it aggregates it  and the operating system has to do it  those numbers actually are very are  relatively coarse grained you cannot  time something that's very short with  time for example with it with the time  command okay in general my experience is  you should even with something like you  can get very fairly fine grained  measurements with with the clock get  time you can get fairly good  measurements there but unless you are  aggregating unless you're running code  that's running around a second certainly  at least a tenth of a second okay if  you're not running things that are that  long you run the risk that you've got  really really bad  you know you get unlucky let me point  that out in in this particular example  so so here's the interrupting strategy  which we talked about briefly this is  where I just control C and at random  intervals and you look at the stack and  say who needs a fancy profiler in fact  there are large companies who use this  for debugging  they are big big coats okay Facebook  comes to mind okay they actually use  this technique it is a really easy  worthwhile technique for figuring out  where time is being spent now there are  some other things the the poor man's  profiler and people have built things  like G prof and so forth to increment to  automate this strategy so you get this  information because then it  automatically AK units the looks at the  stack what's being executed puts that  into a call graph and so forth and  figures out you know where the time is  going but neither of those programs is  accurate if you don't obtain enough  samples and just to give you an example  G profs ampuls only 100 times per second  so if you're gonna use G prof for for  timing something that's only a second  long you only got one  samples how many samples is a hundred  that's not actually a real lot okay and  so you can get wildly inaccurate numbers  from interrupting okay but on the other  hand for a quick type of you know I use  G prof all the time right it's it's  quick we do control seeing all the time  it's really quick I don't have to  install anything I just I just you know  take a look and it gives me a first cut  at at what I want to do okay it all  depends you know you don't need all this  surgically precise tools all the time  sometimes a really dumb tool is adequate  for the job and a lot quicker to deal  with okay hardware counters so one of  the nice things that's happened in  recent years is that there's become  available library called live p FM 4  which is virtualizing all the hardware  counters so that you have access to them  with to all of these types of events on  a per process basis okay so normally  there's hardware counters but then you  switch to some other you know there's  context switching going on to some other  process then what happens to your  calendars they have to be saved they  have to be you know updated so anyway  live p FM 4 does all of that kind of  virtualization to make it so that the  counter you can view it as if it's your  own counter ok and proofs that for  example employees that there are a lot  of esoteric hardware counters and as I  say good luck and figuring out what they  all measure because they often are not  well documented okay a few of the  important ones are well documented but  but most of them are very poorly  documented exactly what it does a really  good example was I we had somebody who  was looking at cache misses okay to  figure out how much bandwidth so last  level cache ok l3 cache misses to count  how much data was being trans  heard from from DRAM okay and they were  getting curious numbers that didn't seem  to measure up okay and it's like wait a  minute have a miss moves from moose from  from DRAM into the onto the chip why is  that not counting how much stuff is  being moved if I count up how many cache  misses times how many bytes in a cache  line which is what machines we're using  how many how many bytes in a cache line  on the machines we're using okay  sixty-four okay gotcha oh you guys are  okay 64 4 but not every machine has that  but anyway so why was this not measuring  how much stuff is being moved I'll give  you a hint it used to measure how much  stuff was being moved but then those  architects they are such pesky clever  people and they put in a great feature  prefetching okay they put in prefetching  they there's things that fetch it and it  doesn't update that counter okay so if  you want you have to count how many  prefetch incidents you have as well okay  and so you can often cobble this  together but good luck figuring out what  some of these two okay also watch out  you may think that the tools let you  measure a lot of counters if you want  but if you read the fine print it turns  out that if you do more than four or  five it starts essentially time sharing  the available counting bandwidth that it  has and it's not it's actually just  doing something statistical rather than  actually counting them so you can't  count more than like four or five  actually four or five I think is a high  number  but somebody I know Wells who knows this  stuff said said four or five is probably  what it is today okay so that's hardware  counters so hardware counter is a good  good technique next one is simulators so  things like cash grin usually rush run  much slower than real time but what's  great about simulators is you can get  repeatable numbers out of them okay you  know you run the code you run it again  if you've setup everything right you can  get and you can see what's going on  inside the downside is that they don't  necessarily it's slower and it doesn't  necessarily model everything going on in  the cache but for things like cache  misses this is a great tool okay to just  figure out what's the fundamental cache  miss and we'll talk about that when we  talk about caches in the next couple  weeks and if you want a particular  statistic in principle you can go in and  if it's an open source simulator like  cache grinned is you can collect it  without perturbing the simulation so any  question about you know these ways of  collecting measurements there are a  whole bunch of ways of doing they all  have pros and cons it all can be useful  in a given context they all have some  some flaws okay a really good strategy  I'll talk about this later is  triangulation I never take one  measurement and believe it I always want  to take at least two measurements okay  in different ways and make sure they're  telling me the same story  okay triangulation if there's a  discrepancy then I want to know what's  causing the discrepancy but I never  trust one number and I never trust any  numbers without having a model for what  I think is is coming up and in fact  that's what we're gonna talk about next  is performance modeling so any questions  about measurements and such so isn't it  good we have AWS run I mean right it's  like number comes back it's the number  and it's actually a pretty good number  okay we worked very hard to make that a  pretty good number  okay so performance modeling so yeah we  cover a lot of stuff in this class as I  think some of you have started to notice  and but really performance software  performance engineering is pretty simple  process okay you you take a program that  you want to make go fast program a you  make a change to program a to produce a  hopefully faster program a prime you  measure the performance of program a  prime if a prime beats a then you set a  equals a prime and if a is still not  fast enough you repeat the process okay  right but that's what we that's  basically what you're doing right it's  pretty pretty simple okay  and as should be apparent if you can't  measure performance reliably it's hard  to make many small changes that add up  because it's hard to tell whether a  beats  a prime sorry whether a prime beats a  right it's hard to tell okay and so as a  consequence what we want to do is want  to model performance so that we're in a  position to draw accurate conclusions so  we want to do things like drive the the  variability of measurement down to zero  and we want to do things like figure out  ways of using statistics okay to give us  more accurate picture of what's going on  then maybe what is appearance and that's  basically what we're going to talk about  for a little bit part of what I'm going  to do is going to talk about statistics  how many people have had a statistics  class or a machine learning class or  something read ill dealt with statistics  beyond probability I mean I mean real  statistics right you know sample sample  means and things like that so a few of  you okay the basics of what you need to  know you can find from Wikipedia or math  world or someplace like that  and I'm not gonna try to teach you  statistics 101 but I point you in some  directions and give you some pointers so  some tools you can do okay so here's a  puzzle suppose you measure the  performance of a deterministic program a  hundred times on a computer with some  interfering background noise what  statistic best represents the raw  performance of the software is that the  mean of the arithmetic mean of those  runs  is it the geometric mean of those runs  is it the median of the runs is that the  maximum of the runs is that the minimum  of the runs okay this is where it's  helpful if we had those you know those  clickers or whatever they have right you  know but we don't so so I'll ask people  to vote now I want everybody to vote  once again it doesn't matter if you're  right or wrong you can be right for the  wrong reasons you can be wrong but have  the idea right but it's fun when  everybody participates certainly more  fun for me when I see hands go up then  when I see people sitting there looking  bored okay okay how many people think  arithmetic mean would be a good way of  measuring the raw performance arithmetic  that's a that's the most common  statistic that we ever gather okay  what about geometric mean okay what  about median okay good what about  maximum one for maximum what about  minimum okay so it turns out actually  these are all good measures to take and  depends upon what you're doing with them  but turns out minimum is does the best  at noise rejection and that's because  you expect any measurements higher than  the minimum if it's a deterministic  program and so forth that's going to be  due to noise so if you're really  interested in knowing how long  fundamentally you're  Co takes on the underlying hardware when  there's other things going on taking the  minimum rejects it now you might say the  median also rejects noise but it doesn't  because because if the you know if you  view your your program as being it's  running time plus noise then the median  is going to give you some number that is  in the midst of the noise right it's  going to have some complement so minimum  is the only one that really rejects all  of them but they're all useful measures  in different contexts and there are ways  that you can use the mean and some of  these other ones as we'll talk about in  a minute to get information about making  decisions okay because the thing that  we're after is not necessarily always  the raw performance of the software  sometimes we're interested in whether a  beats B that's a different question than  how fast does this sort of fundamentally  go now  so there are a lot of different types of  summary statistics and there's lots of  reasons to pick different ones so for  example if you're interested in serving  as many requests as possible in a web  server for example you're going to be  looking at something like the CPU  utilization and taking the the  arithmetic mean to try to understand how  how those things add up if all tasks  have to be completed within 10  milliseconds then you're going to look  at you're looking at the total and  you're going to add it up and you're  going to be interested in making sure  that that each one is is small and  that's also what the mean does and  you're gonna be looking at wall clock  time if you wanted to ensure that most  requests are satisfied within a hundred  milliseconds you might be looking at the  90th percentile behavior okay and you'll  say yes I won't make every one but I  want 90% of the time I want to get it  there and I'll be using something like  wall clock time okay  there may be in in a lot of web  companies there's a thing called a  service level  Raymond this is what they should give  you for your telephone but they don't  tells you what kind of service you can  expect and if they don't meet that  service requirement then they haven't  lived up to it instead we buy these  phones and we get the service and we  just get whatever they decide to give us  right but if you're a big company you  insist that you get some kind of service  out of the people that your you're using  and so there that's typically some  weighted combination and you're using  sort of multiple things you might want  to fit into a machine with a hundred  megabytes of memory some sort of  embedded machine or whatever then you're  going to be interested in maximum of the  memory use so it's not all the  performances not all just time you might  want the least cost possible and you're  looking at things like energy use etc  here are the fastest biggest best  solutions you can see average comes up a  lot you know as one of the ways so I  wanted to though cover one particular  example which I find is the most common  place I see a misuse of summary  statistics and that's for when I'm  summarizing ratios so suppose I have two  programs a and B and I run the four  trials normally you'd run a lot more but  I wanted to fit on a slide okay and  program a you know on whatever trial one  was took took nine seconds say program  two on trial two it took eight two and  ten and program about three to twenty  and - okay so I can compute the mean for  each of those so the mean of the program  a is eight point two five and the mean  of program B is four point seven five  over those four benchmarks okay I can  also take a look at how much is a  winning sorry its be winning over a okay  and so if I take the ratios I then get  three four  one tenth and five for a mean of three  point one three  it's actually three point one two five  but I'm only keeping things to two to  two digits okay and so if I was asked to  summarize this I could perhaps conclude  that program three is more than three  times better than program a based on  these statistics okay  but there's a bug in that reasoning can  anybody see the bug this is wrong  [Music]  it doesn't make sense to take the  arithmetic mean of a bunch of ratios why  is that yeah one thing here is that you  can see is that notice that the mean of  the ratio is not the same as the ratio  of the mean that should be some  suspicious why am i comparing should I  not be comparing the ratio of the mean  or sort of a comparing the mean of the  ratios  okay so so that's not particularly good  another thing is to suppose I take a  look at the ratio B over a and I take  the arithmetic mean then what I discover  is that a is better by a factor of  almost three than B so clearly taking  the the average of the of the of the  means is you know is assuming the  average of the ratios there's something  wrong with that right and in particular  as I say the ratio of the means is not  the mean of the ratios okay and then  your intuition is spot-on  okay I suppose instead I compute the  geometric mean so geometric mean is  basically like taking the average of the  logs the arithmetic mean of the logs so  you're basically taking the product and  taking the nth root of the product okay  and I've computed that for these things  and now I've taken the arithmetic mean  of the run times because that makes  sense that's kind of how an average over  these things how long did things take  and now I look at a over B and B over a  and I get the same the same thing and  it's in fact the case that the ratio of  the means is the mean of the ratios okay  so when you look at ratios there's  another place that comes up where you  look at rates and I've seen people look  at rates and it turns out when you're  looking at rates often it's the harmonic  mean that you want in order to preserve  these good mathematical properties and  make sure that  like it's really bad if I you know if I  do this thing and I say look how much  better a is than B and somebody if I  flip the ratio B would be better than a  right that's really suspect if if if the  identity of whether you're a or B and  which one was the numerator which in the  denominator you know had an impact right  that would be that would be like really  suspect okay so this is the kind of  thing when you're thinking about how  you're reporting things and so forth you  want to be careful about when you're  aggregating things to make sure that you  have the basic mathematical properties  met okay and what's nice is like you  know one divided by one point five seven  here is in fact zero point six four okay  so it didn't matter which way I took the  ratio I got the same answer okay one of  them is a so B is better that by a  factor of about one point six something  like that it was a little bit better  it's not three times better and it for  sure isn't the case that a is better  than B by a factor of three okay good  any questions about that this is kind of  good okay suppose I want to compare two  programs a and B which is faster and I  have a slightly noisy computer on which  to measure their performance what's a  good strategy  what's a good strategy for comparing  them figuring out whether a is better  than B or B better than a sure  so you could take sorry so you're saying  I'm gonna run multiple runs that's a  great idea okay we're gonna do multiple  runs and what am i doing for each of  these runs the low order statistic of  the like for example like minimum were  or 10% or something really low okay so I  take that I have one number for for  program a I have one number for program  B I ran them in times what else well am  i comparing am i comparing the am i  comparing the the two minimums I've got  there or am i comparing each one  I say oh I see you're saying take all  the measurements that go below your your  10% cheapest ones 10% cheapest ones and  then compare those okay that might give  you  so you're substantially doing noise  reduction is what you're saying and then  other than that you're comparing by  comparing means okay that's an  interesting methodology you probably get  something reasonable  I would think but you probably couldn't  prove anything about it but that's an  interesting idea yeah that's that's an  interesting idea what else yeah  good so you're saying do a bunch of  head-to-head runs and mark just who wins  over those things and why is that a why  is that a good so which so one wins more  than the other or the other ones more  than the one what what good is that  yeah so this is actually a very good  strategy okay and actually has some  statistical muscle behind it okay so  what you can do is do in head-to-head  comparisons between a and B okay so in  both these examples is like yeah we  better run it a few times okay and  suppose that a wins more frequency more  frequently okay so now what we want to  do in the statistics where they call the  null hypothesis is the that B beats a so  even though we see a B's be the null  hypothesis that were wrong and that a  beats the B beats a and what we then can  calculate is what they call the p-value  which is the probability we'd observe  that be that a beats be more often than  we did okay so for example for each of  these two imagine that in in the worst  case let's just say that they were equal  in performance and all we're seeing is  the noise then I would expect that I  would get about an even number of things  and so the further that I deviate from  them being even especially once I you  know that distribution is essentially a  Bernoulli distribution the further I get  out or a t-distribution if you have  small numbers as I get further out from  away from the mean by what the variance  is of just flipping coins in that case I  can calculate what's the probability  that I'm seeing something that would be  that extreme and that gives me a good  reason to reject the null hypothesis if  it turns out that deviates by Allah  okay so this is very standard stuff in  the social sciences who's had a course  in this kind of stuff and testing null  hypothesis by ology and so forth and you  remember anything from it or just  vaguely yeah yeah okay that's fine you  know one of the things I have found in  life is that you know I don't remember  anything I learned in college to speak  of right okay what I had to do is  relearn it but I relearned it a lot  faster than when I was in college you  know it and so that's part of this is  also so when you see it again okay now  you know how to go about learning this  so so I would once again you can look  this up on on like Wikipedia or whatever  if you want to use this methodology it's  a it's a good one for being able to  determine even in the presence of noise  what it does do is it doesn't tell you  what the raw performance was but it says  in that environment which one is  actually faster and that may actually be  a more relevant question because you're  not always going to have this completely  quiet system you may have a lot of noise  going on a system you'd like to know  which one is going to behave better in  what that actual system is and so this  methodology is pretty good  note that with a lot of noise we need  lots of trials the last thing that I  want to talk about which I won't is  fitting to a model and this is the issue  of sometimes you measure things but  you're interested in a derived  statistics such as such as in this case  gathering some time accounting and  instructions counting cast measures  counting time and asking okay what's the  you know what can i estimate is the  instruction time and what is the cache  time and to do that you do a least  squares approximation and there's once  again some statistics behind that I'm  not going to go over it here but you can  look in the in the notes there are a  bunch of issues with modeling which is  that you can over fit very easily if you  if you add more basis functions you will  fit the data better  and so how do you know if you're  overfitting answer is if you remove a  base to function it doesn't affect the  quality very much is the model  predictive I'm actually going to just  let you guys look at these because it's  it's pretty good is that I think they're  pretty self-explanatory  giant of science this is Lord Kelvin  what is Kelvin famous for besides Kelvin  okay  he was the Guru of measurement okay and  he said to measure is to know and that's  a good one and he said also if you  cannot measurement it you cannot improve  it  so both very apt sayings from the same  guy so there's a reason he's got a big  forehead I guess so anyway that's it  thanks very much and good luck on the  quiz on Tuesday  you
 

3
00:00:05,769 --> 00:00:08,019

 

4
00:00:08,019 --> 00:00:09,850

 

5
00:00:09,850 --> 00:00:10,930

 

6
00:00:10,930 --> 00:00:13,120

 

7
00:00:13,120 --> 00:00:15,160

 

8
00:00:15,160 --> 00:00:22,829

 

9
00:00:22,829 --> 00:00:28,900

 

10
00:00:28,900 --> 00:00:32,710

 

11
00:00:32,710 --> 00:00:36,069

 

12
00:00:36,069 --> 00:00:39,810

 

13
00:00:39,810 --> 00:00:41,979

 

14
00:00:41,979 --> 00:00:45,939

 

15
00:00:45,939 --> 00:00:49,060

 

16
00:00:49,060 --> 00:00:52,600

 

17
00:00:52,600 --> 00:00:55,619

 

18
00:00:55,619 --> 00:00:57,939

 

19
00:00:57,939 --> 00:00:59,829

 

20
00:00:59,829 --> 00:01:01,540

 

21
00:01:01,540 --> 00:01:04,869

 

22
00:01:04,869 --> 00:01:10,749

 

23
00:01:10,749 --> 00:01:13,210

 

24
00:01:13,210 --> 00:01:16,419

 

25
00:01:16,419 --> 00:01:20,050

 

26
00:01:20,050 --> 00:01:21,609

 

27
00:01:21,609 --> 00:01:25,240

 

28
00:01:25,240 --> 00:01:27,580

 

29
00:01:27,580 --> 00:01:30,130

 

30
00:01:30,130 --> 00:01:32,679

 

31
00:01:32,679 --> 00:01:39,670

 

32
00:01:39,670 --> 00:01:42,810

 

33
00:01:42,810 --> 00:01:46,840

 

34
00:01:46,840 --> 00:01:48,719

 

35
00:01:48,719 --> 00:01:52,060

 

36
00:01:52,060 --> 00:01:54,249

 

37
00:01:54,249 --> 00:01:57,279

 

38
00:01:57,279 --> 00:01:59,139

 

39
00:01:59,139 --> 00:02:01,300

 

40
00:02:01,300 --> 00:02:06,849

 

41
00:02:06,849 --> 00:02:09,310

 

42
00:02:09,310 --> 00:02:12,220

 

43
00:02:12,220 --> 00:02:15,460

 

44
00:02:15,460 --> 00:02:16,900

 

45
00:02:16,900 --> 00:02:19,390

 

46
00:02:19,390 --> 00:02:21,759

 

47
00:02:21,759 --> 00:02:23,890

 

48
00:02:23,890 --> 00:02:27,759

 

49
00:02:27,759 --> 00:02:30,250

 

50
00:02:30,250 --> 00:02:31,960

 

51
00:02:31,960 --> 00:02:34,090

 

52
00:02:34,090 --> 00:02:35,430

 

53
00:02:35,430 --> 00:02:38,050

 

54
00:02:38,050 --> 00:02:39,700

 

55
00:02:39,700 --> 00:02:41,110

 

56
00:02:41,110 --> 00:02:43,030

 

57
00:02:43,030 --> 00:02:45,730

 

58
00:02:45,730 --> 00:02:48,100

 

59
00:02:48,100 --> 00:02:51,070

 

60
00:02:51,070 --> 00:02:53,860

 

61
00:02:53,860 --> 00:03:00,820

 

62
00:03:00,820 --> 00:03:03,550

 

63
00:03:03,550 --> 00:03:12,040

 

64
00:03:12,040 --> 00:03:12,050

 

65
00:03:12,050 --> 00:03:17,150


66
00:03:17,150 --> 00:03:19,830

 

67
00:03:19,830 --> 00:03:21,690

 

68
00:03:21,690 --> 00:03:23,610

 

69
00:03:23,610 --> 00:03:26,160

 

70
00:03:26,160 --> 00:03:28,890

 

71
00:03:28,890 --> 00:03:31,650

 

72
00:03:31,650 --> 00:03:34,830

 

73
00:03:34,830 --> 00:03:38,850

 

74
00:03:38,850 --> 00:03:40,410

 

75
00:03:40,410 --> 00:03:42,060

 

76
00:03:42,060 --> 00:03:44,520

 

77
00:03:44,520 --> 00:03:47,580

 

78
00:03:47,580 --> 00:03:50,430

 

79
00:03:50,430 --> 00:03:54,120

 

80
00:03:54,120 --> 00:03:55,920

 

81
00:03:55,920 --> 00:03:57,180

 

82
00:03:57,180 --> 00:04:01,020

 

83
00:04:01,020 --> 00:04:03,479

 

84
00:04:03,479 --> 00:04:07,550

 

85
00:04:07,550 --> 00:04:12,720

 

86
00:04:12,720 --> 00:04:18,900

 

87
00:04:18,900 --> 00:04:21,060

 

88
00:04:21,060 --> 00:04:24,510

 

89
00:04:24,510 --> 00:04:26,700

 

90
00:04:26,700 --> 00:04:28,500

 

91
00:04:28,500 --> 00:04:32,040

 

92
00:04:32,040 --> 00:04:33,860

 

93
00:04:33,860 --> 00:04:38,190

 

94
00:04:38,190 --> 00:04:41,219

 

95
00:04:41,219 --> 00:04:44,790

 

96
00:04:44,790 --> 00:04:51,320

 

97
00:04:51,320 --> 00:04:55,920

 

98
00:04:55,920 --> 00:04:57,450

 

99
00:04:57,450 --> 00:05:02,190

 

100
00:05:02,190 --> 00:05:02,200

 

101
00:05:02,200 --> 00:05:02,670


102
00:05:02,670 --> 00:05:05,310

 

103
00:05:05,310 --> 00:05:07,080

 

104
00:05:07,080 --> 00:05:09,000

 

105
00:05:09,000 --> 00:05:10,890

 

106
00:05:10,890 --> 00:05:12,240

 

107
00:05:12,240 --> 00:05:20,430

 

108
00:05:20,430 --> 00:05:24,180

 

109
00:05:24,180 --> 00:05:27,420

 

110
00:05:27,420 --> 00:05:32,310

 

111
00:05:32,310 --> 00:05:35,550

 

112
00:05:35,550 --> 00:05:37,140

 

113
00:05:37,140 --> 00:05:38,790

 

114
00:05:38,790 --> 00:05:40,380

 

115
00:05:40,380 --> 00:05:42,270

 

116
00:05:42,270 --> 00:05:46,950

 

117
00:05:46,950 --> 00:05:49,110

 

118
00:05:49,110 --> 00:05:51,990

 

119
00:05:51,990 --> 00:05:54,870

 

120
00:05:54,870 --> 00:05:57,030

 

121
00:05:57,030 --> 00:05:58,110

 

122
00:05:58,110 --> 00:05:59,880

 

123
00:05:59,880 --> 00:06:02,820

 

124
00:06:02,820 --> 00:06:04,470

 

125
00:06:04,470 --> 00:06:06,600

 

126
00:06:06,600 --> 00:06:11,220

 

127
00:06:11,220 --> 00:06:18,450

 

128
00:06:18,450 --> 00:06:26,530

 

129
00:06:26,530 --> 00:06:34,080

 

130
00:06:34,080 --> 00:06:41,219

 

131
00:06:41,219 --> 00:06:43,360

 

132
00:06:43,360 --> 00:06:44,860

 

133
00:06:44,860 --> 00:06:47,400

 

134
00:06:47,400 --> 00:06:49,450

 

135
00:06:49,450 --> 00:06:51,249

 

136
00:06:51,249 --> 00:06:54,990

 

137
00:06:54,990 --> 00:06:58,390

 

138
00:06:58,390 --> 00:06:59,860

 

139
00:06:59,860 --> 00:07:01,840

 

140
00:07:01,840 --> 00:07:04,020

 

141
00:07:04,020 --> 00:07:08,830

 

142
00:07:08,830 --> 00:07:12,070

 

143
00:07:12,070 --> 00:07:13,420

 

144
00:07:13,420 --> 00:07:17,939

 

145
00:07:17,939 --> 00:07:28,520

 

146
00:07:28,520 --> 00:07:32,720

 

147
00:07:32,720 --> 00:07:35,270

 

148
00:07:35,270 --> 00:07:36,680

 

149
00:07:36,680 --> 00:07:38,930

 

150
00:07:38,930 --> 00:07:40,370

 

151
00:07:40,370 --> 00:07:42,950

 

152
00:07:42,950 --> 00:07:45,500

 

153
00:07:45,500 --> 00:07:46,970

 

154
00:07:46,970 --> 00:07:49,070

 

155
00:07:49,070 --> 00:07:51,470

 

156
00:07:51,470 --> 00:07:54,140

 

157
00:07:54,140 --> 00:08:00,530

 

158
00:08:00,530 --> 00:08:01,970

 

159
00:08:01,970 --> 00:08:08,300

 

160
00:08:08,300 --> 00:08:10,400

 

161
00:08:10,400 --> 00:08:13,610

 

162
00:08:13,610 --> 00:08:18,230

 

163
00:08:18,230 --> 00:08:20,030

 

164
00:08:20,030 --> 00:08:23,200

 

165
00:08:23,200 --> 00:08:29,960

 

166
00:08:29,960 --> 00:08:33,399

 

167
00:08:33,399 --> 00:08:35,930

 

168
00:08:35,930 --> 00:08:38,089

 

169
00:08:38,089 --> 00:08:42,490

 

170
00:08:42,490 --> 00:08:42,500

 

171
00:08:42,500 --> 00:08:52,400


172
00:08:52,400 --> 00:08:55,040

 

173
00:08:55,040 --> 00:08:57,980

 

174
00:08:57,980 --> 00:08:59,690

 

175
00:08:59,690 --> 00:09:02,600

 

176
00:09:02,600 --> 00:09:03,950

 

177
00:09:03,950 --> 00:09:10,030

 

178
00:09:10,030 --> 00:09:12,140

 

179
00:09:12,140 --> 00:09:18,110

 

180
00:09:18,110 --> 00:09:20,600

 

181
00:09:20,600 --> 00:09:22,220

 

182
00:09:22,220 --> 00:09:24,140

 

183
00:09:24,140 --> 00:09:26,810

 

184
00:09:26,810 --> 00:09:29,960

 

185
00:09:29,960 --> 00:09:33,890

 

186
00:09:33,890 --> 00:09:37,580

 

187
00:09:37,580 --> 00:09:40,940

 

188
00:09:40,940 --> 00:09:43,880

 

189
00:09:43,880 --> 00:09:45,620

 

190
00:09:45,620 --> 00:09:47,600

 

191
00:09:47,600 --> 00:09:49,400

 

192
00:09:49,400 --> 00:09:55,010

 

193
00:09:55,010 --> 00:09:57,530

 

194
00:09:57,530 --> 00:09:59,210

 

195
00:09:59,210 --> 00:10:02,210

 

196
00:10:02,210 --> 00:10:05,030

 

197
00:10:05,030 --> 00:10:07,790

 

198
00:10:07,790 --> 00:10:12,500

 

199
00:10:12,500 --> 00:10:14,870

 

200
00:10:14,870 --> 00:10:14,880

 

201
00:10:14,880 --> 00:10:18,150


202
00:10:18,150 --> 00:10:20,710

 

203
00:10:20,710 --> 00:10:22,840

 

204
00:10:22,840 --> 00:10:25,600

 

205
00:10:25,600 --> 00:10:31,810

 

206
00:10:31,810 --> 00:10:34,960

 

207
00:10:34,960 --> 00:10:36,670

 

208
00:10:36,670 --> 00:10:45,050

 

209
00:10:45,050 --> 00:10:50,510

 

210
00:10:50,510 --> 00:10:52,460

 

211
00:10:52,460 --> 00:10:52,470

 

212
00:10:52,470 --> 00:10:53,150


213
00:10:53,150 --> 00:10:54,440

 

214
00:10:54,440 --> 00:10:56,690

 

215
00:10:56,690 --> 00:10:58,670

 

216
00:10:58,670 --> 00:11:00,170

 

217
00:11:00,170 --> 00:11:03,550

 

218
00:11:03,550 --> 00:11:06,590

 

219
00:11:06,590 --> 00:11:14,510

 

220
00:11:14,510 --> 00:11:16,010

 

221
00:11:16,010 --> 00:11:20,240

 

222
00:11:20,240 --> 00:11:22,640

 

223
00:11:22,640 --> 00:11:24,530

 

224
00:11:24,530 --> 00:11:26,480

 

225
00:11:26,480 --> 00:11:36,440

 

226
00:11:36,440 --> 00:11:38,090

 

227
00:11:38,090 --> 00:11:40,970

 

228
00:11:40,970 --> 00:11:43,730

 

229
00:11:43,730 --> 00:11:46,100

 

230
00:11:46,100 --> 00:11:48,980

 

231
00:11:48,980 --> 00:11:50,870

 

232
00:11:50,870 --> 00:11:53,900

 

233
00:11:53,900 --> 00:11:55,790

 

234
00:11:55,790 --> 00:12:04,150

 

235
00:12:04,150 --> 00:12:10,530

 

236
00:12:10,530 --> 00:12:16,060

 

237
00:12:16,060 --> 00:12:21,490

 

238
00:12:21,490 --> 00:12:25,960

 

239
00:12:25,960 --> 00:12:28,360

 

240
00:12:28,360 --> 00:12:30,520

 

241
00:12:30,520 --> 00:12:34,450

 

242
00:12:34,450 --> 00:12:37,450

 

243
00:12:37,450 --> 00:12:40,090

 

244
00:12:40,090 --> 00:12:41,980

 

245
00:12:41,980 --> 00:12:44,020

 

246
00:12:44,020 --> 00:12:47,950

 

247
00:12:47,950 --> 00:12:52,510

 

248
00:12:52,510 --> 00:12:54,820

 

249
00:12:54,820 --> 00:12:59,560

 

250
00:12:59,560 --> 00:13:02,260

 

251
00:13:02,260 --> 00:13:03,670

 

252
00:13:03,670 --> 00:13:03,680

 

253
00:13:03,680 --> 00:13:12,480


254
00:13:12,480 --> 00:13:12,490

 

255
00:13:12,490 --> 00:13:15,680


256
00:13:15,680 --> 00:13:17,540

 

257
00:13:17,540 --> 00:13:19,820

 

258
00:13:19,820 --> 00:13:25,190

 

259
00:13:25,190 --> 00:13:27,020

 

260
00:13:27,020 --> 00:13:30,020

 

261
00:13:30,020 --> 00:13:31,490

 

262
00:13:31,490 --> 00:13:35,740

 

263
00:13:35,740 --> 00:13:37,700

 

264
00:13:37,700 --> 00:13:39,110

 

265
00:13:39,110 --> 00:13:40,790

 

266
00:13:40,790 --> 00:13:43,040

 

267
00:13:43,040 --> 00:13:45,110

 

268
00:13:45,110 --> 00:13:47,590

 

269
00:13:47,590 --> 00:13:51,760

 

270
00:13:51,760 --> 00:13:54,260

 

271
00:13:54,260 --> 00:13:57,710

 

272
00:13:57,710 --> 00:14:00,050

 

273
00:14:00,050 --> 00:14:02,780

 

274
00:14:02,780 --> 00:14:05,960

 

275
00:14:05,960 --> 00:14:08,750

 

276
00:14:08,750 --> 00:14:10,700

 

277
00:14:10,700 --> 00:14:13,430

 

278
00:14:13,430 --> 00:14:15,830

 

279
00:14:15,830 --> 00:14:17,870

 

280
00:14:17,870 --> 00:14:20,360

 

281
00:14:20,360 --> 00:14:24,830

 

282
00:14:24,830 --> 00:14:27,740

 

283
00:14:27,740 --> 00:14:31,340

 

284
00:14:31,340 --> 00:14:35,120

 

285
00:14:35,120 --> 00:14:39,890

 

286
00:14:39,890 --> 00:14:41,960

 

287
00:14:41,960 --> 00:14:45,050

 

288
00:14:45,050 --> 00:14:48,140

 

289
00:14:48,140 --> 00:14:50,990

 

290
00:14:50,990 --> 00:14:53,780

 

291
00:14:53,780 --> 00:14:55,790

 

292
00:14:55,790 --> 00:14:58,220

 

293
00:14:58,220 --> 00:14:59,750

 

294
00:14:59,750 --> 00:15:02,330

 

295
00:15:02,330 --> 00:15:07,370

 

296
00:15:07,370 --> 00:15:09,230

 

297
00:15:09,230 --> 00:15:13,340

 

298
00:15:13,340 --> 00:15:18,020

 

299
00:15:18,020 --> 00:15:20,450

 

300
00:15:20,450 --> 00:15:23,210

 

301
00:15:23,210 --> 00:15:29,540

 

302
00:15:29,540 --> 00:15:33,170

 

303
00:15:33,170 --> 00:15:35,930

 

304
00:15:35,930 --> 00:15:38,710

 

305
00:15:38,710 --> 00:15:41,090

 

306
00:15:41,090 --> 00:15:43,820

 

307
00:15:43,820 --> 00:15:46,730

 

308
00:15:46,730 --> 00:15:48,470

 

309
00:15:48,470 --> 00:15:53,180

 

310
00:15:53,180 --> 00:15:56,750

 

311
00:15:56,750 --> 00:15:59,690

 

312
00:15:59,690 --> 00:16:02,840

 

313
00:16:02,840 --> 00:16:06,200

 

314
00:16:06,200 --> 00:16:08,840

 

315
00:16:08,840 --> 00:16:11,810

 

316
00:16:11,810 --> 00:16:16,820

 

317
00:16:16,820 --> 00:16:20,780

 

318
00:16:20,780 --> 00:16:24,130

 

319
00:16:24,130 --> 00:16:27,440

 

320
00:16:27,440 --> 00:16:30,620

 

321
00:16:30,620 --> 00:16:34,700

 

322
00:16:34,700 --> 00:16:37,790

 

323
00:16:37,790 --> 00:16:39,650

 

324
00:16:39,650 --> 00:16:43,220

 

325
00:16:43,220 --> 00:16:45,800

 

326
00:16:45,800 --> 00:16:50,080

 

327
00:16:50,080 --> 00:16:54,260

 

328
00:16:54,260 --> 00:16:56,300

 

329
00:16:56,300 --> 00:16:58,580

 

330
00:16:58,580 --> 00:17:01,640

 

331
00:17:01,640 --> 00:17:03,560

 

332
00:17:03,560 --> 00:17:05,420

 

333
00:17:05,420 --> 00:17:11,480

 

334
00:17:11,480 --> 00:17:13,760

 

335
00:17:13,760 --> 00:17:17,090

 

336
00:17:17,090 --> 00:17:19,280

 

337
00:17:19,280 --> 00:17:23,030

 

338
00:17:23,030 --> 00:17:25,640

 

339
00:17:25,640 --> 00:17:27,020

 

340
00:17:27,020 --> 00:17:28,730

 

341
00:17:28,730 --> 00:17:31,430

 

342
00:17:31,430 --> 00:17:34,130

 

343
00:17:34,130 --> 00:17:35,510

 

344
00:17:35,510 --> 00:17:40,940

 

345
00:17:40,940 --> 00:17:42,710

 

346
00:17:42,710 --> 00:17:45,350

 

347
00:17:45,350 --> 00:17:46,940

 

348
00:17:46,940 --> 00:17:48,350

 

349
00:17:48,350 --> 00:17:51,530

 

350
00:17:51,530 --> 00:17:52,820

 

351
00:17:52,820 --> 00:17:55,040

 

352
00:17:55,040 --> 00:17:56,420

 

353
00:17:56,420 --> 00:18:00,140

 

354
00:18:00,140 --> 00:18:02,630

 

355
00:18:02,630 --> 00:18:04,610

 

356
00:18:04,610 --> 00:18:07,580

 

357
00:18:07,580 --> 00:18:09,730

 

358
00:18:09,730 --> 00:18:11,900

 

359
00:18:11,900 --> 00:18:13,670

 

360
00:18:13,670 --> 00:18:16,640

 

361
00:18:16,640 --> 00:18:18,860

 

362
00:18:18,860 --> 00:18:22,340

 

363
00:18:22,340 --> 00:18:28,000

 

364
00:18:28,000 --> 00:18:30,800

 

365
00:18:30,800 --> 00:18:35,030

 

366
00:18:35,030 --> 00:18:39,940

 

367
00:18:39,940 --> 00:18:44,060

 

368
00:18:44,060 --> 00:18:45,710

 

369
00:18:45,710 --> 00:18:50,510

 

370
00:18:50,510 --> 00:18:52,640

 

371
00:18:52,640 --> 00:18:55,100

 

372
00:18:55,100 --> 00:18:57,470

 

373
00:18:57,470 --> 00:19:00,140

 

374
00:19:00,140 --> 00:19:02,750

 

375
00:19:02,750 --> 00:19:06,890

 

376
00:19:06,890 --> 00:19:10,760

 

377
00:19:10,760 --> 00:19:12,710

 

378
00:19:12,710 --> 00:19:17,840

 

379
00:19:17,840 --> 00:19:24,170

 

380
00:19:24,170 --> 00:19:26,420

 

381
00:19:26,420 --> 00:19:28,130

 

382
00:19:28,130 --> 00:19:30,170

 

383
00:19:30,170 --> 00:19:31,340

 

384
00:19:31,340 --> 00:19:33,710

 

385
00:19:33,710 --> 00:19:39,260

 

386
00:19:39,260 --> 00:19:44,300

 

387
00:19:44,300 --> 00:19:45,980

 

388
00:19:45,980 --> 00:19:48,650

 

389
00:19:48,650 --> 00:19:54,830

 

390
00:19:54,830 --> 00:20:00,280

 

391
00:20:00,280 --> 00:20:03,980

 

392
00:20:03,980 --> 00:20:05,870

 

393
00:20:05,870 --> 00:20:05,880

 

394
00:20:05,880 --> 00:20:06,530


395
00:20:06,530 --> 00:20:09,650

 

396
00:20:09,650 --> 00:20:11,480

 

397
00:20:11,480 --> 00:20:14,930

 

398
00:20:14,930 --> 00:20:16,700

 

399
00:20:16,700 --> 00:20:20,930

 

400
00:20:20,930 --> 00:20:24,080

 

401
00:20:24,080 --> 00:20:25,730

 

402
00:20:25,730 --> 00:20:28,430

 

403
00:20:28,430 --> 00:20:30,830

 

404
00:20:30,830 --> 00:20:32,600

 

405
00:20:32,600 --> 00:20:36,440

 

406
00:20:36,440 --> 00:20:37,700

 

407
00:20:37,700 --> 00:20:39,740

 

408
00:20:39,740 --> 00:20:40,670

 

409
00:20:40,670 --> 00:20:43,520

 

410
00:20:43,520 --> 00:20:46,040

 

411
00:20:46,040 --> 00:20:48,110

 

412
00:20:48,110 --> 00:20:51,200

 

413
00:20:51,200 --> 00:20:53,990

 

414
00:20:53,990 --> 00:20:56,240

 

415
00:20:56,240 --> 00:20:59,390

 

416
00:20:59,390 --> 00:21:02,600

 

417
00:21:02,600 --> 00:21:04,970

 

418
00:21:04,970 --> 00:21:06,710

 

419
00:21:06,710 --> 00:21:08,570

 

420
00:21:08,570 --> 00:21:10,610

 

421
00:21:10,610 --> 00:21:12,650

 

422
00:21:12,650 --> 00:21:15,200

 

423
00:21:15,200 --> 00:21:18,080

 

424
00:21:18,080 --> 00:21:19,670

 

425
00:21:19,670 --> 00:21:21,410

 

426
00:21:21,410 --> 00:21:26,270

 

427
00:21:26,270 --> 00:21:28,250

 

428
00:21:28,250 --> 00:21:30,590

 

429
00:21:30,590 --> 00:21:33,560

 

430
00:21:33,560 --> 00:21:36,440

 

431
00:21:36,440 --> 00:21:37,940

 

432
00:21:37,940 --> 00:21:39,980

 

433
00:21:39,980 --> 00:21:42,500

 

434
00:21:42,500 --> 00:21:45,470

 

435
00:21:45,470 --> 00:21:47,750

 

436
00:21:47,750 --> 00:21:49,580

 

437
00:21:49,580 --> 00:21:53,629

 

438
00:21:53,629 --> 00:21:55,070

 

439
00:21:55,070 --> 00:21:56,629

 

440
00:21:56,629 --> 00:22:02,539

 

441
00:22:02,539 --> 00:22:03,830

 

442
00:22:03,830 --> 00:22:08,659

 

443
00:22:08,659 --> 00:22:11,810

 

444
00:22:11,810 --> 00:22:15,799

 

445
00:22:15,799 --> 00:22:18,080

 

446
00:22:18,080 --> 00:22:20,860

 

447
00:22:20,860 --> 00:22:23,120

 

448
00:22:23,120 --> 00:22:27,710

 

449
00:22:27,710 --> 00:22:32,090

 

450
00:22:32,090 --> 00:22:36,799

 

451
00:22:36,799 --> 00:22:41,840

 

452
00:22:41,840 --> 00:22:43,909

 

453
00:22:43,909 --> 00:22:45,409

 

454
00:22:45,409 --> 00:22:48,769

 

455
00:22:48,769 --> 00:22:50,750

 

456
00:22:50,750 --> 00:22:53,299

 

457
00:22:53,299 --> 00:22:55,279

 

458
00:22:55,279 --> 00:22:56,810

 

459
00:22:56,810 --> 00:22:59,899

 

460
00:22:59,899 --> 00:23:02,200

 

461
00:23:02,200 --> 00:23:04,399

 

462
00:23:04,399 --> 00:23:06,980

 

463
00:23:06,980 --> 00:23:10,639

 

464
00:23:10,639 --> 00:23:13,370

 

465
00:23:13,370 --> 00:23:15,259

 

466
00:23:15,259 --> 00:23:17,870

 

467
00:23:17,870 --> 00:23:19,370

 

468
00:23:19,370 --> 00:23:22,330

 

469
00:23:22,330 --> 00:23:25,610

 

470
00:23:25,610 --> 00:23:27,019

 

471
00:23:27,019 --> 00:23:28,850

 

472
00:23:28,850 --> 00:23:35,259

 

473
00:23:35,259 --> 00:23:39,169

 

474
00:23:39,169 --> 00:23:43,009

 

475
00:23:43,009 --> 00:23:46,759

 

476
00:23:46,759 --> 00:23:49,340

 

477
00:23:49,340 --> 00:23:50,750

 

478
00:23:50,750 --> 00:23:51,440

 

479
00:23:51,440 --> 00:23:53,629

 

480
00:23:53,629 --> 00:23:56,389

 

481
00:23:56,389 --> 00:23:59,419

 

482
00:23:59,419 --> 00:24:02,450

 

483
00:24:02,450 --> 00:24:04,100

 

484
00:24:04,100 --> 00:24:06,529

 

485
00:24:06,529 --> 00:24:09,080

 

486
00:24:09,080 --> 00:24:11,360

 

487
00:24:11,360 --> 00:24:15,440

 

488
00:24:15,440 --> 00:24:16,879

 

489
00:24:16,879 --> 00:24:18,860

 

490
00:24:18,860 --> 00:24:23,000

 

491
00:24:23,000 --> 00:24:25,759

 

492
00:24:25,759 --> 00:24:27,379

 

493
00:24:27,379 --> 00:24:32,450

 

494
00:24:32,450 --> 00:24:33,919

 

495
00:24:33,919 --> 00:24:36,740

 

496
00:24:36,740 --> 00:24:38,480

 

497
00:24:38,480 --> 00:24:39,889

 

498
00:24:39,889 --> 00:24:42,370

 

499
00:24:42,370 --> 00:24:44,960

 

500
00:24:44,960 --> 00:24:46,879

 

501
00:24:46,879 --> 00:24:46,889

 

502
00:24:46,889 --> 00:24:48,700


503
00:24:48,700 --> 00:24:52,490

 

504
00:24:52,490 --> 00:24:54,590

 

505
00:24:54,590 --> 00:24:55,850

 

506
00:24:55,850 --> 00:24:57,830

 

507
00:24:57,830 --> 00:25:00,230

 

508
00:25:00,230 --> 00:25:04,100

 

509
00:25:04,100 --> 00:25:07,250

 

510
00:25:07,250 --> 00:25:09,440

 

511
00:25:09,440 --> 00:25:11,450

 

512
00:25:11,450 --> 00:25:13,820

 

513
00:25:13,820 --> 00:25:16,009

 

514
00:25:16,009 --> 00:25:18,740

 

515
00:25:18,740 --> 00:25:20,120

 

516
00:25:20,120 --> 00:25:22,940

 

517
00:25:22,940 --> 00:25:26,389

 

518
00:25:26,389 --> 00:25:27,950

 

519
00:25:27,950 --> 00:25:29,480

 

520
00:25:29,480 --> 00:25:31,430

 

521
00:25:31,430 --> 00:25:33,590

 

522
00:25:33,590 --> 00:25:36,889

 

523
00:25:36,889 --> 00:25:38,299

 

524
00:25:38,299 --> 00:25:42,560

 

525
00:25:42,560 --> 00:25:45,919

 

526
00:25:45,919 --> 00:25:48,350

 

527
00:25:48,350 --> 00:25:51,830

 

528
00:25:51,830 --> 00:25:53,629

 

529
00:25:53,629 --> 00:25:54,769

 

530
00:25:54,769 --> 00:25:57,009

 

531
00:25:57,009 --> 00:25:59,240

 

532
00:25:59,240 --> 00:26:00,950

 

533
00:26:00,950 --> 00:26:02,480

 

534
00:26:02,480 --> 00:26:03,510

 

535
00:26:03,510 --> 00:26:06,360

 

536
00:26:06,360 --> 00:26:10,110

 

537
00:26:10,110 --> 00:26:11,880

 

538
00:26:11,880 --> 00:26:14,310

 

539
00:26:14,310 --> 00:26:17,670

 

540
00:26:17,670 --> 00:26:20,940

 

541
00:26:20,940 --> 00:26:23,070

 

542
00:26:23,070 --> 00:26:25,620

 

543
00:26:25,620 --> 00:26:27,600

 

544
00:26:27,600 --> 00:26:31,860

 

545
00:26:31,860 --> 00:26:33,540

 

546
00:26:33,540 --> 00:26:37,560

 

547
00:26:37,560 --> 00:26:39,510

 

548
00:26:39,510 --> 00:26:40,530

 

549
00:26:40,530 --> 00:26:43,200

 

550
00:26:43,200 --> 00:26:46,890

 

551
00:26:46,890 --> 00:26:48,420

 

552
00:26:48,420 --> 00:26:50,580

 

553
00:26:50,580 --> 00:26:53,460

 

554
00:26:53,460 --> 00:26:54,840

 

555
00:26:54,840 --> 00:26:56,840

 

556
00:26:56,840 --> 00:26:59,450

 

557
00:26:59,450 --> 00:27:01,800

 

558
00:27:01,800 --> 00:27:03,060

 

559
00:27:03,060 --> 00:27:05,190

 

560
00:27:05,190 --> 00:27:06,780

 

561
00:27:06,780 --> 00:27:10,230

 

562
00:27:10,230 --> 00:27:12,120

 

563
00:27:12,120 --> 00:27:14,610

 

564
00:27:14,610 --> 00:27:16,020

 

565
00:27:16,020 --> 00:27:17,280

 

566
00:27:17,280 --> 00:27:20,640

 

567
00:27:20,640 --> 00:27:23,340

 

568
00:27:23,340 --> 00:27:25,890

 

569
00:27:25,890 --> 00:27:29,250

 

570
00:27:29,250 --> 00:27:32,190

 

571
00:27:32,190 --> 00:27:34,290

 

572
00:27:34,290 --> 00:27:37,140

 

573
00:27:37,140 --> 00:27:40,320

 

574
00:27:40,320 --> 00:27:43,650

 

575
00:27:43,650 --> 00:27:46,950

 

576
00:27:46,950 --> 00:27:51,120

 

577
00:27:51,120 --> 00:27:53,520

 

578
00:27:53,520 --> 00:27:55,560

 

579
00:27:55,560 --> 00:27:58,170

 

580
00:27:58,170 --> 00:28:00,540

 

581
00:28:00,540 --> 00:28:04,320

 

582
00:28:04,320 --> 00:28:06,240

 

583
00:28:06,240 --> 00:28:09,060

 

584
00:28:09,060 --> 00:28:15,660

 

585
00:28:15,660 --> 00:28:17,280

 

586
00:28:17,280 --> 00:28:20,370

 

587
00:28:20,370 --> 00:28:24,390

 

588
00:28:24,390 --> 00:28:27,430

 

589
00:28:27,430 --> 00:28:33,010

 

590
00:28:33,010 --> 00:28:34,900

 

591
00:28:34,900 --> 00:28:37,060

 

592
00:28:37,060 --> 00:28:41,350

 

593
00:28:41,350 --> 00:28:45,010

 

594
00:28:45,010 --> 00:28:46,990

 

595
00:28:46,990 --> 00:28:50,260

 

596
00:28:50,260 --> 00:28:55,299

 

597
00:28:55,299 --> 00:28:59,049

 

598
00:28:59,049 --> 00:29:02,080

 

599
00:29:02,080 --> 00:29:04,680

 

600
00:29:04,680 --> 00:29:07,480

 

601
00:29:07,480 --> 00:29:13,810

 

602
00:29:13,810 --> 00:29:15,730

 

603
00:29:15,730 --> 00:29:20,100

 

604
00:29:20,100 --> 00:29:24,850

 

605
00:29:24,850 --> 00:29:27,040

 

606
00:29:27,040 --> 00:29:31,000

 

607
00:29:31,000 --> 00:29:34,030

 

608
00:29:34,030 --> 00:29:39,430

 

609
00:29:39,430 --> 00:29:42,730

 

610
00:29:42,730 --> 00:29:45,820

 

611
00:29:45,820 --> 00:29:48,610

 

612
00:29:48,610 --> 00:29:57,160

 

613
00:29:57,160 --> 00:29:59,380

 

614
00:29:59,380 --> 00:30:01,930

 

615
00:30:01,930 --> 00:30:04,330

 

616
00:30:04,330 --> 00:30:06,070

 

617
00:30:06,070 --> 00:30:09,790

 

618
00:30:09,790 --> 00:30:12,340

 

619
00:30:12,340 --> 00:30:15,100

 

620
00:30:15,100 --> 00:30:18,610

 

621
00:30:18,610 --> 00:30:22,570

 

622
00:30:22,570 --> 00:30:24,760

 

623
00:30:24,760 --> 00:30:27,130

 

624
00:30:27,130 --> 00:30:28,890

 

625
00:30:28,890 --> 00:30:31,200

 

626
00:30:31,200 --> 00:30:33,990

 

627
00:30:33,990 --> 00:30:36,390

 

628
00:30:36,390 --> 00:30:38,400

 

629
00:30:38,400 --> 00:30:40,650

 

630
00:30:40,650 --> 00:30:43,140

 

631
00:30:43,140 --> 00:30:45,000

 

632
00:30:45,000 --> 00:30:47,460

 

633
00:30:47,460 --> 00:30:52,230

 

634
00:30:52,230 --> 00:30:54,900

 

635
00:30:54,900 --> 00:30:59,400

 

636
00:30:59,400 --> 00:31:01,919

 

637
00:31:01,919 --> 00:31:03,660

 

638
00:31:03,660 --> 00:31:06,870

 

639
00:31:06,870 --> 00:31:10,980

 

640
00:31:10,980 --> 00:31:12,600

 

641
00:31:12,600 --> 00:31:16,410

 

642
00:31:16,410 --> 00:31:19,410

 

643
00:31:19,410 --> 00:31:25,770

 

644
00:31:25,770 --> 00:31:29,190

 

645
00:31:29,190 --> 00:31:31,230

 

646
00:31:31,230 --> 00:31:33,630

 

647
00:31:33,630 --> 00:31:36,870

 

648
00:31:36,870 --> 00:31:38,370

 

649
00:31:38,370 --> 00:31:41,010

 

650
00:31:41,010 --> 00:31:47,940

 

651
00:31:47,940 --> 00:31:53,310

 

652
00:31:53,310 --> 00:31:55,200

 

653
00:31:55,200 --> 00:31:56,070

 

654
00:31:56,070 --> 00:32:00,030

 

655
00:32:00,030 --> 00:32:03,170

 

656
00:32:03,170 --> 00:32:04,980

 

657
00:32:04,980 --> 00:32:09,480

 

658
00:32:09,480 --> 00:32:11,190

 

659
00:32:11,190 --> 00:32:12,810

 

660
00:32:12,810 --> 00:32:15,750

 

661
00:32:15,750 --> 00:32:17,270

 

662
00:32:17,270 --> 00:32:21,300

 

663
00:32:21,300 --> 00:32:25,050

 

664
00:32:25,050 --> 00:32:26,730

 

665
00:32:26,730 --> 00:32:33,320

 

666
00:32:33,320 --> 00:32:35,640

 

667
00:32:35,640 --> 00:32:38,040

 

668
00:32:38,040 --> 00:32:44,780

 

669
00:32:44,780 --> 00:32:48,620

 

670
00:32:48,620 --> 00:32:50,480

 

671
00:32:50,480 --> 00:32:58,240

 

672
00:32:58,240 --> 00:33:01,850

 

673
00:33:01,850 --> 00:33:03,320

 

674
00:33:03,320 --> 00:33:05,000

 

675
00:33:05,000 --> 00:33:09,770

 

676
00:33:09,770 --> 00:33:12,700

 

677
00:33:12,700 --> 00:33:16,460

 

678
00:33:16,460 --> 00:33:20,480

 

679
00:33:20,480 --> 00:33:24,230

 

680
00:33:24,230 --> 00:33:25,790

 

681
00:33:25,790 --> 00:33:27,950

 

682
00:33:27,950 --> 00:33:30,350

 

683
00:33:30,350 --> 00:33:33,440

 

684
00:33:33,440 --> 00:33:36,830

 

685
00:33:36,830 --> 00:33:40,760

 

686
00:33:40,760 --> 00:33:42,920

 

687
00:33:42,920 --> 00:33:44,780

 

688
00:33:44,780 --> 00:33:46,070

 

689
00:33:46,070 --> 00:33:47,600

 

690
00:33:47,600 --> 00:33:50,750

 

691
00:33:50,750 --> 00:33:55,730

 

692
00:33:55,730 --> 00:33:59,840

 

693
00:33:59,840 --> 00:34:01,730

 

694
00:34:01,730 --> 00:34:03,800

 

695
00:34:03,800 --> 00:34:06,350

 

696
00:34:06,350 --> 00:34:08,330

 

697
00:34:08,330 --> 00:34:14,990

 

698
00:34:14,990 --> 00:34:17,030

 

699
00:34:17,030 --> 00:34:20,030

 

700
00:34:20,030 --> 00:34:22,850

 

701
00:34:22,850 --> 00:34:25,310

 

702
00:34:25,310 --> 00:34:26,450

 

703
00:34:26,450 --> 00:34:28,340

 

704
00:34:28,340 --> 00:34:29,780

 

705
00:34:29,780 --> 00:34:33,320

 

706
00:34:33,320 --> 00:34:35,060

 

707
00:34:35,060 --> 00:34:36,619

 

708
00:34:36,619 --> 00:34:39,200

 

709
00:34:39,200 --> 00:34:42,800

 

710
00:34:42,800 --> 00:34:46,520

 

711
00:34:46,520 --> 00:34:48,409

 

712
00:34:48,409 --> 00:34:50,780

 

713
00:34:50,780 --> 00:34:53,690

 

714
00:34:53,690 --> 00:34:57,200

 

715
00:34:57,200 --> 00:34:58,460

 

716
00:34:58,460 --> 00:35:00,710

 

717
00:35:00,710 --> 00:35:02,240

 

718
00:35:02,240 --> 00:35:04,339

 

719
00:35:04,339 --> 00:35:07,870

 

720
00:35:07,870 --> 00:35:10,430

 

721
00:35:10,430 --> 00:35:12,950

 

722
00:35:12,950 --> 00:35:14,720

 

723
00:35:14,720 --> 00:35:18,620

 

724
00:35:18,620 --> 00:35:20,809

 

725
00:35:20,809 --> 00:35:22,370

 

726
00:35:22,370 --> 00:35:24,380

 

727
00:35:24,380 --> 00:35:29,140

 

728
00:35:29,140 --> 00:35:34,870

 

729
00:35:34,870 --> 00:35:43,339

 

730
00:35:43,339 --> 00:35:45,650

 

731
00:35:45,650 --> 00:35:48,230

 

732
00:35:48,230 --> 00:35:50,059

 

733
00:35:50,059 --> 00:35:55,309

 

734
00:35:55,309 --> 00:35:56,569

 

735
00:35:56,569 --> 00:36:00,650

 

736
00:36:00,650 --> 00:36:04,120

 

737
00:36:04,120 --> 00:36:08,390

 

738
00:36:08,390 --> 00:36:10,069

 

739
00:36:10,069 --> 00:36:12,859

 

740
00:36:12,859 --> 00:36:14,450

 

741
00:36:14,450 --> 00:36:18,079

 

742
00:36:18,079 --> 00:36:21,530

 

743
00:36:21,530 --> 00:36:26,000

 

744
00:36:26,000 --> 00:36:30,559

 

745
00:36:30,559 --> 00:36:33,079

 

746
00:36:33,079 --> 00:36:36,800

 

747
00:36:36,800 --> 00:36:39,050

 

748
00:36:39,050 --> 00:36:42,079

 

749
00:36:42,079 --> 00:36:45,349

 

750
00:36:45,349 --> 00:36:48,260

 

751
00:36:48,260 --> 00:36:49,640

 

752
00:36:49,640 --> 00:36:51,200

 

753
00:36:51,200 --> 00:36:52,059

 

754
00:36:52,059 --> 00:36:54,740

 

755
00:36:54,740 --> 00:36:56,960

 

756
00:36:56,960 --> 00:37:00,410

 

757
00:37:00,410 --> 00:37:03,680

 

758
00:37:03,680 --> 00:37:05,120

 

759
00:37:05,120 --> 00:37:08,630

 

760
00:37:08,630 --> 00:37:11,210

 

761
00:37:11,210 --> 00:37:12,359

 

762
00:37:12,359 --> 00:37:13,710

 

763
00:37:13,710 --> 00:37:15,569

 

764
00:37:15,569 --> 00:37:17,970

 

765
00:37:17,970 --> 00:37:20,670

 

766
00:37:20,670 --> 00:37:24,690

 

767
00:37:24,690 --> 00:37:26,700

 

768
00:37:26,700 --> 00:37:28,799

 

769
00:37:28,799 --> 00:37:30,390

 

770
00:37:30,390 --> 00:37:32,819

 

771
00:37:32,819 --> 00:37:34,980

 

772
00:37:34,980 --> 00:37:36,329

 

773
00:37:36,329 --> 00:37:38,099

 

774
00:37:38,099 --> 00:37:39,329

 

775
00:37:39,329 --> 00:37:43,200

 

776
00:37:43,200 --> 00:37:46,200

 

777
00:37:46,200 --> 00:37:47,970

 

778
00:37:47,970 --> 00:37:50,309

 

779
00:37:50,309 --> 00:37:56,370

 

780
00:37:56,370 --> 00:37:58,980

 

781
00:37:58,980 --> 00:38:01,259

 

782
00:38:01,259 --> 00:38:03,089

 

783
00:38:03,089 --> 00:38:06,059

 

784
00:38:06,059 --> 00:38:07,650

 

785
00:38:07,650 --> 00:38:08,999

 

786
00:38:08,999 --> 00:38:11,130

 

787
00:38:11,130 --> 00:38:18,680

 

788
00:38:18,680 --> 00:38:20,519

 

789
00:38:20,519 --> 00:38:22,920

 

790
00:38:22,920 --> 00:38:24,779

 

791
00:38:24,779 --> 00:38:26,819

 

792
00:38:26,819 --> 00:38:30,779

 

793
00:38:30,779 --> 00:38:33,269

 

794
00:38:33,269 --> 00:38:38,999

 

795
00:38:38,999 --> 00:38:40,769

 

796
00:38:40,769 --> 00:38:44,489

 

797
00:38:44,489 --> 00:38:47,160

 

798
00:38:47,160 --> 00:38:48,749

 

799
00:38:48,749 --> 00:38:48,759

 

800
00:38:48,759 --> 00:38:49,370


801
00:38:49,370 --> 00:38:51,930

 

802
00:38:51,930 --> 00:38:53,670

 

803
00:38:53,670 --> 00:38:56,069

 

804
00:38:56,069 --> 00:38:57,779

 

805
00:38:57,779 --> 00:39:01,529

 

806
00:39:01,529 --> 00:39:04,680

 

807
00:39:04,680 --> 00:39:07,680

 

808
00:39:07,680 --> 00:39:09,509

 

809
00:39:09,509 --> 00:39:12,299

 

810
00:39:12,299 --> 00:39:15,299

 

811
00:39:15,299 --> 00:39:18,059

 

812
00:39:18,059 --> 00:39:23,249

 

813
00:39:23,249 --> 00:39:23,259

 

814
00:39:23,259 --> 00:39:25,280


815
00:39:25,280 --> 00:39:28,160

 

816
00:39:28,160 --> 00:39:30,350

 

817
00:39:30,350 --> 00:39:32,330

 

818
00:39:32,330 --> 00:39:35,990

 

819
00:39:35,990 --> 00:39:37,940

 

820
00:39:37,940 --> 00:39:39,770

 

821
00:39:39,770 --> 00:39:42,620

 

822
00:39:42,620 --> 00:39:46,190

 

823
00:39:46,190 --> 00:39:49,490

 

824
00:39:49,490 --> 00:39:51,760

 

825
00:39:51,760 --> 00:39:54,050

 

826
00:39:54,050 --> 00:39:55,790

 

827
00:39:55,790 --> 00:39:57,620

 

828
00:39:57,620 --> 00:40:01,580

 

829
00:40:01,580 --> 00:40:03,920

 

830
00:40:03,920 --> 00:40:05,570

 

831
00:40:05,570 --> 00:40:09,410

 

832
00:40:09,410 --> 00:40:12,740

 

833
00:40:12,740 --> 00:40:15,890

 

834
00:40:15,890 --> 00:40:18,740

 

835
00:40:18,740 --> 00:40:20,870

 

836
00:40:20,870 --> 00:40:24,680

 

837
00:40:24,680 --> 00:40:26,480

 

838
00:40:26,480 --> 00:40:29,180

 

839
00:40:29,180 --> 00:40:31,010

 

840
00:40:31,010 --> 00:40:35,630

 

841
00:40:35,630 --> 00:40:37,760

 

842
00:40:37,760 --> 00:40:39,080

 

843
00:40:39,080 --> 00:40:44,000

 

844
00:40:44,000 --> 00:40:47,060

 

845
00:40:47,060 --> 00:40:48,680

 

846
00:40:48,680 --> 00:40:51,920

 

847
00:40:51,920 --> 00:40:53,540

 

848
00:40:53,540 --> 00:40:56,030

 

849
00:40:56,030 --> 00:40:58,430

 

850
00:40:58,430 --> 00:41:02,990

 

851
00:41:02,990 --> 00:41:05,600

 

852
00:41:05,600 --> 00:41:06,950

 

853
00:41:06,950 --> 00:41:11,200

 

854
00:41:11,200 --> 00:41:15,140

 

855
00:41:15,140 --> 00:41:17,240

 

856
00:41:17,240 --> 00:41:20,660

 

857
00:41:20,660 --> 00:41:25,490

 

858
00:41:25,490 --> 00:41:28,160

 

859
00:41:28,160 --> 00:41:30,200

 

860
00:41:30,200 --> 00:41:32,690

 

861
00:41:32,690 --> 00:41:35,090

 

862
00:41:35,090 --> 00:41:40,349

 

863
00:41:40,349 --> 00:41:44,970

 

864
00:41:44,970 --> 00:41:48,120

 

865
00:41:48,120 --> 00:41:50,430

 

866
00:41:50,430 --> 00:41:52,289

 

867
00:41:52,289 --> 00:41:54,829

 

868
00:41:54,829 --> 00:41:59,759

 

869
00:41:59,759 --> 00:42:01,380

 

870
00:42:01,380 --> 00:42:03,749

 

871
00:42:03,749 --> 00:42:06,720

 

872
00:42:06,720 --> 00:42:09,630

 

873
00:42:09,630 --> 00:42:11,549

 

874
00:42:11,549 --> 00:42:14,069

 

875
00:42:14,069 --> 00:42:16,680

 

876
00:42:16,680 --> 00:42:20,220

 

877
00:42:20,220 --> 00:42:21,779

 

878
00:42:21,779 --> 00:42:24,299

 

879
00:42:24,299 --> 00:42:27,470

 

880
00:42:27,470 --> 00:42:29,700

 

881
00:42:29,700 --> 00:42:31,440

 

882
00:42:31,440 --> 00:42:35,309

 

883
00:42:35,309 --> 00:42:37,529

 

884
00:42:37,529 --> 00:42:40,230

 

885
00:42:40,230 --> 00:42:43,970

 

886
00:42:43,970 --> 00:42:47,999

 

887
00:42:47,999 --> 00:42:50,609

 

888
00:42:50,609 --> 00:42:52,499

 

889
00:42:52,499 --> 00:42:54,630

 

890
00:42:54,630 --> 00:42:57,539

 

891
00:42:57,539 --> 00:42:59,999

 

892
00:42:59,999 --> 00:43:02,759

 

893
00:43:02,759 --> 00:43:04,680

 

894
00:43:04,680 --> 00:43:06,509

 

895
00:43:06,509 --> 00:43:08,970

 

896
00:43:08,970 --> 00:43:11,069

 

897
00:43:11,069 --> 00:43:12,779

 

898
00:43:12,779 --> 00:43:17,460

 

899
00:43:17,460 --> 00:43:19,920

 

900
00:43:19,920 --> 00:43:22,380

 

901
00:43:22,380 --> 00:43:24,989

 

902
00:43:24,989 --> 00:43:26,579

 

903
00:43:26,579 --> 00:43:28,650

 

904
00:43:28,650 --> 00:43:31,289

 

905
00:43:31,289 --> 00:43:32,970

 

906
00:43:32,970 --> 00:43:35,339

 

907
00:43:35,339 --> 00:43:37,769

 

908
00:43:37,769 --> 00:43:40,079

 

909
00:43:40,079 --> 00:43:42,450

 

910
00:43:42,450 --> 00:43:43,890

 

911
00:43:43,890 --> 00:43:46,920

 

912
00:43:46,920 --> 00:43:48,539

 

913
00:43:48,539 --> 00:43:51,930

 

914
00:43:51,930 --> 00:43:55,020

 

915
00:43:55,020 --> 00:43:57,450

 

916
00:43:57,450 --> 00:43:59,670

 

917
00:43:59,670 --> 00:44:07,039

 

918
00:44:07,039 --> 00:44:19,620

 

919
00:44:19,620 --> 00:44:27,900

 

920
00:44:27,900 --> 00:44:30,210

 

921
00:44:30,210 --> 00:44:32,940

 

922
00:44:32,940 --> 00:44:35,819

 

923
00:44:35,819 --> 00:44:38,190

 

924
00:44:38,190 --> 00:44:40,319

 

925
00:44:40,319 --> 00:44:42,870

 

926
00:44:42,870 --> 00:44:44,309

 

927
00:44:44,309 --> 00:44:46,680

 

928
00:44:46,680 --> 00:44:51,870

 

929
00:44:51,870 --> 00:44:54,930

 

930
00:44:54,930 --> 00:44:56,519

 

931
00:44:56,519 --> 00:45:00,029

 

932
00:45:00,029 --> 00:45:02,059

 

933
00:45:02,059 --> 00:45:08,420

 

934
00:45:08,420 --> 00:45:11,339

 

935
00:45:11,339 --> 00:45:13,559

 

936
00:45:13,559 --> 00:45:17,010

 

937
00:45:17,010 --> 00:45:19,589

 

938
00:45:19,589 --> 00:45:22,410

 

939
00:45:22,410 --> 00:45:25,650

 

940
00:45:25,650 --> 00:45:35,150

 

941
00:45:35,150 --> 00:45:39,029

 

942
00:45:39,029 --> 00:45:41,309

 

943
00:45:41,309 --> 00:45:44,400

 

944
00:45:44,400 --> 00:45:46,079

 

945
00:45:46,079 --> 00:45:48,990

 

946
00:45:48,990 --> 00:45:51,559

 

947
00:45:51,559 --> 00:45:55,250

 

948
00:45:55,250 --> 00:45:59,130

 

949
00:45:59,130 --> 00:46:01,289

 

950
00:46:01,289 --> 00:46:04,650

 

951
00:46:04,650 --> 00:46:06,539

 

952
00:46:06,539 --> 00:46:09,150

 

953
00:46:09,150 --> 00:46:13,680

 

954
00:46:13,680 --> 00:46:18,120

 

955
00:46:18,120 --> 00:46:22,200

 

956
00:46:22,200 --> 00:46:25,319

 

957
00:46:25,319 --> 00:46:28,230

 

958
00:46:28,230 --> 00:46:29,670

 

959
00:46:29,670 --> 00:46:32,390

 

960
00:46:32,390 --> 00:46:35,460

 

961
00:46:35,460 --> 00:46:40,020

 

962
00:46:40,020 --> 00:46:43,260

 

963
00:46:43,260 --> 00:46:45,299

 

964
00:46:45,299 --> 00:46:48,620

 

965
00:46:48,620 --> 00:46:51,299

 

966
00:46:51,299 --> 00:46:55,079

 

967
00:46:55,079 --> 00:46:56,970

 

968
00:46:56,970 --> 00:47:01,650

 

969
00:47:01,650 --> 00:47:04,920

 

970
00:47:04,920 --> 00:47:09,120

 

971
00:47:09,120 --> 00:47:11,870

 

972
00:47:11,870 --> 00:47:15,180

 

973
00:47:15,180 --> 00:47:19,319

 

974
00:47:19,319 --> 00:47:22,200

 

975
00:47:22,200 --> 00:47:23,819

 

976
00:47:23,819 --> 00:47:26,160

 

977
00:47:26,160 --> 00:47:28,740

 

978
00:47:28,740 --> 00:47:30,180

 

979
00:47:30,180 --> 00:47:33,180

 

980
00:47:33,180 --> 00:47:35,240

 

981
00:47:35,240 --> 00:47:37,289

 

982
00:47:37,289 --> 00:47:39,569

 

983
00:47:39,569 --> 00:47:42,480

 

984
00:47:42,480 --> 00:47:44,039

 

985
00:47:44,039 --> 00:47:46,170

 

986
00:47:46,170 --> 00:47:48,059

 

987
00:47:48,059 --> 00:47:49,920

 

988
00:47:49,920 --> 00:47:53,519

 

989
00:47:53,519 --> 00:47:55,170

 

990
00:47:55,170 --> 00:47:58,349

 

991
00:47:58,349 --> 00:48:00,299

 

992
00:48:00,299 --> 00:48:03,900

 

993
00:48:03,900 --> 00:48:05,730

 

994
00:48:05,730 --> 00:48:08,430

 

995
00:48:08,430 --> 00:48:10,079

 

996
00:48:10,079 --> 00:48:12,120

 

997
00:48:12,120 --> 00:48:14,069

 

998
00:48:14,069 --> 00:48:17,279

 

999
00:48:17,279 --> 00:48:18,390

 

1000
00:48:18,390 --> 00:48:20,309

 

1001
00:48:20,309 --> 00:48:21,630

 

1002
00:48:21,630 --> 00:48:23,250

 

1003
00:48:23,250 --> 00:48:24,690

 

1004
00:48:24,690 --> 00:48:26,130

 

1005
00:48:26,130 --> 00:48:28,589

 

1006
00:48:28,589 --> 00:48:30,870

 

1007
00:48:30,870 --> 00:48:37,740

 

1008
00:48:37,740 --> 00:48:39,150

 

1009
00:48:39,150 --> 00:48:40,980

 

1010
00:48:40,980 --> 00:48:43,440

 

1011
00:48:43,440 --> 00:48:48,950

 

1012
00:48:48,950 --> 00:48:52,609

 

1013
00:48:52,609 --> 00:48:57,599

 

1014
00:48:57,599 --> 00:49:02,059

 

1015
00:49:02,059 --> 00:49:04,620

 

1016
00:49:04,620 --> 00:49:06,900

 

1017
00:49:06,900 --> 00:49:10,260

 

1018
00:49:10,260 --> 00:49:14,299

 

1019
00:49:14,299 --> 00:49:16,589

 

1020
00:49:16,589 --> 00:49:19,109

 

1021
00:49:19,109 --> 00:49:22,980

 

1022
00:49:22,980 --> 00:49:26,670

 

1023
00:49:26,670 --> 00:49:30,390

 

1024
00:49:30,390 --> 00:49:33,120

 

1025
00:49:33,120 --> 00:49:36,870

 

1026
00:49:36,870 --> 00:49:39,839

 

1027
00:49:39,839 --> 00:49:41,880

 

1028
00:49:41,880 --> 00:49:44,930

 

1029
00:49:44,930 --> 00:49:47,670

 

1030
00:49:47,670 --> 00:49:52,140

 

1031
00:49:52,140 --> 00:49:54,990

 

1032
00:49:54,990 --> 00:49:58,470

 

1033
00:49:58,470 --> 00:50:00,990

 

1034
00:50:00,990 --> 00:50:06,140

 

1035
00:50:06,140 --> 00:50:09,299

 

1036
00:50:09,299 --> 00:50:11,309

 

1037
00:50:11,309 --> 00:50:13,710

 

1038
00:50:13,710 --> 00:50:15,900

 

1039
00:50:15,900 --> 00:50:25,019

 

1040
00:50:25,019 --> 00:50:27,539

 

1041
00:50:27,539 --> 00:50:29,940

 

1042
00:50:29,940 --> 00:50:31,770

 

1043
00:50:31,770 --> 00:50:36,080

 

1044
00:50:36,080 --> 00:50:39,360

 

1045
00:50:39,360 --> 00:50:42,780

 

1046
00:50:42,780 --> 00:50:44,730

 

1047
00:50:44,730 --> 00:50:46,730

 

1048
00:50:46,730 --> 00:50:48,540

 

1049
00:50:48,540 --> 00:50:50,220

 

1050
00:50:50,220 --> 00:50:52,740

 

1051
00:50:52,740 --> 00:50:54,080

 

1052
00:50:54,080 --> 00:50:56,640

 

1053
00:50:56,640 --> 00:50:58,410

 

1054
00:50:58,410 --> 00:51:09,480

 

1055
00:51:09,480 --> 00:51:09,490

 

1056
00:51:09,490 --> 00:51:14,520


1057
00:51:14,520 --> 00:51:17,170

 

1058
00:51:17,170 --> 00:51:18,760

 

1059
00:51:18,760 --> 00:51:21,550

 

1060
00:51:21,550 --> 00:51:25,180

 

1061
00:51:25,180 --> 00:51:27,370

 

1062
00:51:27,370 --> 00:51:31,660

 

1063
00:51:31,660 --> 00:51:36,120

 

1064
00:51:36,120 --> 00:51:39,760

 

1065
00:51:39,760 --> 00:51:41,410

 

1066
00:51:41,410 --> 00:51:48,640

 

1067
00:51:48,640 --> 00:51:51,310

 

1068
00:51:51,310 --> 00:51:53,680

 

1069
00:51:53,680 --> 00:51:55,570

 

1070
00:51:55,570 --> 00:51:58,780

 

1071
00:51:58,780 --> 00:52:01,930

 

1072
00:52:01,930 --> 00:52:04,630

 

1073
00:52:04,630 --> 00:52:08,350

 

1074
00:52:08,350 --> 00:52:11,680

 

1075
00:52:11,680 --> 00:52:13,480

 

1076
00:52:13,480 --> 00:52:17,020

 

1077
00:52:17,020 --> 00:52:19,990

 

1078
00:52:19,990 --> 00:52:21,970

 

1079
00:52:21,970 --> 00:52:24,700

 

1080
00:52:24,700 --> 00:52:26,380

 

1081
00:52:26,380 --> 00:52:29,430

 

1082
00:52:29,430 --> 00:52:33,580

 

1083
00:52:33,580 --> 00:52:35,070

 

1084
00:52:35,070 --> 00:52:39,280

 

1085
00:52:39,280 --> 00:52:42,010

 

1086
00:52:42,010 --> 00:52:44,460

 

1087
00:52:44,460 --> 00:52:46,570

 

1088
00:52:46,570 --> 00:52:49,570

 

1089
00:52:49,570 --> 00:52:52,480

 

1090
00:52:52,480 --> 00:52:55,510

 

1091
00:52:55,510 --> 00:52:58,660

 

1092
00:52:58,660 --> 00:53:00,940

 

1093
00:53:00,940 --> 00:53:02,230

 

1094
00:53:02,230 --> 00:53:04,270

 

1095
00:53:04,270 --> 00:53:06,250

 

1096
00:53:06,250 --> 00:53:08,680

 

1097
00:53:08,680 --> 00:53:11,020

 

1098
00:53:11,020 --> 00:53:13,720

 

1099
00:53:13,720 --> 00:53:15,220

 

1100
00:53:15,220 --> 00:53:17,110

 

1101
00:53:17,110 --> 00:53:21,660

 

1102
00:53:21,660 --> 00:53:24,730

 

1103
00:53:24,730 --> 00:53:26,980

 

1104
00:53:26,980 --> 00:53:28,570

 

1105
00:53:28,570 --> 00:53:30,640

 

1106
00:53:30,640 --> 00:53:33,550

 

1107
00:53:33,550 --> 00:53:36,400

 

1108
00:53:36,400 --> 00:53:40,480

 

1109
00:53:40,480 --> 00:53:43,120

 

1110
00:53:43,120 --> 00:53:45,490

 

1111
00:53:45,490 --> 00:53:47,800

 

1112
00:53:47,800 --> 00:53:49,240

 

1113
00:53:49,240 --> 00:53:52,630

 

1114
00:53:52,630 --> 00:53:55,360

 

1115
00:53:55,360 --> 00:53:57,640

 

1116
00:53:57,640 --> 00:53:59,920

 

1117
00:53:59,920 --> 00:54:03,850

 

1118
00:54:03,850 --> 00:54:06,610

 

1119
00:54:06,610 --> 00:54:09,100

 

1120
00:54:09,100 --> 00:54:14,770

 

1121
00:54:14,770 --> 00:54:16,720

 

1122
00:54:16,720 --> 00:54:21,130

 

1123
00:54:21,130 --> 00:54:25,380

 

1124
00:54:25,380 --> 00:54:28,270

 

1125
00:54:28,270 --> 00:54:32,370

 

1126
00:54:32,370 --> 00:54:35,860

 

1127
00:54:35,860 --> 00:54:38,740

 

1128
00:54:38,740 --> 00:54:41,140

 

1129
00:54:41,140 --> 00:54:42,730

 

1130
00:54:42,730 --> 00:54:44,560

 

1131
00:54:44,560 --> 00:54:47,800

 

1132
00:54:47,800 --> 00:54:49,450

 

1133
00:54:49,450 --> 00:54:51,460

 

1134
00:54:51,460 --> 00:54:56,080

 

1135
00:54:56,080 --> 00:54:57,880

 

1136
00:54:57,880 --> 00:54:59,410

 

1137
00:54:59,410 --> 00:55:03,400

 

1138
00:55:03,400 --> 00:55:06,160

 

1139
00:55:06,160 --> 00:55:09,910

 

1140
00:55:09,910 --> 00:55:11,860

 

1141
00:55:11,860 --> 00:55:13,630

 

1142
00:55:13,630 --> 00:55:16,300

 

1143
00:55:16,300 --> 00:55:18,100

 

1144
00:55:18,100 --> 00:55:20,470

 

1145
00:55:20,470 --> 00:55:22,470

 

1146
00:55:22,470 --> 00:55:27,760

 

1147
00:55:27,760 --> 00:55:33,370

 

1148
00:55:33,370 --> 00:55:35,380

 

1149
00:55:35,380 --> 00:55:39,340

 

1150
00:55:39,340 --> 00:55:41,920

 

1151
00:55:41,920 --> 00:55:49,900

 

1152
00:55:49,900 --> 00:55:54,549

 

1153
00:55:54,549 --> 00:55:58,450

 

1154
00:55:58,450 --> 00:56:04,030

 

1155
00:56:04,030 --> 00:56:08,740

 

1156
00:56:08,740 --> 00:56:11,530

 

1157
00:56:11,530 --> 00:56:13,540

 

1158
00:56:13,540 --> 00:56:15,640

 

1159
00:56:15,640 --> 00:56:20,220

 

1160
00:56:20,220 --> 00:56:22,660

 

1161
00:56:22,660 --> 00:56:30,460

 

1162
00:56:30,460 --> 00:56:33,190

 

1163
00:56:33,190 --> 00:56:36,760

 

1164
00:56:36,760 --> 00:56:38,920

 

1165
00:56:38,920 --> 00:56:46,450

 

1166
00:56:46,450 --> 00:56:49,990

 

1167
00:56:49,990 --> 00:56:52,299

 

1168
00:56:52,299 --> 00:56:55,420

 

1169
00:56:55,420 --> 00:57:01,440

 

1170
00:57:01,440 --> 00:57:05,470

 

1171
00:57:05,470 --> 00:57:07,630

 

1172
00:57:07,630 --> 00:57:11,440

 

1173
00:57:11,440 --> 00:57:12,960

 

1174
00:57:12,960 --> 00:57:16,510

 

1175
00:57:16,510 --> 00:57:18,069

 

1176
00:57:18,069 --> 00:57:19,660

 

1177
00:57:19,660 --> 00:57:22,870

 

1178
00:57:22,870 --> 00:57:25,539

 

1179
00:57:25,539 --> 00:57:28,589

 

1180
00:57:28,589 --> 00:57:32,079

 

1181
00:57:32,079 --> 00:57:34,450

 

1182
00:57:34,450 --> 00:57:37,599

 

1183
00:57:37,599 --> 00:57:41,319

 

1184
00:57:41,319 --> 00:57:43,839

 

1185
00:57:43,839 --> 00:57:45,609

 

1186
00:57:45,609 --> 00:57:48,309

 

1187
00:57:48,309 --> 00:57:50,319

 

1188
00:57:50,319 --> 00:57:51,819

 

1189
00:57:51,819 --> 00:57:51,829

 

1190
00:57:51,829 --> 00:57:53,050


1191
00:57:53,050 --> 00:57:57,640

 

1192
00:57:57,640 --> 00:57:59,620

 

1193
00:57:59,620 --> 00:58:03,340

 

1194
00:58:03,340 --> 00:58:04,980

 

1195
00:58:04,980 --> 00:58:09,910

 

1196
00:58:09,910 --> 00:58:12,160

 

1197
00:58:12,160 --> 00:58:15,040

 

1198
00:58:15,040 --> 00:58:16,930

 

1199
00:58:16,930 --> 00:58:20,590

 

1200
00:58:20,590 --> 00:58:22,330

 

1201
00:58:22,330 --> 00:58:24,430

 

1202
00:58:24,430 --> 00:58:26,380

 

1203
00:58:26,380 --> 00:58:29,320

 

1204
00:58:29,320 --> 00:58:31,330

 

1205
00:58:31,330 --> 00:58:33,040

 

1206
00:58:33,040 --> 00:58:34,840

 

1207
00:58:34,840 --> 00:58:38,020

 

1208
00:58:38,020 --> 00:58:40,140

 

1209
00:58:40,140 --> 00:58:42,490

 

1210
00:58:42,490 --> 00:58:44,770

 

1211
00:58:44,770 --> 00:58:48,690

 

1212
00:58:48,690 --> 00:58:51,340

 

1213
00:58:51,340 --> 00:58:55,030

 

1214
00:58:55,030 --> 00:58:57,460

 

1215
00:58:57,460 --> 00:59:00,970

 

1216
00:59:00,970 --> 00:59:02,950

 

1217
00:59:02,950 --> 00:59:05,470

 

1218
00:59:05,470 --> 00:59:06,760

 

1219
00:59:06,760 --> 00:59:09,520

 

1220
00:59:09,520 --> 00:59:12,700

 

1221
00:59:12,700 --> 00:59:17,560

 

1222
00:59:17,560 --> 00:59:20,010

 

1223
00:59:20,010 --> 00:59:23,290

 

1224
00:59:23,290 --> 00:59:26,290

 

1225
00:59:26,290 --> 00:59:28,930

 

1226
00:59:28,930 --> 00:59:31,450

 

1227
00:59:31,450 --> 00:59:32,800

 

1228
00:59:32,800 --> 00:59:36,160

 

1229
00:59:36,160 --> 00:59:37,630

 

1230
00:59:37,630 --> 00:59:39,970

 

1231
00:59:39,970 --> 00:59:41,950

 

1232
00:59:41,950 --> 00:59:44,050

 

1233
00:59:44,050 --> 00:59:46,840

 

1234
00:59:46,840 --> 00:59:47,980

 

1235
00:59:47,980 --> 00:59:51,520

 

1236
00:59:51,520 --> 00:59:56,110

 

1237
00:59:56,110 --> 00:59:58,300

 

1238
00:59:58,300 --> 01:00:00,100

 

1239
01:00:00,100 --> 01:00:02,020

 

1240
01:00:02,020 --> 01:00:04,120

 

1241
01:00:04,120 --> 01:00:09,199

 

1242
01:00:09,199 --> 01:00:16,229

 

1243
01:00:16,229 --> 01:00:18,329

 

1244
01:00:18,329 --> 01:00:19,739

 

1245
01:00:19,739 --> 01:00:26,910

 

1246
01:00:26,910 --> 01:00:30,630

 

1247
01:00:30,630 --> 01:00:33,660

 

1248
01:00:33,660 --> 01:00:36,959

 

1249
01:00:36,959 --> 01:00:39,809

 

1250
01:00:39,809 --> 01:00:43,319

 

1251
01:00:43,319 --> 01:00:45,059

 

1252
01:00:45,059 --> 01:00:49,440

 

1253
01:00:49,440 --> 01:00:52,469

 

1254
01:00:52,469 --> 01:00:56,459

 

1255
01:00:56,459 --> 01:00:58,049

 

1256
01:00:58,049 --> 01:01:00,239

 

1257
01:01:00,239 --> 01:01:03,059

 

1258
01:01:03,059 --> 01:01:07,589

 

1259
01:01:07,589 --> 01:01:10,319

 

1260
01:01:10,319 --> 01:01:12,209

 

1261
01:01:12,209 --> 01:01:14,609

 

1262
01:01:14,609 --> 01:01:14,619

 

1263
01:01:14,619 --> 01:01:15,689


1264
01:01:15,689 --> 01:01:19,759

 

1265
01:01:19,759 --> 01:01:25,589

 

1266
01:01:25,589 --> 01:01:27,180

 

1267
01:01:27,180 --> 01:01:28,829

 

1268
01:01:28,829 --> 01:01:32,489

 

1269
01:01:32,489 --> 01:01:35,029

 

1270
01:01:35,029 --> 01:01:37,439

 

1271
01:01:37,439 --> 01:01:40,289

 

1272
01:01:40,289 --> 01:01:44,130

 

1273
01:01:44,130 --> 01:01:45,989

 

1274
01:01:45,989 --> 01:01:48,299

 

1275
01:01:48,299 --> 01:01:49,709

 

1276
01:01:49,709 --> 01:01:52,620

 

1277
01:01:52,620 --> 01:01:54,229

 

1278
01:01:54,229 --> 01:01:56,549

 

1279
01:01:56,549 --> 01:01:59,309

 

1280
01:01:59,309 --> 01:02:01,620

 

1281
01:02:01,620 --> 01:02:04,079

 

1282
01:02:04,079 --> 01:02:06,239

 

1283
01:02:06,239 --> 01:02:08,849

 

1284
01:02:08,849 --> 01:02:11,519

 

1285
01:02:11,519 --> 01:02:15,299

 

1286
01:02:15,299 --> 01:02:19,700

 

1287
01:02:19,700 --> 01:02:21,650

 

1288
01:02:21,650 --> 01:02:25,220

 

1289
01:02:25,220 --> 01:02:27,140

 

1290
01:02:27,140 --> 01:02:29,780

 

1291
01:02:29,780 --> 01:02:33,349

 

1292
01:02:33,349 --> 01:02:35,240

 

1293
01:02:35,240 --> 01:02:37,400

 

1294
01:02:37,400 --> 01:02:41,059

 

1295
01:02:41,059 --> 01:02:44,540

 

1296
01:02:44,540 --> 01:02:47,930

 

1297
01:02:47,930 --> 01:02:50,089

 

1298
01:02:50,089 --> 01:02:50,099

 

1299
01:02:50,099 --> 01:02:51,130


1300
01:02:51,130 --> 01:02:54,670

 

1301
01:02:54,670 --> 01:02:58,430

 

1302
01:02:58,430 --> 01:03:00,740

 

1303
01:03:00,740 --> 01:03:03,980

 

1304
01:03:03,980 --> 01:03:06,220

 

1305
01:03:06,220 --> 01:03:08,750

 

1306
01:03:08,750 --> 01:03:12,140

 

1307
01:03:12,140 --> 01:03:14,329

 

1308
01:03:14,329 --> 01:03:16,670

 

1309
01:03:16,670 --> 01:03:18,200

 

1310
01:03:18,200 --> 01:03:21,700

 

1311
01:03:21,700 --> 01:03:24,650

 

1312
01:03:24,650 --> 01:03:26,359

 

1313
01:03:26,359 --> 01:03:29,000

 

1314
01:03:29,000 --> 01:03:30,650

 

1315
01:03:30,650 --> 01:03:33,940

 

1316
01:03:33,940 --> 01:03:35,839

 

1317
01:03:35,839 --> 01:03:40,640

 

1318
01:03:40,640 --> 01:03:42,380

 

1319
01:03:42,380 --> 01:03:44,569

 

1320
01:03:44,569 --> 01:03:49,460

 

1321
01:03:49,460 --> 01:03:54,349

 

1322
01:03:54,349 --> 01:03:58,309

 

1323
01:03:58,309 --> 01:04:04,309

 

1324
01:04:04,309 --> 01:04:06,079

 

1325
01:04:06,079 --> 01:04:07,490

 

1326
01:04:07,490 --> 01:04:14,660

 

1327
01:04:14,660 --> 01:04:17,839

 

1328
01:04:17,839 --> 01:04:19,849

 

1329
01:04:19,849 --> 01:04:21,920

 

1330
01:04:21,920 --> 01:04:24,589

 

1331
01:04:24,589 --> 01:04:27,589

 

1332
01:04:27,589 --> 01:04:28,720

 

1333
01:04:28,720 --> 01:04:30,920

 

1334
01:04:30,920 --> 01:04:33,110

 

1335
01:04:33,110 --> 01:04:34,640

 

1336
01:04:34,640 --> 01:04:37,730

 

1337
01:04:37,730 --> 01:04:41,990

 

1338
01:04:41,990 --> 01:04:45,740

 

1339
01:04:45,740 --> 01:04:47,960

 

1340
01:04:47,960 --> 01:04:52,130

 

1341
01:04:52,130 --> 01:04:53,870

 

1342
01:04:53,870 --> 01:04:57,860

 

1343
01:04:57,860 --> 01:04:59,510

 

1344
01:04:59,510 --> 01:05:01,310

 

1345
01:05:01,310 --> 01:05:05,300

 

1346
01:05:05,300 --> 01:05:07,190

 

1347
01:05:07,190 --> 01:05:11,270

 

1348
01:05:11,270 --> 01:05:12,740

 

1349
01:05:12,740 --> 01:05:15,890

 

1350
01:05:15,890 --> 01:05:18,350

 

1351
01:05:18,350 --> 01:05:20,180

 

1352
01:05:20,180 --> 01:05:21,950

 

1353
01:05:21,950 --> 01:05:23,900

 

1354
01:05:23,900 --> 01:05:27,040

 

1355
01:05:27,040 --> 01:05:29,810

 

1356
01:05:29,810 --> 01:05:33,580

 

1357
01:05:33,580 --> 01:05:35,840

 

1358
01:05:35,840 --> 01:05:39,710

 

1359
01:05:39,710 --> 01:05:42,080

 

1360
01:05:42,080 --> 01:05:43,790

 

1361
01:05:43,790 --> 01:05:46,100

 

1362
01:05:46,100 --> 01:05:48,110

 

1363
01:05:48,110 --> 01:05:49,700

 

1364
01:05:49,700 --> 01:05:52,070

 

1365
01:05:52,070 --> 01:05:56,000

 

1366
01:05:56,000 --> 01:06:00,080

 

1367
01:06:00,080 --> 01:06:01,520

 

1368
01:06:01,520 --> 01:06:05,630

 

1369
01:06:05,630 --> 01:06:07,160

 

1370
01:06:07,160 --> 01:06:08,210

 

1371
01:06:08,210 --> 01:06:09,350

 

1372
01:06:09,350 --> 01:06:11,750

 

1373
01:06:11,750 --> 01:06:13,400

 

1374
01:06:13,400 --> 01:06:14,540

 

1375
01:06:14,540 --> 01:06:17,960

 

1376
01:06:17,960 --> 01:06:20,030

 

1377
01:06:20,030 --> 01:06:22,610

 

1378
01:06:22,610 --> 01:06:26,300

 

1379
01:06:26,300 --> 01:06:28,040

 

1380
01:06:28,040 --> 01:06:30,110

 

1381
01:06:30,110 --> 01:06:31,700

 

1382
01:06:31,700 --> 01:06:36,610

 

1383
01:06:36,610 --> 01:06:40,670

 

1384
01:06:40,670 --> 01:06:42,260

 

1385
01:06:42,260 --> 01:06:43,370

 

1386
01:06:43,370 --> 01:06:45,680

 

1387
01:06:45,680 --> 01:06:47,529

 

1388
01:06:47,529 --> 01:06:49,819

 

1389
01:06:49,819 --> 01:06:51,740

 

1390
01:06:51,740 --> 01:06:54,380

 

1391
01:06:54,380 --> 01:06:56,480

 

1392
01:06:56,480 --> 01:06:59,450

 

1393
01:06:59,450 --> 01:07:01,720

 

1394
01:07:01,720 --> 01:07:06,200

 

1395
01:07:06,200 --> 01:07:08,749

 

1396
01:07:08,749 --> 01:07:10,700

 

1397
01:07:10,700 --> 01:07:12,890

 

1398
01:07:12,890 --> 01:07:14,480

 

1399
01:07:14,480 --> 01:07:17,089

 

1400
01:07:17,089 --> 01:07:20,829

 

1401
01:07:20,829 --> 01:07:23,509

 

1402
01:07:23,509 --> 01:07:25,400

 

1403
01:07:25,400 --> 01:07:27,559

 

1404
01:07:27,559 --> 01:07:29,150

 

1405
01:07:29,150 --> 01:07:32,059

 

1406
01:07:32,059 --> 01:07:34,519

 

1407
01:07:34,519 --> 01:07:35,960

 

1408
01:07:35,960 --> 01:07:38,269

 

1409
01:07:38,269 --> 01:07:41,200

 

1410
01:07:41,200 --> 01:07:45,019

 

1411
01:07:45,019 --> 01:07:48,019

 

1412
01:07:48,019 --> 01:07:50,539

 

1413
01:07:50,539 --> 01:07:53,180

 

1414
01:07:53,180 --> 01:07:56,120

 

1415
01:07:56,120 --> 01:07:59,029

 

1416
01:07:59,029 --> 01:08:02,990

 

1417
01:08:02,990 --> 01:08:04,849

 

1418
01:08:04,849 --> 01:08:08,150

 

1419
01:08:08,150 --> 01:08:12,740

 

1420
01:08:12,740 --> 01:08:16,550

 

1421
01:08:16,550 --> 01:08:19,999

 

1422
01:08:19,999 --> 01:08:22,550

 

1423
01:08:22,550 --> 01:08:26,780

 

1424
01:08:26,780 --> 01:08:29,269

 

1425
01:08:29,269 --> 01:08:31,999

 

1426
01:08:31,999 --> 01:08:34,129

 

1427
01:08:34,129 --> 01:08:37,430

 

1428
01:08:37,430 --> 01:08:41,110

 

1429
01:08:41,110 --> 01:08:45,800

 

1430
01:08:45,800 --> 01:08:49,700

 

1431
01:08:49,700 --> 01:08:54,559

 

1432
01:08:54,559 --> 01:08:58,370

 

1433
01:08:58,370 --> 01:08:59,360

 

1434
01:08:59,360 --> 01:09:00,740

 

1435
01:09:00,740 --> 01:09:04,010

 

1436
01:09:04,010 --> 01:09:07,459

 

1437
01:09:07,459 --> 01:09:09,919

 

1438
01:09:09,919 --> 01:09:12,919

 

1439
01:09:12,919 --> 01:09:15,800

 

1440
01:09:15,800 --> 01:09:18,019

 

1441
01:09:18,019 --> 01:09:19,999

 

1442
01:09:19,999 --> 01:09:39,580

 

1443
01:09:39,580 --> 01:09:39,590

 

1444
01:09:39,590 --> 01:09:45,349


1445
01:09:45,349 --> 01:09:47,450

 

1446
01:09:47,450 --> 01:09:51,140

 

1447
01:09:51,140 --> 01:09:55,310

 

1448
01:09:55,310 --> 01:09:58,670

 

1449
01:09:58,670 --> 01:10:02,240

 

1450
01:10:02,240 --> 01:10:05,110

 

1451
01:10:05,110 --> 01:10:08,240

 

1452
01:10:08,240 --> 01:10:10,100

 

1453
01:10:10,100 --> 01:10:11,750

 

1454
01:10:11,750 --> 01:10:11,760

 

1455
01:10:11,760 --> 01:10:12,460


1456
01:10:12,460 --> 01:10:16,420

 

1457
01:10:16,420 --> 01:10:18,830

 

1458
01:10:18,830 --> 01:10:20,900

 

1459
01:10:20,900 --> 01:10:24,590

 

1460
01:10:24,590 --> 01:10:26,870

 

1461
01:10:26,870 --> 01:10:32,350

 

1462
01:10:32,350 --> 01:10:38,690

 

1463
01:10:38,690 --> 01:10:41,810

 

1464
01:10:41,810 --> 01:10:43,280

 

1465
01:10:43,280 --> 01:10:46,660

 

1466
01:10:46,660 --> 01:10:50,030

 

1467
01:10:50,030 --> 01:10:54,830

 

1468
01:10:54,830 --> 01:10:56,410

 

1469
01:10:56,410 --> 01:11:00,860

 

1470
01:11:00,860 --> 01:11:04,490

 

1471
01:11:04,490 --> 01:11:07,340

 

1472
01:11:07,340 --> 01:11:09,980

 

1473
01:11:09,980 --> 01:11:12,770

 

1474
01:11:12,770 --> 01:11:15,800

 

1475
01:11:15,800 --> 01:11:18,350

 

1476
01:11:18,350 --> 01:11:22,370

 

1477
01:11:22,370 --> 01:11:24,590

 

1478
01:11:24,590 --> 01:11:26,270

 

1479
01:11:26,270 --> 01:11:27,710

 

1480
01:11:27,710 --> 01:11:31,010

 

1481
01:11:31,010 --> 01:11:35,180

 

1482
01:11:35,180 --> 01:11:37,760

 

1483
01:11:37,760 --> 01:11:41,780

 

1484
01:11:41,780 --> 01:11:43,610

 

1485
01:11:43,610 --> 01:11:46,490

 

1486
01:11:46,490 --> 01:11:48,920

 

1487
01:11:48,920 --> 01:11:51,290

 

1488
01:11:51,290 --> 01:11:53,300

 

1489
01:11:53,300 --> 01:11:55,880

 

1490
01:11:55,880 --> 01:11:57,710

 

1491
01:11:57,710 --> 01:11:59,240

 

1492
01:11:59,240 --> 01:12:01,670

 

1493
01:12:01,670 --> 01:12:03,110

 

1494
01:12:03,110 --> 01:12:05,420

 

1495
01:12:05,420 --> 01:12:07,520

 

1496
01:12:07,520 --> 01:12:10,840

 

1497
01:12:10,840 --> 01:12:13,280

 

1498
01:12:13,280 --> 01:12:14,750

 

1499
01:12:14,750 --> 01:12:18,470

 

1500
01:12:18,470 --> 01:12:20,240

 

1501
01:12:20,240 --> 01:12:24,100

 

1502
01:12:24,100 --> 01:12:26,030

 

1503
01:12:26,030 --> 01:12:28,310

 

1504
01:12:28,310 --> 01:12:30,110

 

1505
01:12:30,110 --> 01:12:31,730

 

1506
01:12:31,730 --> 01:12:34,030

 

1507
01:12:34,030 --> 01:12:39,920

 

1508
01:12:39,920 --> 01:12:41,750

 

1509
01:12:41,750 --> 01:12:46,010

 

1510
01:12:46,010 --> 01:12:47,510

 

1511
01:12:47,510 --> 01:12:50,480

 

1512
01:12:50,480 --> 01:12:52,820

 

1513
01:12:52,820 --> 01:12:55,100

 

1514
01:12:55,100 --> 01:12:56,780

 

1515
01:12:56,780 --> 01:12:59,330

 

1516
01:12:59,330 --> 01:13:00,830

 

1517
01:13:00,830 --> 01:13:05,590

 

1518
01:13:05,590 --> 01:13:14,110

 

1519
01:13:14,110 --> 01:13:18,140

 

1520
01:13:18,140 --> 01:13:22,520

 

1521
01:13:22,520 --> 01:13:24,830

 

1522
01:13:24,830 --> 01:13:26,150

 

1523
01:13:26,150 --> 01:13:32,000

 

1524
01:13:32,000 --> 01:13:34,400

 

1525
01:13:34,400 --> 01:13:35,690

 

1526
01:13:35,690 --> 01:13:57,640

 

1527
01:13:57,640 --> 01:14:01,330

 

1528
01:14:01,330 --> 01:14:05,020

 

1529
01:14:05,020 --> 01:14:08,170

 

1530
01:14:08,170 --> 01:14:09,820

 

1531
01:14:09,820 --> 01:14:18,370

 

1532
01:14:18,370 --> 01:14:20,320

 

1533
01:14:20,320 --> 01:14:24,820

 

1534
01:14:24,820 --> 01:14:26,590

 

1535
01:14:26,590 --> 01:14:28,210

 

1536
01:14:28,210 --> 01:14:36,910

 

1537
01:14:36,910 --> 01:14:42,580

 

1538
01:14:42,580 --> 01:14:45,520

 

1539
01:14:45,520 --> 01:14:56,790

 

1540
01:14:56,790 --> 01:14:59,370

 

1541
01:14:59,370 --> 01:15:02,370

 

1542
01:15:02,370 --> 01:15:06,000

 

1543
01:15:06,000 --> 01:15:10,650

 

1544
01:15:10,650 --> 01:15:10,660

 

1545
01:15:10,660 --> 01:15:12,810


1546
01:15:12,810 --> 01:15:15,090

 

1547
01:15:15,090 --> 01:15:17,640

 

1548
01:15:17,640 --> 01:15:19,920

 

1549
01:15:19,920 --> 01:15:23,640

 

1550
01:15:23,640 --> 01:15:27,300

 

1551
01:15:27,300 --> 01:15:28,320

 

1552
01:15:28,320 --> 01:15:30,870

 

1553
01:15:30,870 --> 01:15:34,710

 

1554
01:15:34,710 --> 01:15:36,510

 

1555
01:15:36,510 --> 01:15:49,729

 

1556
01:15:49,729 --> 01:15:51,650

 

1557
01:15:51,650 --> 01:15:55,310

 

1558
01:15:55,310 --> 01:15:58,760

 

1559
01:15:58,760 --> 01:16:01,100

 

1560
01:16:01,100 --> 01:16:02,540

 

1561
01:16:02,540 --> 01:16:14,229

 

1562
01:16:14,229 --> 01:16:16,910

 

1563
01:16:16,910 --> 01:16:20,080

 

1564
01:16:20,080 --> 01:16:23,930

 

1565
01:16:23,930 --> 01:16:25,820

 

1566
01:16:25,820 --> 01:16:30,350

 

1567
01:16:30,350 --> 01:16:32,090

 

1568
01:16:32,090 --> 01:16:34,220

 

1569
01:16:34,220 --> 01:16:37,670

 

1570
01:16:37,670 --> 01:16:40,850

 

1571
01:16:40,850 --> 01:16:43,040

 

1572
01:16:43,040 --> 01:16:47,870

 

1573
01:16:47,870 --> 01:16:49,610

 

1574
01:16:49,610 --> 01:16:52,610

 

1575
01:16:52,610 --> 01:16:56,360

 

1576
01:16:56,360 --> 01:16:58,760

 

1577
01:16:58,760 --> 01:17:01,250

 

1578
01:17:01,250 --> 01:17:04,280

 

1579
01:17:04,280 --> 01:17:10,250

 

1580
01:17:10,250 --> 01:17:13,430

 

1581
01:17:13,430 --> 01:17:16,700

 

1582
01:17:16,700 --> 01:17:19,160

 

1583
01:17:19,160 --> 01:17:22,580

 

1584
01:17:22,580 --> 01:17:26,300

 

1585
01:17:26,300 --> 01:17:28,850

 

1586
01:17:28,850 --> 01:17:32,720

 

1587
01:17:32,720 --> 01:17:34,190

 

1588
01:17:34,190 --> 01:17:36,590

 

1589
01:17:36,590 --> 01:17:42,260

 

1590
01:17:42,260 --> 01:17:45,729

 

1591
01:17:45,729 --> 01:17:49,130

 

1592
01:17:49,130 --> 01:17:51,200

 

1593
01:17:51,200 --> 01:17:53,300

 

1594
01:17:53,300 --> 01:17:55,160

 

1595
01:17:55,160 --> 01:17:58,220

 

1596
01:17:58,220 --> 01:18:01,220

 

1597
01:18:01,220 --> 01:18:03,390

 

1598
01:18:03,390 --> 01:18:06,720

 

1599
01:18:06,720 --> 01:18:08,940

 

1600
01:18:08,940 --> 01:18:10,920

 

1601
01:18:10,920 --> 01:18:13,830

 

1602
01:18:13,830 --> 01:18:17,070

 

1603
01:18:17,070 --> 01:18:21,000

 

1604
01:18:21,000 --> 01:18:23,850

 

1605
01:18:23,850 --> 01:18:25,650

 

1606
01:18:25,650 --> 01:18:27,420

 

1607
01:18:27,420 --> 01:18:30,180

 

1608
01:18:30,180 --> 01:18:32,010

 

1609
01:18:32,010 --> 01:18:35,130

 

1610
01:18:35,130 --> 01:18:38,100

 

1611
01:18:38,100 --> 01:18:40,170

 

1612
01:18:40,170 --> 01:18:41,850

 

1613
01:18:41,850 --> 01:18:46,380

 

1614
01:18:46,380 --> 01:18:49,140

 

1615
01:18:49,140 --> 01:18:52,050

 

1616
01:18:52,050 --> 01:18:53,790

 

1617
01:18:53,790 --> 01:18:55,320

 

1618
01:18:55,320 --> 01:18:57,090

 

1619
01:18:57,090 --> 01:18:59,460

 

1620
01:18:59,460 --> 01:19:01,800

 

1621
01:19:01,800 --> 01:19:03,780

 

1622
01:19:03,780 --> 01:19:06,900

 

1623
01:19:06,900 --> 01:19:08,340

 

1624
01:19:08,340 --> 01:19:11,220

 

1625
01:19:11,220 --> 01:19:12,570

 

1626
01:19:12,570 --> 01:19:14,130

 

1627
01:19:14,130 --> 01:19:16,560

 

1628
01:19:16,560 --> 01:19:18,660

 

1629
01:19:18,660 --> 01:19:22,710

 

1630
01:19:22,710 --> 01:19:25,890

 

1631
01:19:25,890 --> 01:19:27,920

 

1632
01:19:27,920 --> 01:19:30,810

 

1633
01:19:30,810 --> 01:19:32,670

 

1634
01:19:32,670 --> 01:19:34,530

 

1635
01:19:34,530 --> 01:19:40,230

 

1636
01:19:40,230 --> 01:19:42,150

 

1637
01:19:42,150 --> 01:19:43,650

 

1638
01:19:43,650 --> 01:19:46,950

 

1639
01:19:46,950 --> 01:19:49,610

 

1640
01:19:49,610 --> 01:19:52,500

 

1641
01:19:52,500 --> 01:19:54,360

 

1642
01:19:54,360 --> 01:19:58,290

 

1643
01:19:58,290 --> 01:20:00,750

 

1644
01:20:00,750 --> 01:20:02,160

 

1645
01:20:02,160 --> 01:20:04,350

 

1646
01:20:04,350 --> 01:20:07,380

 

1647
01:20:07,380 --> 01:20:10,350

 

1648
01:20:10,350 --> 01:20:12,960

 

1649
01:20:12,960 --> 01:20:14,090

 

1650
01:20:14,090 --> 01:20:16,040

 

1651
01:20:16,040 --> 01:20:19,010

 

1652
01:20:19,010 --> 01:20:20,480

 

1653
01:20:20,480 --> 01:20:23,330

 

1654
01:20:23,330 --> 01:20:24,590

 

1655
01:20:24,590 --> 01:20:26,090

 

1656
01:20:26,090 --> 01:20:30,230

 

1657
01:20:30,230 --> 01:20:33,580

 

1658
01:20:33,580 --> 01:20:36,560

 

1659
01:20:36,560 --> 01:20:42,670

 

1660
01:20:42,670 --> 01:20:42,680

 

1661
01:20:42,680 --> 01:20:44,060


1662
01:20:44,060 --> 01:20:49,880

 

1663
01:20:49,880 --> 01:20:54,170

 

1664
01:20:54,170 --> 01:20:56,690

 

1665
01:20:56,690 --> 01:20:59,300

 

1666
01:20:59,300 --> 01:20:59,310

 

1667
01:20:59,310 --> 01:20:59,800


1668
01:20:59,800 --> 01:21:04,730

 

1669
01:21:04,730 --> 01:21:07,250

 

1670
01:21:07,250 --> 01:21:11,390

 

1671
01:21:11,390 --> 01:21:14,900

 

1672
01:21:14,900 --> 01:21:25,640

 

1673
01:21:25,640 --> 01:21:25,650

 

1674
01:21:25,650 --> 01:21:27,710