EM

Syllabus

EM.Syllabus History

Hide minor edits - Show changes to output

May 11, 2017, at 03:52 PM EST by 129.24.246.253 -
Changed line 85 from:
* Participation  10 pts
to:
* Participation  15 pts
Changed line 88 from:
* Paper discussion 15 pts
to:
* Paper presentation 10 pts
April 14, 2017, at 02:53 PM EST by 129.24.247.210 -
Added lines 158-160:
* '''Supporting Exploratory Hypothesis Testing and Analysis''' [[http://dl.acm.org/citation.cfm?id=2701430&CFID=713582597&CFTOKEN=71055008|Link]]
-> ''Guimei Liu, Haojun Zhang, Mengling Feng, Limsoon Wong, and See-Kiong Ng. 2015. Supporting Exploratory Hypothesis Testing and Analysis. ACM Trans. Knowl. Discov. Data 9, 4, Article 31 (June 2015), 24 pages.''

Deleted lines 163-164:
* '''Supporting Exploratory Hypothesis Testing and Analysis''' [[http://dl.acm.org/citation.cfm?id=2701430&CFID=713582597&CFTOKEN=71055008|Link]]
-> ''Guimei Liu, Haojun Zhang, Mengling Feng, Limsoon Wong, and See-Kiong Ng. 2015. Supporting Exploratory Hypothesis Testing and Analysis. ACM Trans. Knowl. Discov. Data 9, 4, Article 31 (June 2015), 24 pages.''
April 14, 2017, at 02:52 PM EST by 129.24.247.210 -
Changed lines 203-204 from:
* '''Towards effective adaptive random testing for higher-dimensional input domains''' [[http://dl.acm.org/citation.cfm?id=1144323&CFID=713582597&CFTOKEN=71055008 |Link]]
-> ''Johannes Mayer. 2006. Towards effective adaptive random testing for higher-dimensional input domains. In Proceedings of the 8th annual conference on Genetic and evolutionary computation (GECCO '06). ACM, New York, NY, USA, 1955-1956. ''
to:
* '''Scientific Benchmarking of Parallel Computing Systems''' [[http://htor.inf.ethz.ch/publications/index.php?pub=222 |Link]]
-> ''JT. Hoefler, R. Belli. 2015. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15). ACM, ISBN: 978-1-4503-3723-6, Nov. 2015. ''
April 14, 2017, at 02:49 PM EST by 129.24.247.210 -
Changed line 187 from:
to:
* [[https://www.youtube.com/watch?v=HwEpXIWAWTU| Scientific Benchmarking of Parallel Computing Systems talk]]
Added lines 199-202:

* '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[http://dl.acm.org/citation.cfm?id=1050204|Link]]
-> ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''

Deleted lines 225-227:

* '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[http://dl.acm.org/citation.cfm?id=1050204|Link]]
-> ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''
Changed lines 235-237 from:
!!! Analysis

to:
!!! Analysis and Interpretation

Changed line 244 from:
* Interpretation
to:
* Interpretation of results
Added lines 179-203:
!!! Measuring and Testing

!!!! Measuring computer performance

* Metrics of performance
* Average performance and variability
* Errors in experimental measurements
* Comparing alternatives


!!!! Testing

* Statistical methods
* Monte Carlo testing
* Bootstrap
* Randomization tests
* Jackknife and cross validation
* Non parametric tests

!!!! Reading:
* '''Towards effective adaptive random testing for higher-dimensional input domains''' [[http://dl.acm.org/citation.cfm?id=1144323&CFID=713582597&CFTOKEN=71055008 |Link]]
-> ''Johannes Mayer. 2006. Towards effective adaptive random testing for higher-dimensional input domains. In Proceedings of the 8th annual conference on Genetic and evolutionary computation (GECCO '06). ACM, New York, NY, USA, 1955-1956. ''


Deleted lines 232-253:
!!! Measuring and Testing

!!!! Measuring computer performance

* Metrics of performance
* Average performance and variability
* Errors in experimental measurements
* Comparing alternatives


!!!! Testing

* Statistical methods
* Monte Carlo testing
* Bootstrap
* Randomization tests
* Jackknife and cross validation
* Non parametric tests

!!!! Reading:
* '''Towards effective adaptive random testing for higher-dimensional input domains''' [[http://dl.acm.org/citation.cfm?id=1144323&CFID=713582597&CFTOKEN=71055008 |Link]]
-> ''Johannes Mayer. 2006. Towards effective adaptive random testing for higher-dimensional input domains. In Proceedings of the 8th annual conference on Genetic and evolutionary computation (GECCO '06). ACM, New York, NY, USA, 1955-1956. ''
Changed lines 58-59 from:
Homework will be assigned to reinforce concepts covered in class. Homework may include exercises, coding, or data analysis. Homework accounts for 20% of your final grade and no late homework will be accepted.
to:
Homework will be assigned to reinforce concepts covered in class. Homework may include exercises, coding, or data analysis. Homework accounts for 10% of your final grade and no late homework will be accepted.
Changed lines 65-74 from:
* Paper reviews are due at 8 am on the day of the paper discussion and account for 15% of your final grade. Late paper reviews won't be accepted.


!!! Weekly quizzes

Readings in this syllabus are mandatory, you can expect short individual quizzes every week. Those quizzes will be graded and they account for 10% of your final grade.

Make up assignments for missed quizzes will be given only when a valid reason was provided by the student in advance.


to:
* Paper reviews are due at 8 am on the day of the paper discussion and account for 25% of your final grade. Late paper reviews won't be accepted.


Changed lines 86-87 from:
* Homework  20 pts
* Paper reviews 15 pts
to:
* Homework    10 pts
* Paper reviews 25 pts
Changed lines 89-90 from:
* Quizzes 10 pts
* Exams
30 pts
to:
* Exams 40 pts
Changed line 10 from:
Most lectures will be based on the following '''optional''' textbooks and additional [[Resources|reading material]]:
to:
Most lectures will be loosely based on the following '''optional''' textbooks and additional [[Resources|reading material]]:
Changed lines 8-10 from:
!!! Textbook

Most lectures will be based on the following '''optional''' textbooks:
to:
!!! Textbooks

Most lectures will be based on the following '''optional''' textbooks and additional [[Resources|reading material]]:
Changed lines 10-12 from:
Most lectures will be based on the following optional textbooks:

TBD
to:
Most lectures will be based on the following '''optional''' textbooks:

'''Empirical Methods for Artificial Intelligence'''
* Author: Paul Cohen
* Publisher: A Bradford Book (August 3, 1995)
* ISBN-10: 0262032252
* ISBN-13: 978-0262032254

'''A Guide to Experimental Algorithmics'''
* Author: Catherine McGeoch
* Publisher: Cambridge University Press; 1 edition (January 30, 2012)
* ISBN-10: 0521173019
* ISBN-13: 978-0521173018

'''Measuring Computer Performance: A Practitioner's Guide'''
* Author: David J. Lilja
* Publisher: Cambridge University Press; 1 edition (September 8, 2005)
* ISBN-10: 0521646707
* ISBN-13: 978-0521646703
Changed lines 199-200 from:
!!! Measuring computer performance
to:
!!! Measuring and Testing

!
!!! Measuring computer performance
Changed lines 209-210 from:
!!! Testing
to:
!!!! Testing
Changed line 243 from:
!!! Final remarks
to:
!!! Final Considerations
Changed lines 140-142 from:
 

to:
Added line 186:
Added lines 197-204:


!!! Measuring computer performance

* Metrics of performance
* Average performance and variability
* Errors in experimental measurements
* Comparing alternatives
Added lines 120-121:
* '''Experimental Computer Science: The Need for a Cultural Change''' [[http://www.cs.huji.ac.il/~feit/papers/exp05.pdf | Link]]
-> ''Dror G. Feitelson; Experimental Computer Science: The Need for a Cultural Change; 2006''
Added lines 206-209:

!!!! Reading:
* '''Towards effective adaptive random testing for higher-dimensional input domains''' [[http://dl.acm.org/citation.cfm?id=1144323&CFID=713582597&CFTOKEN=71055008 |Link]]
-> ''Johannes Mayer. 2006. Towards effective adaptive random testing for higher-dimensional input domains. In Proceedings of the 8th annual conference on Genetic and evolutionary computation (GECCO '06). ACM, New York, NY, USA, 1955-1956. ''
Changed line 149 from:
* '''A Multiple Test Correction for Streams and Cascades of Statistical Hypothesis Tests''' [[http://dl.acm.org/citation.cfm?id=2939775&CFID=713582597&CFTOKEN=71055008|Link]]
to:
* '''A Multiple Test Correction for Streams and Cascades of Statistical Hypothesis Tests''' [[http://dl.acm.org/citation.cfm?id=2939775|Link]]
Added lines 151-153:

* '''Supporting Exploratory Hypothesis Testing and Analysis''' [[http://dl.acm.org/citation.cfm?id=2701430&CFID=713582597&CFTOKEN=71055008|Link]]
-> ''Guimei Liu, Haojun Zhang, Mengling Feng, Limsoon Wong, and See-Kiong Ng. 2015. Supporting Exploratory Hypothesis Testing and Analysis. ACM Trans. Knowl. Discov. Data 9, 4, Article 31 (June 2015), 24 pages.''
Added lines 148-150:
!!!! Reading:
* '''A Multiple Test Correction for Streams and Cascades of Statistical Hypothesis Tests''' [[http://dl.acm.org/citation.cfm?id=2939775&CFID=713582597&CFTOKEN=71055008|Link]]
-> ''Geoffrey I. Webb and Franηois Petitjean. 2016. A Multiple Test Correction for Streams and Cascades of Statistical Hypothesis Tests. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). ACM, New York, NY, USA, 1255-1264.''
Added lines 133-137:
* '''Visualising a state-wide patient data collection: a case study to expand the audience for healthcare data''' [[http://dl.acm.org/citation.cfm?id=1862312&CFID=713582597&CFTOKEN=71055008|Link]]
-> ''Wei Luo, Marcus Gallagher, Di O'Kane, Jason Connor, Mark Dooris, Col Roberts, Lachlan Mortimer, and Janet Wiles. 2010. Visualising a state-wide patient data collection: a case study to expand the audience for healthcare data. In Proceedings of the Fourth Australasian Workshop on Health Informatics and Knowledge Management - Volume 108 (HIKM '10), Anthony Maeder and David Hansen (Eds.), Vol. 108. Australian Computer Society, Inc., Darlinghurst, Australia, Australia,''


Changed line 117 from:
* '''Computing as a Science: A Survey of Competing Viewpoints'''
to:
* '''Computing as a Science: A Survey of Competing Viewpoints''' [[http://link.springer.com/article/10.1007/s11023-011-9240-4 | Link]]
Changed lines 131-132 from:
* '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[[[http://dl.acm.org/citation.cfm?id=1050204|Link]]
-> ''Fabrizio Petrini
, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''
to:
* '''Looking at Data''' [[http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4536092|Link]]
-> ''Dror G. Feitelson; Looking at data. International Parallel and Distributed Processing Symposium
, 2008, 2008''
Changed lines 154-158 from:
to:
!!!! Reading:
* '''How to Design Computer Security Experiments''' [[http://link.springer.com/chapter/10.1007/978-0-387-73269-5_19|Link]]
-> ''Peisert S., Bishop M. (2007) How to Design Computer Security Experiments. In: Fifth World Conference on Information Security Education. IFIP — International Federation for Information Processing, vol 237. Springer''

Added lines 176-179:

* '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[http://dl.acm.org/citation.cfm?id=1050204|Link]]
-> ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''

Changed line 118 from:
* ''Matti Tedre, Computing as a Science: A Survey of Competing Viewpoints, Minds and Machines, August 2011, Volume 21, Issue 3, pp 361–387''
to:
-> ''Matti Tedre, Computing as a Science: A Survey of Competing Viewpoints, Minds and Machines, August 2011, Volume 21, Issue 3, pp 361–387''
Changed lines 131-132 from:
** '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[[[http://dl.acm.org/citation.cfm?id=1050204|Link]]
** ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''
to:
* '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[[[http://dl.acm.org/citation.cfm?id=1050204|Link]]
-> ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''
Changed lines 172-179 from:
** '''On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?''' [[http://ieeexplore.ieee.org/document/4604660/|Link]]
** ''Edi Shmueli, Dror G. Feitelson; On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?; in IEEE Transactions on Parallel and Distributed Systems, Vol 20, issue 7''


** '''Towards a Discipline of Experimental Algorithmics''' [[https://www.researchgate.net/publication/37441549_Towards_a_discipline_of_experimental_algorithmics|Link]]
** ''Bernard M. E. Moret; Towards a Discipline of Experimental Algorithmics; Communications of the ACM, 1999 ''

to:
* '''On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?''' [[http://ieeexplore.ieee.org/document/4604660/|Link]]
-> ''Edi Shmueli, Dror G. Feitelson; On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?; in IEEE Transactions on Parallel and Distributed Systems, Vol 20, issue 7''

* '''Towards a Discipline of Experimental Algorithmics''' [[https://www.researchgate.net/publication/37441549_Towards_a_discipline_of_experimental_algorithmics|Link]]
-> ''Bernard M. E. Moret; Towards a Discipline of Experimental Algorithmics; Communications of the ACM, 1999 ''

Changed lines 198-199 from:
to:
* Interpretation
Changed lines 201-214 from:
** '''Multiple Comparisons in Induction Algorithms''' [[http://link.springer.com/article/10.1023/A:1007631014630 |Link]]
** ''Jensen, D.D. & Cohen, P.R. Machine Learning (2000) 38: 309.''




* Interpretation

!!!! Reading:
** '''Twenty tips for interpreting scientific
claims''' [[http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183 |Link]]
** ''William J. Sutherland, David Spiegelhalter, Mark Burgman; Twenty tips for interpreting scientific claims; Nature 503, 335–337 (21 November 2013) ''


to:
* '''Multiple Comparisons in Induction Algorithms''' [[http://link.springer.com/article/10.1023/A:1007631014630 |Link]]
-> ''Jensen, D.D. & Cohen, P.R. Machine Learning (2000) 38: 309.''

* '''Twenty tips for interpreting scientific claims''' [[http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183 |Link]]
-> ''William J. Sutherland, David Spiegelhalter, Mark Burgman; Twenty tips for interpreting scientific claims; Nature 503, 335–337 (21 November 2013) ''


Changed lines 215-219 from:
** '''From Repeatability to Reproducibility and Corroboration''' [[http://dl.acm.org/citation.cfm?id=2723875|Link]]
** ''Dror G. Feitelson. 2015. From Repeatability to Reproducibility and Corroboration. SIGOPS Oper. Syst. Rev. 49, 1 (January 2015), 3-11.''

** '''Ethical Issues in Empirical Studies of Software Engineering''' [[http://dl.acm.org/citation.cfm?id=631303 |Link]]
** ''Janice Singer and Norman G. Vinson. 2002. Ethical Issues in Empirical Studies of Software Engineering. IEEE Trans. Softw. Eng. 28, 12 (December 2002), 1171-1180.''
to:
* '''From Repeatability to Reproducibility and Corroboration''' [[http://dl.acm.org/citation.cfm?id=2723875|Link]]
-> ''Dror G. Feitelson. 2015. From Repeatability to Reproducibility and Corroboration. SIGOPS Oper. Syst. Rev. 49, 1 (January 2015), 3-11.''

* '''Ethical Issues in Empirical Studies of Software Engineering''' [[http://dl.acm.org/citation.cfm?id=631303 |Link]]
-> ''Janice Singer and Norman G. Vinson. 2002. Ethical Issues in Empirical Studies of Software Engineering. IEEE Trans. Softw. Eng. 28, 12 (December 2002), 1171-1180.''
Changed lines 131-132 from:
* '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[[[http://dl.acm.org/citation.cfm?id=1050204|Link]]
* ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''
to:
** '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[[[http://dl.acm.org/citation.cfm?id=1050204|Link]]
** ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''
Changed lines 171-179 from:
->!!!! Reading:
->* '''On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?''' [[http://ieeexplore.ieee.org/document/4604660/|Link]]
->* ''Edi Shmueli, Dror G. Feitelson; On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?; in IEEE Transactions on Parallel and Distributed Systems, Vol 20, issue 7''

!!!! Reading:
*
'''Towards a Discipline of Experimental Algorithmics''' [[https://www.researchgate.net/publication/37441549_Towards_a_discipline_of_experimental_algorithmics|Link]]
* ''Bernard M. E. Moret; Towards a Discipline of Experimental Algorithmics; Communications of the ACM, 1999 ''

to:
!!!! Readings:
** '''On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?''' [[http://ieeexplore.ieee.org/document/4604660/|Link]]
** ''Edi Shmueli, Dror G. Feitelson; On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?; in IEEE Transactions on Parallel and Distributed Systems, Vol 20, issue 7''


** '''Towards a Discipline of Experimental Algorithmics''' [[https://www.researchgate.net/publication/37441549_Towards_a_discipline_of_experimental_algorithmics|Link]]
** ''Bernard M. E. Moret; Towards a Discipline of Experimental Algorithmics; Communications of the ACM, 1999 ''

Changed lines 201-206 from:
* '''Multiple Comparisons in Induction Algorithms''' [[http://link.springer.com/article/10.1023/A:1007631014630 |Link]]
* ''Jensen, D.D. & Cohen, P.R. Machine Learning (2000) 38: 309.''



to:
** '''Multiple Comparisons in Induction Algorithms''' [[http://link.springer.com/article/10.1023/A:1007631014630 |Link]]
** ''Jensen, D.D. & Cohen, P.R. Machine Learning (2000) 38: 309.''



Changed lines 210-214 from:
* '''Twenty tips for interpreting scientific claims''' [[http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183 |Link]]
* ''William J. Sutherland, David Spiegelhalter, Mark Burgman; Twenty tips for interpreting scientific claims; Nature 503, 335–337 (21 November 2013) ''


to:
** '''Twenty tips for interpreting scientific claims''' [[http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183 |Link]]
** ''William J. Sutherland, David Spiegelhalter, Mark Burgman; Twenty tips for interpreting scientific claims; Nature 503, 335–337 (21 November 2013) ''


Changed lines 220-227 from:
!!!! Reading:
* '''From Repeatability to Reproducibility and Corroboration''' [[http://dl.acm.org/citation.cfm?id=2723875|Link]]
* ''Dror G. Feitelson. 2015. From Repeatability to Reproducibility and Corroboration. SIGOPS Oper. Syst. Rev. 49, 1 (January 2015), 3-11.''


!!!! Reading:
*
'''Ethical Issues in Empirical Studies of Software Engineering''' [[http://dl.acm.org/citation.cfm?id=631303 |Link]]
* ''Janice Singer and Norman G. Vinson. 2002. Ethical Issues in Empirical Studies of Software Engineering. IEEE Trans. Softw. Eng. 28, 12 (December 2002), 1171-1180.''
to:
!!!! Readings:
** '''From Repeatability to Reproducibility and Corroboration''' [[http://dl.acm.org/citation.cfm?id=2723875|Link]]
** ''Dror G. Feitelson. 2015. From Repeatability to Reproducibility and Corroboration. SIGOPS Oper. Syst. Rev. 49, 1 (January 2015), 3-11.''

** '''Ethical Issues in Empirical Studies of Software Engineering''' [[http://dl.acm.org/citation.cfm?id=631303 |Link]]
** ''Janice Singer and Norman G. Vinson. 2002. Ethical Issues in Empirical Studies of Software Engineering. IEEE Trans. Softw. Eng. 28, 12 (December 2002), 1171-1180.''
Changed lines 167-171 from:
!!!! Reading:
* '''On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?''' [[http://ieeexplore.ieee.org/document/4604660/|Link]]
* ''Edi Shmueli, Dror G. Feitelson; On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?; in IEEE Transactions on Parallel and Distributed Systems, Vol 20, issue 7''

to:
Added lines 170-173:

->!!!! Reading:
->* '''On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?''' [[http://ieeexplore.ieee.org/document/4604660/|Link]]
->* ''Edi Shmueli, Dror G. Feitelson; On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?; in IEEE Transactions on Parallel and Distributed Systems, Vol 20, issue 7''
Changed line 131 from:
* '''[[http://dl.acm.org/citation.cfm?id=1050204|The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q]]'''
to:
* '''The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q''' [[[[http://dl.acm.org/citation.cfm?id=1050204|Link]]
Added lines 167-171:
!!!! Reading:
* '''On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?''' [[http://ieeexplore.ieee.org/document/4604660/|Link]]
* ''Edi Shmueli, Dror G. Feitelson; On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?; in IEEE Transactions on Parallel and Distributed Systems, Vol 20, issue 7''

Changed lines 175-179 from:
to:
!!!! Reading:
* '''Towards a Discipline of Experimental Algorithmics''' [[https://www.researchgate.net/publication/37441549_Towards_a_discipline_of_experimental_algorithmics|Link]]
* ''Bernard M. E. Moret; Towards a Discipline of Experimental Algorithmics; Communications of the ACM, 1999 ''

Changed lines 200-214 from:
to:
!!!! Reading:
* '''Multiple Comparisons in Induction Algorithms''' [[http://link.springer.com/article/10.1023/A:1007631014630 |Link]]
* ''Jensen, D.D. & Cohen, P.R. Machine Learning (2000) 38: 309.''




* Interpretation

!!!! Reading:
* '''Twenty tips for interpreting scientific claims''' [[http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183 |Link]]
* ''William J. Sutherland, David Spiegelhalter, Mark Burgman; Twenty tips for interpreting scientific claims; Nature 503, 335–337 (21 November 2013) ''


Changed lines 220-231 from:



*
* Towards a Discipline of Experimental Algorithmics
* On Simulation and Design of Parallel-Systems Schedulers
: Are We Doing the Right Thing ?

* Multiple Comparisons in Induction Algorithms
* DataMill: Rigorous Performance Evaluation Made Easy
* [[http://www
.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183 | Twenty tips for interpreting scientific claims]]
* From Repeatability to Reproducibility and Corroboration
* Ethical Issues in Empirical Studies of Software Engineering
to:
!!!! Reading:
* '''From Repeatability to Reproducibility and Corroboration''' [[http://dl.acm.org/citation.cfm?id=2723875|Link]]
* ''Dror G. Feitelson. 2015. From Repeatability to Reproducibility and Corroboration. SIGOPS Oper. Syst. Rev. 49, 1 (January 2015), 3-11
.''


!!!! Reading:
* '''Ethical Issues in Empirical Studies of Software Engineering''' [[http://dl.acm.org/citation.cfm?id=631303 |Link]]
* ''Janice Singer and Norman G. Vinson. 2002. Ethical Issues in Empirical Studies of Software Engineering. IEEE Trans. Softw. Eng. 28, 12 (December 2002), 1171-1180.''








Changed lines 130-135 from:
to:
!!!! Reading:
* '''[[http://dl.acm.org/citation.cfm?id=1050204|The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q]]'''
* ''Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC '03).''
 

Changed line 203 from:
* The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q
to:
Changed lines 117-118 from:
'''Computing as a Science: A Survey of Competing Viewpoints'''
->Matti Tedre, Computing as a Science: A Survey of Competing Viewpoints, Minds and Machines, August 2011, Volume 21, Issue 3, pp 361–387
to:
* '''Computing as a Science: A Survey of Competing Viewpoints'''
* ''Matti Tedre, Computing as a Science: A Survey of Competing Viewpoints, Minds and Machines, August 2011, Volume 21, Issue 3, pp 361–387''
Changed lines 116-117 from:
Reading: '''Computing as a Science: A Survey of Competing Viewpoints'''
to:
!!!! Reading:
'''Computing as a Science: A Survey of Competing Viewpoints'''
Changed line 117 from:
Matti Tedre, Computing as a Science: A Survey of Competing Viewpoints, Minds and Machines, August 2011, Volume 21, Issue 3, pp 361–387
to:
->Matti Tedre, Computing as a Science: A Survey of Competing Viewpoints, Minds and Machines, August 2011, Volume 21, Issue 3, pp 361–387
Changed lines 115-116 from:
!!!! Reading: '''Computing as a Science: A Survey of Competing Viewpoints'''
to:

Reading: '''Computing as a Science: A Survey of Competing Viewpoints'''
Changed lines 115-116 from:
!!!! Reading:
to:
!!!! Reading: '''Computing as a Science: A Survey of Competing Viewpoints'''
Matti Tedre, Computing as a Science: A Survey of Competing Viewpoints, Minds and Machines, August 2011, Volume 21, Issue 3, pp 361–387


Changed line 193 from:
* Computing as a Science: A Survey of Competing Viewpoints
to:
*
Changed lines 115-116 from:

to:
!!!! Reading:
Added lines 189-198:

* Computing as a Science: A Survey of Competing Viewpoints
* Towards a Discipline of Experimental Algorithmics
* On Simulation and Design of Parallel-Systems Schedulers: Are We Doing the Right Thing ?
* The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q
* Multiple Comparisons in Induction Algorithms
* DataMill: Rigorous Performance Evaluation Made Easy
* [[http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183 | Twenty tips for interpreting scientific claims]]
* From Repeatability to Reproducibility and Corroboration
* Ethical Issues in Empirical Studies of Software Engineering
Added lines 126-134:
!!! Hypothesis

* Statistical inference
* Hypothesis testing
* Sampling distributions
* Parameter estimation and confidence intervals


Deleted lines 142-150:



!!! Hypothesis

* Statistical inference
* Hypothesis testing
* Sampling distributions
* Parameter estimation and confidence intervals
Deleted lines 32-33:

Deleted lines 72-80:



!! FEEDBACK

I value student's opinions regarding the course and I will take them in consideration to make this course as exciting and engaging as possible. Thus, through the semester I will ask students formal and informal feedback. Formal feedback includes short surveys on my teaching effectiveness, preferred teaching methods, and pace of the class. Informal feedback will be in the form of polls or in-class questions regarding learning preferences. You can also leave anonymous feedback in the form of a note in my departmental mail box, under my office door, or %target=_blank%[[https://docs.google.com/spreadsheet/viewform?fromEmail=true&formkey=dHZvUElpWXJsTXV6RTAwZGlCTkYtQVE6MQ | using this form]]. Remember that it is in the best interest of the class if you bring up to my attention if something is not working properly (e.g the pace of the class is too slow, the projects are boring, my teaching style is not effective) so that I can make the corrective steps.

----------

Changed lines 93-102 from:
 
to:



!! FEEDBACK

I value student's opinions regarding the course and I will take them in consideration to make this course as exciting and engaging as possible. Thus, through the semester I will ask students formal and informal feedback. Formal feedback includes short surveys on my teaching effectiveness, preferred teaching methods, and pace of the class. Informal feedback will be in the form of polls or in-class questions regarding learning preferences. You can also leave anonymous feedback in the form of a note in my departmental mail box, under my office door, or %target=_blank%[[https://docs.google.com/spreadsheet/viewform?fromEmail=true&formkey=dHZvUElpWXJsTXV6RTAwZGlCTkYtQVE6MQ | using this form]]. Remember that it is in the best interest of the class if you bring up to my attention if something is not working properly (e.g the pace of the class is too slow, the projects are boring, my teaching style is not effective) so that I can make the corrective steps.



Changed line 100 from:
''' Please note that'''
to:
''' Please note:'''
Changed lines 32-35 from:
Change of grade to CR/NC after the semester deadline will be granted ONLY under special, documented extenuating circumstances.


to:


Added line 100:
''' Please note that'''
Added line 102:
* Change of grade to CR/NC after the semester deadline will be granted ONLY under special, documented extenuating circumstances.
Changed lines 97-98 from:
A A- B+ B B- C+ C C- F
95 93 90 85 83 80 77 75 <75
to:
A A- B+ B B- C+ C F
95 93 90 85 83 80 75 <75
Changed lines 24-25 from:
Any and all acts of plagiarism will result in an immediate dismissal from the course and an official report to the dean of students.
to:
''' Any and all acts of plagiarism will result in an immediate dismissal from the course and an official report to the dean of students. '''
Added line 60:
Deleted line 0:
Changed lines 113-116 from:
Empirical research
Types of empirical studies

to:
* Empirical research
* Types of empirical studies

Changed lines 119-125 from:
Data
Diagnosing data
Data Cleaning
Exploratory data analysis
Data Visualization

to:
* Data
* Diagnosing data
* Data Cleaning
* Exploratory data analysis
* Data Visualization

Changed lines 128-137 from:
Experimental design
Control
Sampling bias
Pilot experiments
Factorial design

Challenges and opportunities of experimental design in CS


to:
* Experimental design
* Control
* Sampling bias
* Pilot experiments
* Factorial design
* Challenges and opportunities of experimental design in CS


Changed lines 139-144 from:
Statistical inference
Hypothesis testing
Sampling distributions
Parameter estimation and confidence intervals

to:
* Statistical inference
* Hypothesis testing
* Sampling distributions
* Parameter estimation and confidence intervals

Changed lines 147-160 from:
Modeling
Linear models
Probabilistic models
Stochastic models

Simulation
Simulation of stochastic processes
Monte Carlo simulation
Discrete event simulation

Experimental algorithmics
Tuning algorithms

to:
!!!! Modeling
* Linear models
* Probabilistic models
* Stochastic models

!!!! Simulation
* Simulation of stochastic processes
* Monte Carlo simulation
* Discrete event simulation

!!!! Experimental algorithmics
* Tuning algorithms

Changed lines 163-170 from:
Statistical methods
Monte Carlo testing
Bootstrap
Randomization tests
Jackknife and cross validation
Non parametric tests

to:
* Statistical methods
* Monte Carlo testing
* Bootstrap
* Randomization tests
* Jackknife and cross validation
* Non parametric tests

Changed lines 173-180 from:
Modern statistical analysis
Descriptive, diagnostic, predictive, and prescriptive analytics
Regression analysis
Bayesian
analysis
Power analysis
Sensitivity analysis

to:

!!!!
Descriptive, diagnostic, predictive, and prescriptive analytics
* Modern statistical analysis
* Regression analysis
* Bayesian analysis
* Power analysis
*
Sensitivity analysis

Changed lines 184-185 from:
Reproducibility
Ethical issues
to:
* Reproducibility
* Ethical issues
Changed lines 1-194 from:
This course explores the design, experimentation, testing, and pitfalls of empirical research in Computer Science. In particular, students will learn how to use a data-driven approach to understand computing phenomena, formulate hypotheses, design computing experiments to test and validate or refute said hypotheses, evaluate and interpret empirical results. Overall, the goal of this course is to provide the students with the foundations of rigorous empirical research.
to:

! CS 533 - Experimental Methods in Computer Science

!! Course description:

This course explores the design, experimentation, testing, and pitfalls of empirical research in Computer Science. In particular, students will learn how to use a data-driven approach to understand computing phenomena, formulate hypotheses, design computing experiments to test and validate or refute said hypotheses, evaluate and interpret empirical results. Overall, the goal of this course is to provide the students with the foundations of rigorous empirical research.


!!! Textbook

Most lectures will be based on the following optional textbooks:

TBD

------

!! POLICIES

!!! Academic honesty:

Unless otherwise specified, you must write/code your own homework assignments. You cannot use the web to find answers to any assignment. If you do not have time to complete an assignment, it is better to submit your partial solutions than to get answers from someone else. Cheating students will be prosecuted according to University guidelines. Students should get acquainted with their rights and responsibilities as explained in the Student Code of Conduct
* %target=_blank%http://dos.unm.edu/student-conduct/academic-integrityhonesty.html


Any and all acts of plagiarism will result in an immediate dismissal from the course and an official report to the dean of students.

Instances of plagiarism include, but are not limited to: downloading code and snippets from the Internet without explicit permission from the instructor and/or without proper acknowledgment, citation, or license use; using code from a classmate or any other past or present student; quoting text directly or slightly paraphrasing from a source without proper reference; any other act of copying material and trying to make it look like it is yours.

Note that dismissal from the class means that the student will be dropped with an F from the course.

The best way of avoiding plagiarism is to start your assignments early. Whenever you feel like you cannot keep up with the course material, your instructor is happy to find a way to help you. Make an appointment or come to office hours, but DO NOT plagiarize; it is not worth it!

Change of grade to CR/NC after the semester deadline will be granted ONLY under special, documented extenuating circumstances.



!!! Class attendance:

Attendance to class is expected (read mandatory) and note taking encouraged. Important information (about exams, assignments, projects, policies) may be communicated only in the lectures. We may also cover additional material (not available in the book) during the lecture. If you miss a lecture, you should find what material was covered and if any announcement was made.


!! ASSIGNMENTS

!!! Homework

Homework will be assigned to reinforce concepts covered in class. Homework may include exercises, coding, or data analysis. Homework accounts for 20% of your final grade and no late homework will be accepted.

!!! Paper discussions

Papers will be discussed every week. Students are required to read and prepare a one page review of the paper. Each time one student will act as the discussion leader and he/she is expected to prepare slides or other adequate material for the presentation of main points in the paper. Depending on the topic, the rest of the class will participate on an open discussion or a debate of the paper.

* Presentation and discussion of the paper will be done during class time and account for 15% of your final grade.
* Paper reviews are due at 8 am on the day of the paper discussion and account for 15% of your final grade. Late paper reviews won't be accepted.


!!! Weekly quizzes

Readings in this syllabus are mandatory, you can expect short individual quizzes every week. Those quizzes will be graded and they account for 10% of your final grade.

Make up assignments for missed quizzes will be given only when a valid reason was provided by the student in advance.

!!! Daily assignments and participation

You can expect to have simple exercises every meeting. These daily assignments will be done in groups specified by the instructor and they will account for your participation grade (10% of your final grade)




!! EXAM

Exams are our formal evaluation tool. In the exams you will be tested with respect to the learning goals of this course (see the schedule below for the list of learning goals). Exams will comprise a mix of practical exercises and concepts. I don't encourage you to learn concepts and definitions by hart, but to be able to explain them with your own words and to place these concepts into the broader context they belong to. There will be one '''midterm exam on TBD''' and one '''final exam by the end of April '''

'''The exam is open notes, but only personal, hand-written notes are accepted'''. Restrictions in this matter include (but are not limited): you cannot download notes from Internet, you cannot use the electronic notes of the course, and you cannot photocopy notes from your classmates.  In fact, the key point is that they must be '''your own hand-written notes''' because I expect you to reinforce what you learned in class by writing down key concepts.





!! FEEDBACK

I value student's opinions regarding the course and I will take them in consideration to make this course as exciting and engaging as possible. Thus, through the semester I will ask students formal and informal feedback. Formal feedback includes short surveys on my teaching effectiveness, preferred teaching methods, and pace of the class. Informal feedback will be in the form of polls or in-class questions regarding learning preferences. You can also leave anonymous feedback in the form of a note in my departmental mail box, under my office door, or %target=_blank%[[https://docs.google.com/spreadsheet/viewform?fromEmail=true&formkey=dHZvUElpWXJsTXV6RTAwZGlCTkYtQVE6MQ | using this form]]. Remember that it is in the best interest of the class if you bring up to my attention if something is not working properly (e.g the pace of the class is too slow, the projects are boring, my teaching style is not effective) so that I can make the corrective steps.

----------

!! GRADING

* Participation  10 pts
* Homework  20 pts
* Paper reviews 15 pts
* Paper discussion 15 pts
* Quizzes 10 pts
* Exams 30 pts


Grades will be based on your earned points, following this grade scale. You need to get the specified number of points or more to obtain the grade from the same column. Scores will be rounded to the closest integer value.
[@
A A- B+ B B- C+ C C- F
95 93 90 85 83 80 77 75 <75
@]

* Incomplete can be assigned only for a documented medical reason

 

!! ADA:
In accordance with University Policy 2310 and the Americans with Disabilities Act (ADA), academic accommodations may be made for any student who notifies the instructor of the need for an accommodation. If you have a disability, either permanent or temporary, contact Accessibility Resource Center at 277-3506 for additional information.


----------

!! SCHEDULE

!!! Introduction
Empirical research
Types of empirical studies


!!! Observation

Data
Diagnosing data
Data Cleaning
Exploratory data analysis
Data Visualization


!!! Design

Experimental design
Control
Sampling bias
Pilot experiments
Factorial design

Challenges and opportunities of experimental design in CS



!!! Hypothesis

Statistical inference
Hypothesis testing
Sampling distributions
Parameter estimation and confidence intervals


!!! Experimentation

Modeling
Linear models
Probabilistic models
Stochastic models

Simulation
Simulation of stochastic processes
Monte Carlo simulation
Discrete event simulation

Experimental algorithmics
Tuning algorithms


!!! Testing

Statistical methods
Monte Carlo testing
Bootstrap
Randomization tests
Jackknife and cross validation
Non parametric tests


!!! Analysis

Modern statistical analysis
Descriptive, diagnostic, predictive, and prescriptive analytics
Regression analysis
Bayesian analysis
Power analysis
Sensitivity analysis


!!! Final remarks

Reproducibility
Ethical issues







Added line 1:
This course explores the design, experimentation, testing, and pitfalls of empirical research in Computer Science. In particular, students will learn how to use a data-driven approach to understand computing phenomena, formulate hypotheses, design computing experiments to test and validate or refute said hypotheses, evaluate and interpret empirical results. Overall, the goal of this course is to provide the students with the foundations of rigorous empirical research.