COMPSCI 296.2, Fall 2004
Experimental Methods in Computer Systems
Course Announcement
When: Monday and Wednesday 2:50 - 4:05 pm
Where: LSRC D243
Who: Professor Carla Ellis
Why:
If Computer Science is to live up to its name, then it is important for researchers to develop skills in the scientific method. Many studies suffer because the results are based on seriously flawed experimental technique and/or fail to provide compelling support for the claims being made. Don't let this be you!
What:
Topics will include:
- Measurement and simulation techniques -- how to choose.
- Workload issues -- characterization, workload selection (standard benchmark suites, micro benchmarks, synthetic benchmarks, workload generators).
- Appropriate metrics for the questions being asked.
- Effective experimental design -- how not to waste your time.
- Dealing with data -- data collection, data analysis, and data presentation.
This has been designated a "regular" course for graduate students and a "research" course for Curriculum 2000 undergrads.
How:
- Term project -- to be developed in a systematic way, exercising the research skills taught in the class. It is acceptable to combine this project with systems projects in other courses, assuming all instructors agree. Teams of two students are encouraged.
- Readings -- examples of good and bad experimental studies from the literature. The textbook will be Jain, The Art of Computer Systems Performance Analysis.
- Class presentations -- students will be assigned particular areas of expertise to develop and be able to contribute to classroom discussions, the term project will be presented in a mini-conference at the end of the term, and there may be a requirement for a guest lecture in the class.
- Exam -- there will probably be two in-class exams.