Data Are Everywhere

These days, we have access to more data than we can possibly understand and the main challenge we face is how to turn that data into information we can use. These web sites are examples of people trying to start that process: Swivel, GapMinder, and ManyEyes.

Specification

In teams, write a program that allows users to visualize a variety of data found on the web.

Design Criteria

There are many extensions to the basic specifications possible; some are listed below. From the stand point of your grade, the most important thing is that your program is designed well (i.e., that it is possible to index new kinds of data sources, new ways to process data, and new visualizations of the data simply by adding O(1) line to your existing code to include the new functionality). This means your design should be open to adding new kinds of input, new kinds of filters and processors, and new kinds of output formats, while closed to changing the API for your data format.

Next in importance to your grade is your project should be thoroughly tested to prove to the course staff that your confidence in it is justified. You should include whatever data files, unit tests, or other driver programs (as well as documentation on how to use them) you have used to test your program in your submission. If you do all of the above well, the maximum grade you can receive is an A-.

Finally, the extensions given below are intended to stretch your design further and to differentiate your program from others in order to capture the global data visualiation market, your team should agree on one area of extensions to focus on if you want to be considered for a grade in the A range. These extensions must further the good design of your program and not simply be hacks of code added at the last minute. If you do not have time to implement an extension, partial extra credit may be given for excellent justification of how your design either supports adding such a feature already or how it would need to changed sufficiently to support such a feature.

• Support other data types besides row-column data tables
  • Text based data, produce interesting visualizations of text data, such as Tag Clouds or Text Trees
  • Graph based data, like that used in social networking sites
  • Image based data, especially useful for geo-located data
• Support different output formats than just graphics
  • Images, to be included on a website
  • HTML, to be displayed as a web page
• Allow users to customize a visualization via imported templates
  • A background to display behind the data
  • A color palette that swaps the colors displayed
• Allow users to manage their data sites
  • Organize data sites into history, favorites, or other organizational structure (like tags)
  • Alert users when the data they are using has changed on the original web site
• Allow users to interact with the visualized data
  • Animate the data over some variable such as time (or provide a slider to let the user control the animation)
  • Sort the data displayed on different criteria, like in most column data displayed on the web
  • Change the values displayed in the axes (or swap the current axes)
  • Change the displayed data to produce "what-if" scenarios
• Create an aggregation language for users
  • Textual, like SQL, the Structured Query Language for accessing data base systems
  • Graphical, like Yahoo Pipes for combining web sources

However, note that the amount of extra credit will be in proportion to the amount of intellectual effort needed to implement the option. For example, adding yet another way to filter key words would not be worth very much because your design should already support it. Of course, a well-tested, perfectly working program that has fewer features (but plenty of clear paths to easy expansion) is always worth more than the leaky kitchen sink.

In short, to maximize your grade, you should implement enough variety in your program to clearly demonstrate that your design supports further such extensions.

Resources

The following online resources may help you think about indexing algorithms.

• MapReduce: Simplified Data Processing on Large Clusters by J. Dean and S. Ghemawat
• On the Criteria to be used in Decomposing a System into Modules by D.L. Parnas
• Component Programming - a fresh look at software components by M. Jazayeri

Deliverables

1. Friday, October 9. vision and estimate due
   1. Create a web site for your project that includes
      • a name for your team and each team member's primary and secondary responsibility on the project
      • a vision for the project: describe what your team wants to accomplish with this project, specifically the priority of the extensions your team intends to implement
      • a list of issues that arise as you try to pin down the requirements, e.g., vague, ambiguous, conflicting requirements
      • a description of the classes you envision as part of implementing this project
      • an estimate of how long you expect it to take you to complete this project; include as specific a task list as possible to support your estimate (i.e., break each of the requirements into several steps, the smaller the better your estimate will be)
   2. No code is required for this deliverable. Make this information available on a web page (linked to by everyone on the team), and email the URL to Professor Duvall. We will check this page frequently to check on your team's progress, so you will need to update this site as you develop your project.
   3. This week you may meet with your mentor TA to discuss your design and your implementation plan
2. Tuesday, October 13. Version 1.0 of the project is due
   • This version should, at a minimum, be able to read from multiple formats and visualize in a similar way
   • Document your project in the style of a project artifact that completely describes the API for your data representation
   • After submitting this deliverable, your team may demo this program and discuss its design with your mentor TA.
3. Sunday, October 18. Final Version of the project is due
   • This should be the final version of the program, including all documentation (external and in code)
   • After submitting this deliverable, your team must demo this program and discuss your final design with your mentor TA.
4. Tuesday, October 20. Your individual project analysis is due.