CPS 512 (Duke University) Distributed Systems

Spring 2022 labs: DSLabs

This semester's lab work uses DSLabs from UW-Seattle. We require only the first two DSLabs to get your feet wet. This semester also requires an independent project. You may do labs 3-4 as your project if you choose.

For DSLabs, you will need a standard Java environment and know how to use git and github. Please let us know if you need assistance (try the docs first). We recommend that you clone the dslabs repo. You can use the master branch, but the handout branch should work "out of the box" without any trouble. E.g.:
git clone https://github.com/emichael/dslabs; cd dslabs; git checkout handout

Grading is offline. Include any notes or sources of assistance in a new file called README.txt. Details for submission and team management will be announcement.

DSLabs README, including hints for using Windows

No open repos! Do not make your solution code public! In particular, it is a violation of course policy and of your honor to place your solution code in any public repository. Some students post their code as an attraction to potential employers. Employers are learning that anyone who does that should not be hired. If you release your solution code, I will hunt you down. Sorry if this sounds mean but I spend an afternoon each year chasing after former students to take down these repos.

Where to develop and test your code? You can use your own computer and your favorite IDE for these Java labs. But you will want a gigabyte of memory or more available for Java, or some tests may fail for the later labs. If you are running Linux or MacOS you should be in good shape. For MacOS you may need to install Xcode from Apple and some other software (e.g., coreutils) from Homebrew. For Windows, see the hints in the DSLabs README. Another option is to use Docker Desktop to get Linux containers on MacOS or on Windows: see these instructions. Duke also runs a virtual machine service VCM that can give you a private Linux VM for your use. For Linux containers and VMs, select Ubuntu 18.04 and use openJDK; you may need to install missing packages (apt-get install).

Overview of DSLabs

The goal of the labs is to build a working sharded key-value store with Paxos replication. From the DSLabs descriptions:

Lab 1 implements a simple reliable RPC and key-value storage API.
Lab 2 uses primary/backup replication, assisted by a view service that decides which machines are alive. The view service allows the primary/backup service to work correctly in the presence of network partitions. The view service itself is not replicated and is a single point of failure.
Lab 3 [Optional this semester] uses the Paxos protocol for replication with no single point of failure and handles network partitions correctly. This service is slower than a non-replicated server would be, but is fault tolerant.
Lab 4 [Optional this semester] is a sharded, transactional key/value database, where each shard replicates its state using Paxos. This key/value service can perform operations in parallel on different shards, allowing it to support applications that can put a high load on a storage system. Lab 4 has a replicated configuration service, which tells the shards for what key range they are responsible. It can change the assignment of keys to shards, for example, in response to changing load. It also supports transactions, allowing atomic update of keys held on different shards. Lab 4 has the core of a real-world design for thousands of servers.

Using other people's code, and collaboration

We all know that there are various fragments of code available on the web that are relevant to the assignments. You are not prohibited from looking at material on the web. We believe that we have provided you with enough to start with that it will not benefit you to import source code from anywhere else. The point of the lab is to learn, and we want to make it easy for you to learn!

When you hand in the lab, we expect that your code is your own. If you incorporate source code from some external source, you must add a comment indicating the imported code and the source of that code. You must note all sources of imported code in your submitted README file. You should be able to explain any part of your code in detail, and why you chose to do it that way and not some other way. If you encounter code from some external source that diminishes your opportunity to learn by doing the lab, please inform us (we may not be aware of it).

See also the collaboration policy in the formal syllabus [PDF] and course policies.

Academic dishonesty. Be sure that you are familiar with the policy on collaboration and other policies. In particular, sharing of code among groups is not allowable. Do not obtain code from previous semesters or from students at other schools using these labs. Take care to acknowledge any sources of code or assistance in your README.