CompSci 590 : Spring 2026

Using Duke's LLMs and Servers

These instructions will ensure each person can use Duke's LLM API and deploy it on a Server.

Install Docker Software

Download and install Rancher Desktop to create and use Docker images.
In a Terminal, test the installation is correct by running the command:
```
docker run hello-world
```
Verify that the Docker image hello-world is listed in your Rancher Desktop app.

Configure Duke's AI Gateway API Key

If you already have an API key for another LLM service, feel free to use that instead if you prefer (in which case, you will need to change the example code to use its API URL).

Visit Duke’s AI Dashboard
Select “Access AI Dashboard” and enter your NetID and password
Go to the AI Gateway tab and select “Create API Key”
Give it a name (like “CompSci 590”), but do not provide a Fund Code
Read the warnings very carefully as you may be charged for your use
Copy the resulting key text to somewhere very memorable since you will not be able to access it again
Read Duke’s AI Gateway FAQ if you have further questions

Clone the Chatbot Example Code to your Local Computer

Clone the example code:

git clone git@coursework.cs.duke.edu:compsci590_2026spring/example_chatbot.git

Change into the project's code folder:
```
cd example_chatbot/src
```
Edit Docker’s expected secret file, .env, to include your API key:
```
cp .env.sample .env
nano .env
```
Note, this new file will not be tracked by GIT since the .env file is ignored in this project.
Build and run the Docker images:
```
docker compose up --build
```
Verify two new Docker images are listed in your Rancher Desktop app
Access the running website from your browser by visiting:
```
http://localhost:4083
```

Set up your Deployment Server

Reserve a Duke VCM instance based on a LAMP Stack
Copy the public SSH key you used for Gitlab to the Duke directory entry for your NetID
Note, on that same page, set your VCM shell to your preferred shell (zsh or bash)
Log in to your VCM instance:
```
ssh NETID@vcm-ID.vm.duke.edu
```
Note, no password should be needed!
Update to latest version by running this command:
```
sudo apt upgrade -y
```
Note, type your NetID password when prompted.
After it restarts, visit its webpage:
```
http://vcm-YOUR_ID.vm.duke.edu/
```
Note, your browser may complain that the website is not secure
Make some change to the website!
```
cd /var/www/html/
ls -l
sudo chown -R NETID .
ls -l
nano index.html
```
Note, refresh the page to see your change.

Install Docker on Server

Install Docker on your server (it may be easier to copy these instructions from Steps 1 and 2):

# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add current repository as APT sources
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF

# Update the package manager so it sees these new sources
sudo apt update

# Install necessary Docker packages
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Note, these steps are overly complex since you are doing it “by hand” instead of using a “script” like those provided by software tools like Rancher, Brew, etc. Consider reading the complete instructions for more detailed explanations.

Check that the Docker daemon is running using the admin command systemctl:
```
sudo systemctl status docker
```
Manage Docker as a non-root user (so you don’t have to put sudo in front of every command):
```
sudo usermod -aG docker $USER
newgrp docker
```
Test the installation is correct by running the command:
```
docker run hello-world
```
List the installed Docker images:
```
docker images
```

Clone the Chatbot Example Code to your Sever

Clone the example code:

git clone git@coursework.cs.duke.edu:compsci590_2026spring/example_chatbot.git

Change into the project's code folder:
```
cd example_chatbot/src
```
Edit Docker’s expected secret file, .env, to include your API key:
```
cp .env.sample .env
nano .env
```
Note, you will need to add your API key again since it is not stored in Gitlab.
Note, change the HOST from localhost to vcm-YOUR_ID.vm.duke.edu
Build and run the Docker images:
```
docker compose up --build
```
Verify two new Docker images are installed:
```
docker images
```
Access the running website from your browser by visiting:
```
http://vcm-YOUR_ID.vm.duke.edu:4083
```
You can stop your images by using this command in the same folder:
```
docker compose down
```

Note, there is no need to copy these files to a special location, as this site is not managed by the Server's web application (Apache), but by Docker. However, you may still want to create a “special” folder for the installed Docker source code to make it easier to manage them separately.