mathdatech/Compose-Examples
1
0
Fork 0
mirror of https://github.com/Haxxnet/Compose-Examples synced 2025-02-23 16:38:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
Compose-Examples/examples/ollama-ui/README.md

3.8 KiB
Raw Blame History

References

Notes

You can spawn Ollama first and then download the respective LLM models via docker exec. Alternatively, spawn the whole stack directly and download LLM models within Open WebUI using a browser.

# spawn ollama and ui
docker compose up -d

# (optional) download an llm model via docker exec
docker exec ollama ollama run llama3:8b

Afterwards, we can browse Open WebUI on http://127.0.0.1:8080 and register our first user account. You may want to disable open user registration later on by uncommenting the env ENABLE_SIGNUP variable and restarting the Open WebUI container.

Tip

You likely want to pass a GPU into the Ollama container. Please read this.

Windows WSL 2 Setup

  1. Enable Nested/Hardware Virtualization in your BIOS
  2. Install Nvidia drivers for Windows
  3. Install Docker Desktop for Windows

Then open an administrative PowerShell and proceed as follows:

# install wsl
wsl --install

# configure wsl 2 as default
wsl --set-default-version 2

# install ubuntu
wsl --install -d ubuntu

After Ubuntu installation, you are prompted for a username and password.

Afterwards, you will find youself in a Ubuntu terminal.

Windows Docker Desktop Setup

Open Docker Desktop and enable WSL 2 via the setting Use WSL 2 based engine:

image

Then open Resources > WSL Integration and enable Docker for your Ubuntu WSL VM(s):

image

WSL 2 Ubuntu Setup

After Docker Desktop adjustments, return to the PowerShell with your Ubuntu terminal.

Proceed in Ubuntu terminal with the following commands from Ollama documentation:

# add repo
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
    | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
    | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
    | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update

# install nvidia container toolkit
sudo apt-get install -y nvidia-container-toolkit

# add your user to docker group
sudo usermod -aG docker <your-username>
newgrp docker

# test docker and compose
docker -v
docker compose -v

Warning

You likely have to restart Docker Desktop and re-open the Ubuntu WSL VM

Now you can make use of Docker in your WSL 2 Ubuntu VM.

  1. Store the docker-compose.yml from this repo into your WSL 2 Ubuntu VM
  2. Start the Docker stack consisting of Ollama and OpenWebUI
docker compose up -d

Afterwards, you can open a web browser of your choice on your Windows and browse http://127.0.0.1:8080.

You will be greeted by OpenWebUI and must setup your administrative user account.

Afterwards:

  1. Browse the settings at http://127.0.0.1:8080/admin/settings
  2. Visit the Connections area
  3. Disable OpenAI API
  4. Click the Manage download button at Ollama and download your preferred models (see https://ollama.com/library)

image

Unsure which Deepseek R1 model your GPU can run? Read this Reddit post.

Here my results on Windows 11 with RTX 3080 Ti 12GB VRAM image