Phase 1 of 3 · Implementation

Phase 1 — Build the rack, run inference, open the stream

Physically stand up the donated GPU rack, put Ollama and llama.cpp in production, and stream answers to student harnesses on other networks — with keys.

In progress

Eight GTX 1070s, donated to OCS

This capstone starts with generosity. OCS was donated a GPU rig with eight GTX 1070s — enough local compute to host open-weight models instead of renting them. The work is not to buy intelligence. It is to turn a gifted machine into a classroom service: assembled, served, and reachable from the networks students already use.

Donated hardware reduces purchase costs. The proposed hybrid service also needs a budget for electricity, cloud services, storage, networking, and maintenance.

What this phase is for

✓ Assemble the donated 8× GTX 1070 production rack.

✓ Serve open models with Ollama and llama.cpp (Nikhil, Adi, Mihir).

✓ Stream to harnesses across networks, with key issuance and management.

✓ Hold about five concurrent users without OOM or a crawl.

Donated 8× GTX 1070 Ollama / llama.cpp Live stream Keys & auth

How the work shows up

Physically build the GPU rack

Take the donated 8× GTX 1070 rig from hardware to a machine that stays powered, cooled, and ready to host a model for the class.

Inference engineering

Run open-weight models in production with Ollama and llama.cpp so students get useful answers, not a lab curiosity.

Websocket and live stream

Let users send requests through their harnesses from separate networks. The session should feel live — a stream — not a demo that only works in the same room.

Auth and keys

Issuing and managing keys is part of access. A shared brain without a front door is not a school service.

Done when

→ A student on another network can send a request through their harness and get a streamed reply.

→ Keys can be issued and managed so access is intentional.

→ About five people can use it at once without OOM, a hard crash, or token speed falling to a crawl.

Research question 1 · Primary — answer first, everything else depends on it

How do we turn donated GPU hardware into a shared open LLM that OCS students can actually code with — without a paid subscription?

Stakes Coursework shouldn't depend on who can afford a subscription.

Gift A donated rack with 8× GTX 1070s lets OCS host open-weight models locally instead of renting them.

Aim Stand the rack up, run inference with Ollama and llama.cpp, and stream answers to student harnesses on other networks — with keys, not hope.

Research endpoint: Answered when a student on another network can send a request through their harness and get a streamed reply, with keys issued and managed, without picking one specific model forever.

Priorities for this phase

P0 A student on another network can send a request through their harness and get a streamed reply.

P0 Keys can be issued and revoked so access is intentional, not an open port.

P1 Hold about five concurrent users without OOM or a crawl.

P2 Physically build and cable the rack so it stays powered and cooled unattended.

Serving endpoints

The literal endpoints student harnesses talk to once a model is loaded on the rack.

Endpoint	Engine	What it's for
POST /api/generate	Ollama	One-shot completion against whichever model is currently loaded.
POST /api/chat	Ollama	Multi-turn chat endpoint, used by student harnesses that keep conversation history.
POST /v1/chat/completions	llama.cpp server	OpenAI-compatible endpoint so existing tools (OpenRouter-style clients, Claude Code, Pi) can point at the rack with a config change, not a rewrite.

Why electricity is the whole bill

flowchart TD
    A[Paid subscriptions] --> B[Donated GPU rack]
    B --> C[Host open models locally]
    C --> D[Electricity only]
    D --> E[Stream, keys, harness]
    E --> F[Student on another network]
    F --> G[Same chance to code with a model]

Students connect from tools they already use — OpenRouter, Pi, or Claude Code — over a live stream, not a one-off laptop demo.

If this phase works

The donated rack becomes a public good inside OCS, not a closet of unused cards.

Coursework can assume a capable model without assuming a credit card.

Track operating costs while reducing dependence on individual subscriptions.

Who carries the donated rack

Team 1 puts the gifted production box in front of students. Team 2 keeps experiments off that path.

Production

Rig 1

Donated production rack — host models and stream them to students

Team 1 Nikhil, Adi & Mihir

860 EVO 250GB + SanDisk 480GB Ollama / llama.cpp

Experiment

Rig 2

Testing and experiment rig

Team 2 Yash Squared & Anvay

Samsung M.2 250GB Shared experiments

Literature

Resource	Type	What it means
Baseten Inference Engineering	Book	Practical guide to the serving stack: kernels, metrics, and production inference.
Orca	Paper	Iteration-level scheduling so many student requests can share one running model.
LLM Model Card	Model card	One candidate LLM we are evaluating: size, license, and intended use.