Phase 0 of 3 · Implementation

Phase 0 — Define the problem, pick a direction

Name the problem without jumping to a solution, read the serving literature, and pick a candidate model class — before any hardware work locks in an approach.

Ongoing — research questions still evolving

What this phase is for

✓ Write the problem statement and both research questions.

✓ Review the serving/quantization/scheduling literature (see table below).

✓ Pick a first candidate model to prototype with, without committing to it long-term.

Literature review Benchmark comparison Problem framing

How the work shows up

Problem framing

Write down what's actually broken (paid AI access gates coursework) before deciding how to fix it, so the hardware work in Phase 1 answers a real question instead of just being a fun build.

Literature review

Read the serving, quantization, and scheduling papers that make consumer GPUs behave like a shared production service, and keep the reading list current as the project runs.

Candidate model search

Compare open-weight models against public benchmarks for coding ability and footprint, and stay ready to swap in a better one — no single model is the final answer.

Done when

→ The problem statement and both research questions are written down and reviewable on their own, with no hardware details required to understand them.

→ At least one candidate open model is picked to prototype with, with a documented reason and an explicit plan to re-evaluate it.

→ The literature list has enough coverage of serving, quantization, and scheduling to justify the Phase 1 approach.

Living questions, not a fixed spec — revised as each phase turns up real numbers.

Research question 1 · Primary — answer first, everything else depends on it

How do we turn donated GPU hardware into a shared open LLM that OCS students can actually code with — without a paid subscription?

Stakes Coursework shouldn't depend on who can afford a subscription.

Gift A donated rack with 8× GTX 1070s lets OCS host open-weight models locally instead of renting them.

Aim Stand the rack up, run inference with Ollama and llama.cpp, and stream answers to student harnesses on other networks — with keys, not hope.

Research endpoint: Answered when a student on another network can send a request through their harness and get a streamed reply, with keys issued and managed, without picking one specific model forever.

Research question 2 · Secondary — only answerable once RQ1's service exists

Once the service exists, how do we keep it fair, visible, and usable when many students need it at once?

Fairness Every OCS student should be able to reach the same service, not a lucky few on the same LAN.

Load A small class wave — about five people at once — should not empty VRAM, crash the API, or drag token speed to a crawl.

Trust Deployment and monitoring prove the promise: it works as intended, and it is still there when the room fills up.

Research endpoint: Answered when ~5 concurrent users hold steady without OOM, a hard crash, or token speed falling to a crawl, and every OCS student — not just the builders — can get a key.

Priorities for this phase

P0 Problem statement and research questions written and reviewable.

P0 First candidate model chosen, with a stated reason and a re-evaluation plan.

P1 Literature list broad enough to justify the serving approach used in Phase 1.

If this phase works

Phase 1 starts from a reasoned choice, not a guess.

Mentors can see the reasoning, not just the finished rack.

Literature

Resource	Type	What it means
Speculative Decoding	Paper	A small draft model proposes tokens that the serving model accepts or rejects, aiming for 60+ tokens per second.
YaRN	Paper	YaRN (Yet Another RoPE extensioN method) extends a model's context window past its training length.
LLM Model Card	Model card	One candidate LLM we are evaluating: size, license, and intended use.
TD-Pipe	Paper	Splits prefill and decode across pipeline stages so GPUs stay busy and throughput goes up.
PipeMax	Paper	Offline LLM inference on commodity GPU servers, closer to the hardware OCS actually has.
FlexGen	Paper	High-throughput generation on one GPU by offloading weights to CPU and disk when VRAM is tight.
HeteGen	Paper	Splits inference across CPU and GPU on machines that are short on resources.
PowerInfer	Paper	Serves an LLM on a consumer GPU by keeping hot neurons on GPU and cold ones on CPU.
PagedAttention / vLLM	Paper	Pages the KV cache like virtual memory so serving does not waste GPU RAM.
Orca	Paper	Iteration-level scheduling so many student requests can share one running model.
AWQ	Paper	Quantizes weights using activation stats so large models fit in less VRAM with less quality loss.
GPTQ	Paper	Post-training quantization for GPT-style models — another way to shrink models for our GPUs.
Baseten Inference Engineering	Book	Practical guide to the serving stack: kernels, metrics, and production inference.
TTFT, TPS, ITL	Metrics	Time to first token, tokens per second, and inter-token latency — the numbers we track to know the service still feels fast under load.
OCS Intelligence issue #4	Tracker	GitHub issue that tracks literature, metrics, and serving goals for this project.