OCS Intelligence LLM
OCS Intelligence LLM
Reduce dependence on paid AI subscriptions enabling student development and growth.
The problem
Capable AI help is rationed by who can pay
The strongest coding assistants live behind a monthly subscription. A student whose family can afford that seat gets a tireless tutor and pair programmer; a student who can't gets the same assignment and a worse starting point. That gap has nothing to do with effort or ability — it tracks a credit card.
Eight GTX 1070s, donated to OCS
This capstone starts with generosity. OCS was donated a GPU rig with eight GTX 1070s — enough local compute to host open-weight models instead of renting them. The work is not to buy intelligence. It is to turn a gifted machine into a classroom service: assembled, served, and reachable from the networks students already use.
Donated hardware reduces purchase costs. The proposed hybrid service also needs a budget for electricity, cloud services, storage, networking, and maintenance.
How this works
Two pictures: the path from gift to classroom service, and how a student actually reaches it.
From donation to three phases
flowchart LR
A[Donated 8x GTX 1070 rack] --> P0[Phase 0]
P0 --> B[Phase 1]
B --> C[Phase 2]
P0 --> P01[Problem and research questions]
P0 --> P02[Literature review]
P0 --> P03[Pick a candidate model]
B --> B1[Build rack]
B --> B2[Ollama and llama.cpp]
B --> B3[Stream and keys]
C --> C1[Deploy service]
C --> C2[Monitor under load]
C --> C3[Every student gets in]
How a student reaches the model
flowchart TD
A[Student at school or home] --> B[Authenticated OCS API]
B --> C[Broker and request queue]
C --> D[Mini prepares session context]
D --> E[GPU worker generates answer]
E --> F[Stream returned to student]
Teacher’s vision: a shared OCS AI compute service
Our proposed implementation separates the public student service from the physical compute lab. AWS hosts identity, the API, the broker, and durable records; Mac Minis prepare student sessions and context; GPU rigs run inference. This is a build and research plan, not a claim that the infrastructure is already deployed.
flowchart TD A[Student at school or home] -->|HTTPS| B[Proposed OCS AI endpoint] B --> C[AWS EC2: NGINX and authenticated Python API] C --> D[OCS broker and scheduler] D --> E[RDS: durable sessions and project records] D --> F[Redis: queue, assignments and worker health] D -->|NetBird private overlay| G[Mac Mini session and context workers] G -->|Private inference requests| H[Rig 1: production GPU workers] G -->|Isolated test jobs| I[Rig 2: development GPU workers]
1. Give students one secure entry point
If an OCS endpoint is provisioned, students will use HTTPS at the proposed ai.opencodingsociety.com address. NGINX will forward requests to an authenticated Python API on EC2. The API will verify the student’s role and project access, enforce request-size and usage limits, and stream responses. Students will not need NetBird installed for this public HTTPS path; the cloud service and lab machines will join the private overlay. Before that endpoint exists, we will pilot with enrolled NetBird clients.
- Define a request containing session_id, project_id, files, question, and context. Validate ownership and payload limits before accepting a job; never trust a client-supplied session ID alone.
- Keep Redis, databases, worker APIs, and inference ports private. Allow only the service-to-service connections each role requires, with authenticated internal requests.
- Test access from school and home, including unauthorized requests and revoked access. Record whether each NetBird connection is direct or relayed, alongside latency, throughput, and reconnect behavior.
How we will verify it: A student can submit an authorized coding question from both networks and receive a streamed answer; another student cannot read that session. Compare direct and relay paths rather than treating relay use itself as a failure.
2. Build the broker and separate three kinds of state
We will start with a FastAPI broker and Redis-backed queue. The broker will track sessions, register workers, and schedule work in two steps: assign a healthy Mini to prepare context, then reserve a compatible GPU worker for inference. A local broker on a designated Mini can support the first lab prototype; the proposed cloud deployment moves the control plane to EC2.
- RDS holds durable session ownership, conversation history, project references, and request status. Define retention and access rules for student content.
- Redis holds active Mini and GPU registries, queue entries, current assignments, and expiring reservation leases. Durable request IDs let us reconcile unfinished work after a restart.
- Mini caches hold disposable context and recent project files. Recover a session from durable records when a Mini disappears; never make its local cache the only copy.
- Expose job states such as queued, preparing, generating, complete, and failed. Return a job ID so a client can reconnect and check status without silently submitting duplicate work.
How we will verify it: Restart a Mini, Redis, and the broker separately. Verify that session ownership survives, unfinished jobs are reconciled, and temporary worker state can be rebuilt.
3. Turn Mac Minis into session and context workers
The teacher’s target is up to 30 converted Minis, split into production and development pools of roughly 10–15 each, subject to hardware inventory. We will prove the worker design on one Linux Mini first, then expand. Each node will run NetBird, a containerized Mini agent, bounded session/project caches, and monitoring.
- Have each agent register its identity and capacity, then send periodic heartbeats. Expire stale registrations so new sessions only reach healthy workers.
- Prefer the healthy Mini already serving a session to reuse cached context. Assign new sessions using available capacity and queue depth.
- Prepare a bounded prompt from authorized project files and recent conversation state, submit inference, and forward the result stream. Set cache size limits and expiration times.
- On worker loss, release expired leases and rebuild context on another Mini. Surface interrupted generation to the client; retry only within a bounded policy tied to the original request ID.
How we will verify it: Disconnect an assigned Mini during a request. Show that it stops receiving work, the student retains their session, and a replacement worker can continue with recovered context.
4. Schedule GPU inference around the hardware we have
We will prototype with llama.cpp and llama-server using quantized GGUF coding and general-purpose model candidates. The GTX 1070 software environment is a research constraint: validate and pin the Linux, NVIDIA driver, CUDA toolkit, llama.cpp revision, and model checksums on the actual rig before rollout. We will not assume a current vLLM or PyTorch package supports these cards.
- Start with one small model and one GPU worker. Measure model loading, memory use, answer quality, and streaming before increasing concurrency.
- Register each GPU worker’s health, loaded model, available memory, and active reservations. Prefer an idle compatible worker with the requested model already loaded.
- Use independent single-GPU workers as the baseline. Treat total rig VRAM as distributed memory, not one automatically pooled allocation.
- When all compatible workers are busy, use a bounded queue with per-student fairness, timeouts, cancellation, and a clear busy response. Reserve capacity atomically to prevent competing Minis from overbooking a GPU.
- Evaluate grouped GPUs only after the baseline works. Compare the same workload against single-GPU allocation and record communication overhead, latency, throughput, and memory limits.
How we will verify it: Publish a reproducible environment manifest and benchmark results. Choose models and allocation strategies from measured coding usefulness and performance, not aggregate VRAM or model size alone.
5. Keep experiments separate and rehearse recovery
The target topology uses two rigs with eight GTX 1070s each: Rig 1 for production and Rig 2 for development, each paired with its own Mini pool. Team 1 will own the production inference path; Team 2 will validate candidates and failure experiments on Rig 2. Both teams will agree on the broker/worker contract before connecting the pools.
- Separate production and test queues, credentials, and worker registrations so experiments cannot take classroom capacity by accident.
- Promote only a tested software/model configuration. Keep the previous configuration available for rollback.
- Rehearse taking Rig 1 offline: pause admissions or show a maintenance state, drain work when possible, reconcile interrupted requests, and explicitly promote a validated Rig 2 worker pool.
- Track queue wait, time to first token, inter-token latency, tokens per second, errors, GPU memory, and worker availability. Keep student code and secrets out of routine logs.
How we will verify it: Run an initial five-student load test and a planned rig outage. Report recovery time, failed requests, p50/p95 latency, and whether session history survives. Five users is a first test target, not a proven capacity claim.
How we will deliver this through the existing phases
We will build one complete request path before scaling to the full hardware pool. Each phase will end with a demonstration and recorded evidence that mentors can review.
- Phase 0 — Inventory both rigs and Minis; validate a Pascal-compatible inference environment; compare NetBird paths; select the first model and agree on latency, quality, and recovery targets.
- Phase 1 — Demonstrate one client → broker → Mini → GPU → streamed response. Add authentication, durable session records, queue states, and worker heartbeats; then add a second worker to demonstrate routing.
- Phase 2 — Provision the approved HTTPS entry point and cloud control plane, expand the production/test pools, benchmark classroom load, and rehearse Mini failure, broker restart, rig promotion, and rollback.
- Document compute, electricity, cloud hosting, database, storage, networking, and maintenance costs. Reusing donated hardware reduces purchase costs but does not make the proposed hybrid service electricity-only.
How we will verify it: Maintain a benchmark matrix and issue checklist for each phase. The central research question is whether this heterogeneous pool can deliver secure, reliable, low-latency coding help for distributed students at a sustainable operating cost.
Living questions, not a fixed spec — revised as each phase turns up real numbers.
Research question 1 · Primary — answer first, everything else depends on it
How do we turn donated GPU hardware into a shared open LLM that OCS students can actually code with — without a paid subscription?
Research endpoint: Answered when a student on another network can send a request through their harness and get a streamed reply, with keys issued and managed, without picking one specific model forever.
Research question 2 · Secondary — only answerable once RQ1's service exists
Once the service exists, how do we keep it fair, visible, and usable when many students need it at once?
Research endpoint: Answered when ~5 concurrent users hold steady without OOM, a hard crash, or token speed falling to a crawl, and every OCS student — not just the builders — can get a key.
Three phases, one promise
The tabs are the implementation path. First we name the problem and pick a direction. Then the gift becomes a live service. Then that service has to survive a class.
Phase 0 — Define the problem, pick a direction
Name the problem without jumping to a solution, read the serving literature, and pick a candidate model class — before any hardware work locks in an approach.
Open this phase Phase 1 · In progressPhase 1 — Build the rack, run inference, open the stream
Physically stand up the donated GPU rack, put Ollama and llama.cpp in production, and stream answers to student harnesses on other networks — with keys.
Open this phase Phase 2 · NextPhase 2 — Deploy, watch, and make sure the class can get in
Put the service in front of real students, watch it under load, and keep it reachable so access is a class right — not a lucky connection.
Open this phaseTeam split
Production inference on the donated rack — Ollama and llama.cpp, plus the path students use to reach it.
Testing and experiments on Rig 2 so production stays stable while ideas get tried.
Staying accountable
How we keep mentors in the loop
Progress is written down as it happens, not summarized right before grading. Anyone mentoring this capstone can follow it without asking for a status meeting.
Dated write-ups under this capstone (the Phase 1/Phase 2 pages themselves) log what shipped, what broke, and what changed about the research questions.
Literature, metrics, and serving goals live in Open-Coding-Society/OCS-Intelligence issue #4, updated as the questions evolve.
Justification
Students connect from tools they already use — OpenRouter, Pi, or Claude Code — over a live stream, not a one-off laptop demo.
Three implementation phases
A shared mind for the class
Turn a donated GPU rack into a dependable companion for coding — then keep the door open when the class shows up.
Why this is challenging
The hard part isn't turning on a GPU — it's turning eight second-hand GTX 1070s (8GB VRAM each, no NVLink) into something that behaves like one reliable service. Model weights that don't fit on a single card, a KV cache that grows with every concurrent student, and a class of ~25 people hitting it in the same ten minutes all compete for the same limited memory. The research question is whether consumer-grade, donated hardware can be engineered around those limits well enough that nobody in the room notices the difference from a paid API.
Goal
We are not locking into one model. Candidates get re-evaluated as better open weights ship — the target is whichever coding companion fits this hardware and stays good enough that a subscription is optional, not required.
Impact
Two rigs
The donated production rack is the student-facing gift. A second box holds experiments so the class path stays calm.
Rig 1
Donated production rack — host models and stream them to students
Rig 2
Testing and experiment rig
Capability we want in the room
We look at public scores to ask a human question: can an open model be good enough for student coding that nobody in the room needs a paid seat?
