Design-Based Research Capstone · Jarvis Project Proposal · 2026-2027

Jarvis3000

A system that captures live classroom footage and object & segmentation detection to identify and track people and hardware

Team Kanban Board

Jarvis3000 Team Kanban 30 cards Open on GitHub

Project snapshot fetched Oct 28, 2026 at 18:02 UTC. Open on GitHub for the latest changes.

How Jarvis3000 works: classroom cameras observe the room, Jarvis detects objects, and the system reports objects that moved
Jarvis3000 observes classroom objects over time and highlights changes in their location.
Camera Capture Timestamped Images
→
Object Detector Object Detection
→
SAM 3 Precise Segmentation
→
Room Model Identity + Location

Problem Statement

Classrooms change throughout the day. Our computer science classroom is no exception. People come and go, and hardware can be moved, misplaced, or left behind. This makes it difficult to know whether the room is as expected at any given time. Missing or out-of-place items can go unnoticed, causing confusion, wasted time, or the loss of expensive equipment.

System Architecture

Jarvis captures classroom images and sends them to a proposed VPC for AI processing and image comparison. SAM 3 and other AI models needed by the system will use a GPU system located in another part of the classroom. The VPC compares images over time and tracks changes. Results return to Jarvis for storage and updates to the room model.

flowchart TD
    CAMERA["Camera 1"] --> CAPTURE["Jarvis on local Linux device
Capture timestamped images"]
    CAPTURE -->|Send images + prompts| AI
    subgraph VPC["VPC: Processing Environment"]
      AI["SAM 3 + other AI models
Object detection and segmentation"]
      AI --> TRACK["Image comparison + temporal tracking
Future multi-camera matching"]
    end
    GPU["GPU system"] -->|Provides GPU computing| AI
    AI -->|Return detections, masks + overlays| STORE["Jarvis local storage
Original images + AI results"]
    STORE -->|Send earlier images for comparison| TRACK
    TRACK -->|Return tracked identities, locations + changes| ROOM["Room model
Identity + location + history"]
    CAPTURE -->|Save originals| STORE
    CAMERA2["Future Camera 2
Second Linux capture device"] -.->|Additional timestamped images| CAPTURE

Core Research Question

Can a time-aware multi-camera system using object detection, segmentation, tracking, and schedule data reliably identify expected classroom objects and people and detect anomalies in their status, presence, or location?

Hardware Layout

Camera 1
Camera 1 Wall A
→
Main Linux computer
Main Linux Computer Capture + Detection + Job Storage
→
Frames + Room Model Local storage
NetBird
Netbird / SSH
Camera 2
Camera 2 Wall B, planned
→
Second Linux computer
Second Linux Computer Capture only, planned
Main Linux Computer Audio output
→
Ultimate Ears WONDERBOOM Classroom speaker

Project Phases

flowchart LR
    P1["Phase 1
Camera Research & Capture
Sept 2026"] --> P2["Phase 2
Object Recognition & Segmentation
Sept - Nov 2026"]
    P2 --> P3["Phase 3
Multi-Camera Tracking & Room Model
Nov 2026"]
    style P1 fill:#3b82f622,stroke:#3b82f6
    style P2 fill:#f59e0b22,stroke:#f59e0b
    style P3 fill:#a855f722,stroke:#a855f7
Done Phase 1: Camera Research & Capture Foundations

Test and calibrate the existing Logitech BRIO + Linux capture pipeline, benchmark FFmpeg/OpenCV at the 10-second capture interval, and begin collecting classroom images for the six core object classes.

Done Phase 2: Object Recognition & Segmentation

Annotate the dataset, train and evaluate an object-detection model on the six core classroom classes, and integrate SAM 3 segmentation, targeting the 80% classification threshold on held-out images.

In Progress Phase 3: Multi-Camera Tracking & Room Model

Bring the second camera and Linux node online, implement multi-camera matching and the object observation state machine, and validate the room model against the 80% object-state accuracy target.

See the In-Depth Timeline on the Technical Detail page for the week-by-week build schedule. Treat scope and dates as the team's current target, not a fixed commitment.

Explore the Project

Technical Detail & Flow

Capture and processing loops, observation state machine, data models, and privacy design.

View Technical Detail Research 1: Camera & Hardware

Multi-camera capture setup, network sync between Linux nodes, calibration, and bill of materials.

Explore Research 1 Research 2: Object Detection & SAM 3 Segmentation

Object classification, confidence scoring, bounding-box localization, and pixel-precise mask generation for classroom objects.

Explore Research 2 Research 3: Spatial Perception & Distance Estimation

Depth models, cross-view feature matching, and 3D point-cloud mapping across two wide-spaced cameras.

Explore Research 3 Research 4: Visual Memory & Scene Comparison

Shadow/light invariance, edge background subtraction, patent reviews, and commercial product comparisons.

Explore Research 4 Progress

The next release checkpoint and planned work.

View Progress Future Plans

Feature development and integration goals for the October 25th target release.

View Future Plans
Project Team Arnav Mittal, Shriya Paladagu, Soni Dhenuva, Xavier Thompson
Mentor Jake Anderson
Prototype Stage View Repo