Design-Based Research Capstone · Jarvis Project Proposal · 2026-2027

Jarvis3000

A system that captures live classroom footage and object & segmentation detection to identify and track people and hardware

Camera Capture Timestamped Images

→

YOLO Object Detection

→

SAM 3 Precise Segmentation

→

Room Model Identity + Location

Problem Statement

Classrooms change throughout the day. Our computer science classroom is no exception. People come and go, and hardware can be moved, misplaced, or left behind. This makes it difficult to know whether the room is as expected at any given time. Missing or out-of-place items can go unnoticed, causing confusion, wasted time, or the loss of expensive equipment.

System Architecture

flowchart TD
    C1["Camera 1
Main Linux computer"] --> CAP["FFmpeg capture
Timestamped frame"]
    C2["Camera 2
Second Linux computer"] --> TX["SSH / NetBird"]
    TX --> CAP
    CAP --> YOLO["YOLO
Object detection"]
    YOLO --> SAM["SAM 3
Object segmentation"]
    SAM --> FUSE["Multi-camera matching
and temporal tracking"]
    FUSE --> ROOM["Room model
Identity + location + history"]

Core Research Question

Can a time-aware multi-camera system using object detection, segmentation, tracking, and schedule data reliably identify expected classroom objects and people and detect anomalies in their status, presence, or location?

Hardware Layout

Camera 1 Wall A

→

Main Linux Computer Capture + YOLO + SAM 3

→

Frames + Room Model Local storage

Netbird / SSH

Camera 2 Wall B, planned

→

Second Linux Computer Capture only, planned

Project Phases

flowchart LR
    P1["Phase 1
Camera Research & Capture
Sept 2026"] --> P2["Phase 2
Object Recognition & Segmentation
Sept - Nov 2026"]
    P2 --> P3["Phase 3
Multi-Camera Tracking & Room Model
Nov 2026"]
    style P1 fill:#3b82f622,stroke:#3b82f6
    style P3 fill:#a855f722,stroke:#a855f7

In Progress Phase 1: Camera Research & Capture Foundations

Test and calibrate the existing Logitech BRIO + Linux capture pipeline, benchmark FFmpeg/OpenCV at the 10-second capture interval, and begin collecting classroom images for the six core object classes.

Planned Phase 2: Object Recognition & Segmentation

Annotate the dataset, train and evaluate YOLO on the six core classroom classes, and integrate SAM 3 segmentation, targeting the 80% classification threshold on held-out images.

Planned Phase 3: Multi-Camera Tracking & Room Model

Bring the second camera and Linux node online, implement multi-camera matching and the object observation state machine, and validate the room model against the 80% object-state accuracy target.

See the In-Depth Timeline on the Technical Detail page for the week-by-week build schedule. Treat scope and dates as the team's current target, not a fixed commitment.

Explore the Project

Technical Detail & Flow

Capture and processing loops, observation state machine, data models, and privacy design.

View Technical Detail Research 1: Camera & Hardware

Multi-camera capture setup, network sync between Linux nodes, calibration, and bill of materials.

Explore Research 1 Research 2: YOLO Object Detection & SAM 3 Segmentation

Object classification, confidence scoring, bounding-box localization, and pixel-precise mask generation for classroom objects.

Explore Research 2 Research 3: Spatial Perception & Distance Estimation

Depth models, cross-view feature matching, and 3D point-cloud mapping across two wide-spaced cameras.

Explore Research 3 Research 4: Visual Memory & Scene Comparison

Shadow/light invariance, edge background subtraction, patent reviews, and commercial product comparisons.

Explore Research 4