Camera Setup & Object Detection — Research 3
Multi-camera fusion, temporal object tracking, state machine transitions, and room model change detection for Jarvis.
Multi-Camera Spatial Perception & Distance Estimation
Two Logitech BRIO 4K cameras, mounted on opposite sides of the room.
Individual research by Sonika Dhenuva Konda
People and equipment constantly move around a busy classroom. A standard camera can see what's in the room, but not how far away it is or where it sits in 3D. Two wide-spaced cameras, fused with depth models and geometry, can fix that.
How can a setup of two wide-spaced 4K cameras combine separate 2D video feeds using deep learning models and geometry to accurately measure real-world distances and build a continuous 3D spatial map of a classroom?
flowchart TD
C1["Camera 1, BRIO
Wall A"] --> FM["SuperPoint + LightGlue
Cross-view matching"]
C2["Camera 2, BRIO
Wall B"] --> FM
C1 --> DA["Depth Anything V2
Per-frame depth map"]
C2 --> DA
FM --> GEO["OpenCV + NumPy
Calibration + 3D projection"]
DA --> GEO
GEO --> SCALE["ZoeDepth
Metric scale, m / ft"]
SCALE --> CLOUD["Open3D
Point cloud / room map"]
C1 --> YOLO["YOLOv8
Object boxes"]
C2 --> YOLO
YOLO --> CLOUD
USB 3.0, opposite walls, facing inward.
Calibration, distortion fix, 3D projection.
Runs the deep learning models on GPU.
Linear algebra for coordinate rotation.
Renders the room as a point cloud.
Dense depth map from a single frame.
Keypoints matched across wide baselines.
Boxes people/objects before distancing.
Cheap, standard RGB webcams; needs calibration + good light.
Very accurate; too costly, hurt by sunlight.
Fast full-frame depth; whiteboards cause interference.
Great up close; range drops off past 3–5m.
- Foundation model for single-image (monocular) depth estimation.
- Trained on millions of synthetic and pseudo-labeled images.
- Generates a dense depth map from just one frame.
- Fills in depth on plain classroom walls where matching alone fails.
- Neural network that matches keypoints between two images.
- Faster and more accurate than its predecessor, SuperGlue.
- Adaptive: less compute on easy pairs, more on hard ones.
- Built for exactly this project's wide-baseline, opposite-wall pair.
- Adds a metric prediction head onto a relative-depth model.
- Outputs real distances in meters, not just relative ordering.
- Zero-shot transfer across different scenes.
- Used here to scale outputs into real classroom feet/meters.
- Early work using single-image cues: shading, texture, perspective lines.
- Combines those cues with multi-camera stereo geometry.
- Shows the combination beats either method alone.
- Basis for pairing an AI depth model with physical camera geometry here.
- Multiple overlapping cameras share depth information.
- Depth built as a weighted combination of a shared "depth basis."
- Works even without large overlap between camera views.
- Blueprint for merging the two BRIO feeds into one coordinate space.
- Stereo vision that keeps working when cameras aren't on a rigid mount.
- Autocalibration corrects fast (vibration) and slow (thermal) shifts over time.
- Supports wide baselines, over 2 meters apart.
- Directly applies since the two BRIOs sit on opposite walls, not a fixed bar.
- Compares active sensors (LiDAR, ToF) against passive vision (stereo).
- Background for the depth-sensing comparison above.
- Reference for cv2.calibrateCamera() and lens-distortion correction.
- Basis for the calibration step in the pipeline diagram.