NVIDIA just released DeepStream 9.1. The update targets a persistent problem in video analytics. Tracking one object across many cameras traditionally requires manual camera calibration and complicated calculations. DeepStream 9.1 addresses this with two additions: Multi-View 3D Tracking (MV3DT) and AutoMagicCalib (AMC). Both ship as agentic skills for coding agents. As a result, developers move from concept to a running pipeline faster.

What is DeepStream 9.1

To understand the update, start with the base platform. DeepStream is NVIDIA’s streaming analytics toolkit for AI-based video and image understanding. It provides a GStreamer-based framework for multi-stream, multi-model inference on NVIDIA GPUs. Pipelines combine hardware-accelerated decoding and encoding, TensorRT inference, object tracking, and message-broker integration.

Building on that base, version 9.1 adds five notable items:

1. 13 agentic skills for coding agents.
2. The MV3DT skill for cross-camera tracking.
3. The AMC skill for automatic calibration.
4. NVIDIA JetPack 7.2 support for Jetson Orin and Thor edge devices.
5. A unified open-source GitHub repository under CC-BY-4.0 AND Apache-2.0.

How MV3DT Tracks Objects Across Cameras

Among those additions, MV3DT is the main skill, so consider how it works. At its core, MV3DT projects detections from multiple calibrated cameras into a shared 3D coordinate system. It then associates observations of the same object across camera views. Finally, it assigns one globally consistent object ID.

Concretely, the data flow runs in four stages. For detection, each camera stream runs an object detector. MV3DT supports three models out of the box:

• PeopleNetTransformer: a transformer-based people detector, the default for pedestrian scenes.
• PeopleNet v2.6.3: a high-efficiency detector based on the DetectNet_v2 architecture.
• RT-DETR 2D: a multi-class detector for pedestrians, transporters, and forklifts.

Next, for monocular 3D perception, each camera uses a 3×4 projection matrix stored in a YAML calibration file. This back-projects 2D bounding boxes into 3D world-space coordinates using a ground-plane assumption. Then, for multi-view association, the tracker shares tracklets using Message Queuing Telemetry Transport (MQTT). MQTT is a lightweight pub/sub messaging protocol. When two cameras observe the same person, it matches tracklets by proximity in 3D world space.

After association, results stream out in three forms. The On-Screen Display (OSD) shows a tiled grid with 2D and 3D bounding boxes. The Bird’s-Eye View (BEV) renders a top-down trajectory map. Kafka messaging delivers per-frame protobuf metadata, including sensor ID, object ID, and 3D bounding box.

How AutoMagicCalib Removes Manual Setup

MV3DT depends on calibrated cameras, which traditionally means checkerboards and downtime. Instead, AMC calibrates a network by analyzing tracked objects in existing video files or streams. It estimates each camera’s intrinsic parameters (focal length, principal point, lens distortion). It also estimates extrinsic parameters (rotation, translation, world position).

Under the hood, the pipeline runs five stages. These are per-camera trajectory extraction, single-view rectification, multi-view tracklet matching, bundle adjustment, and optional VGGT refinement. VGGT (Visual Geometry Grounded Transformer) helps when object movement is limited. AMC runs as a microservice with REST APIs and a web interface. Users supply only a layout image and a few alignment points.

The Agentic Skills Workflow

With MV3DT and AMC defined, the delivery mechanism is the skills themselves. Rather than editing configuration files, you describe intent in natural language. The skills work with Claude Code, Codex, Cursor, and similar agents. Setup is short:

git clone https://github.com/NVIDIA/DeepStream.git
cd DeepStream
# Copy skills into your agent's skill directory (Codex shown)
mkdir -p ~/.codex/skills
cp -r skills/* ~/.codex/skills/

After launching the agent, a single prompt runs the reference app:

deploy mv3dt on the 12-camera sample dataset

From there, the MV3DT skill validates prerequisites, pulls the container, and installs Kafka and Mosquitto broker services. It also downloads model weights, generates the pipeline config, and launches tracking. Notably, if calibration files are missing, it triggers the AMC skills automatically.

DeepStream 9.0 vs 9.1

For context, the table below shows what changed between releases.

Use Cases With Examples

Given these capabilities, the features map to concrete deployments:

• Warehouse safety: track a worker near forklifts across aisles with one ID, using RT-DETR 2D.
• Retail analytics: follow a shopper between camera zones to measure dwell time without re-identification errors.
• Smart-building monitoring: count occupancy across floors and feed Kafka metadata to dashboards.
• Robotics and smart cities: share consistent world coordinates for navigation and incident review.

Interactive Explainer

To see the mechanism, the embedded demo below animates one person walking between three camera fields of view. Toggle between naive per-camera 2D tracking and MV3DT 3D fusion to watch the object ID stay consistent.

‘;
camsEl.appendChild(d);
});

// build pipeline steps
var steps=[
{t:’1 · Detection’,d:’Per-camera detector finds 2D boxes (PeopleNetTransformer / RT-DETR 2D).’},
{t:’2 · Monocular 3D’,d:’3×4 projection matrix back-projects boxes to 3D via ground-plane.’},
{t:’3 · Association’,d:’MQTT shares tracklets; match by proximity in 3D world space.’},
{t:’4 · Output’,d:’Global ID to OSD, BEV, and Kafka protobuf metadata.’}
];
var stepsEl=document.getElementById(‘steps’);
steps.forEach(function(s,i){
var d=document.createElement(‘div’); d.className=”step”; d.id=’step’+i;
d.innerHTML=’

STEP ‘+(i+1)+’

‘+s.t+’

‘+s.d+’

‘;
stepsEl.appendChild(d);
});

function lerp(a,b,f){return a+(b-a)*f;}
function personPos(tt){
tt=((tt%1)+1)%1; // wrap into [0,1) so negatives are safe
var n=path.length, p=tt*n, i=Math.floor(p)%n, f=p-Math.floor(p);
if(i<0)i+=n;
var a=path[i], b=path[(i+1)%n];
return {x:lerp(a[0],b[0],f), y:lerp(a[1],b[1],f)};
}
function inFov(c,px,py){
var dx=px-c.x, dy=py-c.y, dist=Math.hypot(dx,dy);
if(dist>c.range) return false;
var a=Math.atan2(dy,dx);
var diff=Math.atan2(Math.sin(a-c.ang),Math.cos(a-c.ang));
return Math.abs(diff)<=c.fov;
}

function draw(){
ctx.clearRect(0,0,W,H);
// floor grid
ctx.strokeStyle=”#181818″; ctx.lineWidth=1;
for(var gx=0;gx<=W;gx+=30){ctx.beginPath();ctx.moveTo(gx,0);ctx.lineTo(gx,H);ctx.stroke();}
for(var gy=0;gy<=H;gy+=30){ctx.beginPath();ctx.moveTo(0,gy);ctx.lineTo(W,gy);ctx.stroke();}
ctx.strokeStyle=”#2a2a2a”; ctx.strokeRect(8,8,W-16,H-16);

var p=personPos(t);
var active=[];
// draw FOV cones
cams.forEach(function(c,i){
var seen=inFov(c,p.x,p.y);
if(seen) active.push(i);
ctx.beginPath();
ctx.moveTo(c.x,c.y);
ctx.arc(c.x,c.y,c.range,c.ang-c.fov,c.ang+c.fov);
ctx.closePath();
ctx.fillStyle=seen?’rgba(118,185,0,0.10)’:’rgba(43,108,176,0.07)’;
ctx.fill();
ctx.strokeStyle=seen?’rgba(118,185,0,0.5)’:’rgba(43,108,176,0.35)’;
ctx.lineWidth=1; ctx.stroke();
// camera body
ctx.beginPath(); ctx.arc(c.x,c.y,7,0,6.29);
ctx.fillStyle=seen?’#76B900′:’#2b6cb0′; ctx.fill();
ctx.fillStyle=”#9a9a9a”; ctx.font=”10px sans-serif”;
ctx.fillText(c.name.replace(‘Camera ‘,”),c.x-3,c.y+3);
});

// trajectory trail
ctx.strokeStyle=”rgba(118,185,0,0.25)”; ctx.lineWidth=2; ctx.beginPath();
for(var k=0;k<=20;k++){var q=personPos(t-k*0.004);if(k===0)ctx.moveTo(q.x,q.y);else ctx.lineTo(q.x,q.y);}
ctx.stroke();

// person
ctx.beginPath(); ctx.arc(p.x,p.y,9,0,6.29);
ctx.fillStyle=”#76B900″; ctx.fill();
ctx.strokeStyle=”#0b0b0b”; ctx.lineWidth=2; ctx.stroke();

// ID label follows person
var shownId = mv3dt ? (‘#’+globalId) : (‘#’+naiveId);
ctx.fillStyle=”#0b0b0b”; ctx.fillRect(p.x+11,p.y-18,34,16);
ctx.strokeStyle=”#76B900″; ctx.lineWidth=1; ctx.strokeRect(p.x+11,p.y-18,34,16);
ctx.fillStyle=”#76B900″; ctx.font=”bold 11px sans-serif”;
ctx.fillText(shownId,p.x+15,p.y-6);

updatePanels(active);
updateId(active);
updateSteps(active);
}

function updatePanels(active){
cams.forEach(function(c,i){
var el=document.getElementById(‘cam’+i), det=document.getElementById(‘det’+i), bar=document.getElementById(‘bar’+i);
var on=active.indexOf(i)>-1;
el.className=”cam”+(on?’ active’:”);
det.textContent=on?’detecting → 2D box’:’idle’;
bar.style.width=on?’100%’:’0′;
});
}

function updateId(active){
var idVal=document.getElementById(‘idVal’), idMsg=document.getElementById(‘idMsg’);
// handoff detection: a camera newly becomes primary
var primary = active.length? active[0] : -1;
if(primary!==-1 && primary!==lastActive){
if(!mv3dt) naiveId++; // naive mode: new ID on each new camera
lastActive=primary;
}
if(active.length===0) lastActive=-1;
if(mv3dt){
idVal.textContent=”#”+globalId;
idVal.style.color=”#76B900″;
idMsg.innerHTML=’Consistent across every camera. One person = one ID.’;
}else{
idVal.textContent=”#”+naiveId;
idVal.style.color=”#e07a5f”;
idMsg.innerHTML=’Switches at each new camera — the re-identification problem.’;
}
}

function updateSteps(active){
var anyDet=active.length>0;
var state=[anyDet, anyDet, active.length>1, anyDet];
if(!mv3dt) state=[anyDet,false,false,anyDet];
for(var i=0;i<4;i++){
document.getElementById(‘step’+i).className=”step”+(state[i]?’ on’:”);
}
}

function loop(){
requestAnimationFrame(loop); // reschedule first so nothing can freeze the loop
if(playing){ t+=speed; if(t>1)t-=1; }
try{ draw(); }catch(e){ if(window.console)console.warn(‘draw skipped’,e); }
}

document.getElementById(‘play’).onclick=function(){
playing=!playing; this.textContent=playing?’Pause’:’Play’;
};
document.getElementById(‘reset’).onclick=function(){
t=0; globalId=1; naiveId=1; lastActive=-1;
};
var sw=document.getElementById(‘sw’); sw.className=”switch on”;
sw.onclick=function(){
mv3dt=!mv3dt; this.className=”switch”+(mv3dt?’ on’:”);
document.getElementById(‘modeTag’).textContent=mv3dt?’MV3DT ON’:’NAIVE 2D’;
document.getElementById(‘note’).textContent = mv3dt
? ‘MV3DT mode: the ID stays #1 across every camera handoff. Toggle left to see naive single-camera 2D tracking assign a new ID each time the person enters a new view.’
: ‘Naive per-camera 2D mode: each camera tracks independently, so the same person gets a new ID at every handoff. This is the problem MV3DT solves.’;
naiveId=1; lastActive=-1;
};

loop();

// auto-resize for WordPress srcdoc iframe
function postH(){
var h=document.getElementById(‘root’).offsetHeight+40;
if(window.parent) window.parent.postMessage({mv3dtHeight:h},’*’);
}
window.addEventListener(‘load’,postH);
window.addEventListener(‘resize’,postH);
setInterval(postH,1200);
})();