How depth sensing works
Every camera on the comparison page does the same job: turn a picture into a distance for every pixel. There are four ways of doing that in the hardware you can actually buy, and they fail in completely different ways. Which one is inside a camera tells you more about how it will behave in your room than most of the numbers on its spec sheet do.
This is the long version. If you already know how these work and you want the specs, the comparison page is the one you want.
What a depth sensor is
An ordinary camera records light. Every pixel is a colour, three numbers for red, green and blue, and a video signal is that grid thirty or sixty times a second. It tells you what the scene looked like and nothing about where anything was: a person two metres away and a photo of them on the wall behind are the same thing to it.
A depth sensor records distance instead. Same grid of pixels, same frame rate, but the number in each pixel is how far away that piece of the scene is. As a picture a depth frame is a fog, near surfaces one shade and far ones another, with no texture and no idea what colour anything was. Most of these devices carry both, a colour camera beside the depth one, lined up so a pixel in one is the same piece of the world as the pixel in the other. That is where RGBD comes from: red, green, blue, depth. Depth camera, depth sensor and RGBD camera all mean roughly the same thing.
A distance plus the direction it was measured in, which is just where in the frame the pixel sits, is a position in space. Do that for every pixel and the frame becomes a point cloud: one point per measurement, on the surfaces that faced the sensor, which you can look at from an angle the camera never had, each point carrying its colour as well. One sensor sees one side of things, so a single cloud is a shell with a hollow back and a gap behind every object, which is why capture setups tend to grow a second and a third sensor.
Two ways to measure a distance
Time of flight measures the light itself: the phase of what comes back tells you how far it travelled. The other three measure a shift instead. The same point, seen from two places, lands in two different spots, and the difference between those spots is the distance. Structured light triangulates between a projector and a camera, stereo triangulates between two cameras, and active stereo is stereo with a projector added to give the cameras something to look at.
That split is why the failures are so different. A time of flight camera gets confused by light arriving the long way round. A triangulating camera gets confused when it cannot find the same thing twice.
The four diagrams below are drawn to one scene: same sensor, same subject two metres away, same wall behind it. Only the machinery changes.
Time of flight
The sensor floods the scene with infrared light and modulates its brightness, and every pixel measures the phase of the light that comes back. Light that travelled further returns later, so its phase is shifted further, and that shift is the distance. There is no stopwatch and no single pulse being timed: the emitter runs continuously and each pixel demodulates whatever lands on it. A Kinect v2 frame is built from ten correlation images, nine of which go into a three phase reconstruction at three modulation frequencies, which is how it sorts out a phase that has gone round more than once.
The consequence worth carrying away is that every pixel gets its own measurement, made at that photosite, with no reference to its neighbours. This is the one class where the output resolution really is the measurement resolution. The price is absolute resolution: a pixel has to be big enough to collect the light coming back, so these sensors run one or two orders of magnitude coarser than an ordinary video camera. A one megapixel time of flight sensor is a high end depth sensor and a poor image sensor.
Cameras in this class: the Azure Kinect DK, the Orbbec Femto models built on the same Microsoft module, and the Kinect v2.
The L515 is not this mechanism. It sweeps a beam with a mirror and reads one photodiode, so neither half of the below describes it.
Good at: measuring per pixel, so the output resolution really is the measurement resolution. Untextured walls, which defeat stereo entirely. Wide fields of view. Accuracy that holds up across the frame rather than falling away with distance.
Bad at: multipath, where light reaches a pixel by two routes and the phase reads long. Flying pixels along depth edges. Motion smear, because one frame is several subframes, which a shorter exposure reduces. Phase wrapping past the unambiguous range. Sunlight, and dark or shiny surfaces, all of which cost signal. Thermal drift until the unit is warm. Interference between units pointed at the same volume, which hardware sync fixes.
Multipath. Point one of these into the corner of a room and some of the infrared light reaches a pixel the long way round: out to one wall, off it onto the other, and only then back to the sensor. That light arrives carrying the phase of the longer route, mixed in with the light that came straight back, so the pixel reports a distance that belongs to neither and the corner comes out further away than it is. Shiny floors do the same thing, and so do concave surfaces generally. Microsoft's own accuracy figure for the Azure Kinect is quoted "without multi-path interference", which tells you how much of it there can be. Their fix is a higher modulation frequency plus a filter that spots the ambiguity and invalidates the pixel, and that is worth knowing about: the mitigation deletes the measurement rather than correcting it. For concavities the literature still calls it open.
Flying pixels. A pixel sees a patch of the scene rather than a point. Where that patch straddles an edge, part of it lands on the subject and part on whatever is behind, both returns arrive in the same pixel, and the depth that comes out is a blend of the two. What you get is a point floating in empty space between the foreground and the background, and the awkward part is that it looks exactly like a good one: the camera reports it with the same confidence as a measurement it got right. Formally this is multipath again at a much smaller scale. The fix is detection rather than correction once more, since a flying pixel sits in a patch of extremely high depth variance and a filter can throw it out, at the cost of taking real measurements with it.
Structured light
A projector throws a fixed pattern of infrared dots into the scene, and a camera a few centimetres away photographs it. Because the camera looks from a different place than the projector, the dots land in a different position depending on how far away the surface is, and the depth comes from working out how far each patch of the pattern moved.
It does that by comparison rather than by measurement. The camera stores a reference image of the same pattern falling on a flat plane at a known distance, then takes a small window of the live image, roughly nine pixels across, and slides it along the reference until it matches. Two things follow from that, and both of them matter more than they sound. Neighbouring depth pixels are computed from overlapping windows, so they are not independent measurements. And nothing finer than the spacing between the projected dots survives at all: geometric structures smaller than the distance between the points are lost.
Cameras in this class: the Orbbec Astra family and the original Kinect v1.
Good at: dense, stable depth close in, including on surfaces with no texture at all. Cheap, and light on the host, because the depth is computed on the device.
Bad at: sunlight, which washes the pattern out. Shadows, because the projector sits beside the camera and anything it cannot see has no depth. Overlapping units, which spoil each other's pattern, so the fix is geometric: move them, or stagger them in time. And detail that falls away with distance as the projected dots spread.
The projector's shadow. The projector sits a few centimetres from the camera, which is the whole reason the triangulation works, and it is also why there is a strip beside every object where no depth comes back. The camera can see past someone to the wall behind them; the projector, looking from a slightly different place, cannot light that strip. No pattern lands there, so there is nothing to match against the reference and no depth is estimated. It shows up as a hole down one side of every foreground object, always the same side, widest when the subject is close and the background far. Thin objects come off worst, because the missing strip is a decent fraction of their own width. A shorter baseline would shrink the shadow, but the depth error grows by the same factor, so that is a trade rather than a fix: with one unit the geometry is the geometry. A second sensor from another viewpoint fills the hole, at the price of two patterns overlapping, which is a failure of its own. At least this one is honest: those pixels come back marked invalid, unlike the flying pixels further up, which arrive looking like measurements.
Sunlight. Daylight carries plenty of infrared and the sensor cannot tell the sun's from its own. Enough of it and the camera saturates: the dots are still being projected, they are just no longer darker or brighter than what is around them, and a pattern with no contrast matches nothing. What you get is holes, or outdoors usually a whole frame of them. A Kinect v1 stops returning valid data at about a tenth of the ambient infrared level of direct sunlight, which is why these cameras are sold for indoor use and why one strong shaft of sun through a window can put a hole in an otherwise fine capture. There is no mitigation at the sensor. The narrowband filter over the camera is already there, and it is what makes the thing work indoors at all.
Stereo, passive and active
Two ordinary cameras, a known distance apart. A point in the scene lands at a different horizontal position in each image, that difference is the disparity, and the distance follows from it directly: depth is the focal length times the baseline, divided by the disparity. Nothing is emitted, which is why this is the only one of the four drawings with no cone of light in it.
Depth is inversely proportional to disparity, and that single fact is the character of the whole class. The same fraction of a pixel of matching error is worth more millimetres the further away the subject is, so the error grows with the square of the distance. Intel publishes the arithmetic for their own cameras: depth error is the distance squared, times the subpixel matching error, divided by focal length times baseline. Run it for a D435, using Intel's own worked parameters of a 50 mm baseline, a 424 pixel focal length and 0.08 pixels of matching error, and two metres comes out at about 15 mm. Six metres, same camera, same matcher, comes out at about 136.
Matching happens over a window here too, exactly as it does in structured light. Intel's D4xx cameras run a 7x7 census transform and aggregate the result over a 7x7 block, so neighbouring depth pixels share most of the image they were computed from. Their own engineers measured what that costs, which is the worked example at the end of this section.
Cameras in this class: the ZED 2 and ZED 2i passively, the RealSense D435 and D415 actively.
What the projector adds
Active stereo is the same two cameras doing the same arithmetic, with a projector between them throwing a pattern into the scene. The pattern is not a code to be recognised: the depth still comes from matching the two camera images against each other, not against anything stored. All the projector does is put texture on surfaces that had none, so the matcher has something to lock onto. That is why it is the standard fix for the one failure that kills passive stereo outright, and it is the only difference between the two diagrams above.
It is also why an active stereo camera still works outdoors in a way a structured light camera does not. When the sun washes the pattern out, structured light has nothing left to compare against and active stereo falls back to the texture that was already there.
Good at: daylight and outdoors, where every emitter-based sensor struggles. Long range. Any number of units in one room, because a passive camera projects nothing to interfere with. High resolution, since it is two ordinary image sensors.
Bad at: untextured surfaces, which give the matcher nothing to match. Active stereo exists exactly to fix that, by projecting a pattern to serve as texture. Occlusion, since a point has to be visible to both lenses. Error that grows with the square of distance, so a figure quoted at one metre says little about six. And host compute, because the matching usually happens on your machine.
A worked example of the lateral-detail point, from the people who build these: Intel's own engineers reported their R200 merging gaps between edges narrower than about five pixels at 480x360, roughly one percent of the distance to the subject. Every sensor in this class has an equivalent limit and almost none of them publish it.
Dropout on an untextured surface. This is the one failure in the class that is a hole rather than a degradation. The matcher works by finding the same patch of image in both cameras, and on a plain painted wall every patch looks like every other patch, so there is nothing to lock onto. What the left camera sees pins the point to a line and no further, and it is the right camera that has to say where along that line it sits. On a blank wall the patch matches about as well at every position along the search, and every position would put the point at a different distance. Zollhoefer puts it flatly: uniformly coloured objects cannot be reconstructed, because no reliable match between the two cameras can be found. What you get instead is an outlier, and the left to right consistency check these cameras run exists to throw outliers away, so what reaches you is a hole. Intel say the same about their own hardware, and even StereoLabs, who sell a passive camera, warn that white walls, green screens and reflective areas give outlier measurements and temporal instability. What makes this different from the projector shadow further up is that nothing went missing. The wall is evenly lit, both cameras see it perfectly well, the light comes back. None of what comes back is distinguishable from the rest of it. The fix is the projector, which is the entire reason active stereo exists: it is not a code to be decoded, it just puts texture where there was none, and Intel note that any external projector will do as long as it is on continuously or flickering faster than 50 kHz.
Error that grows with the square of the distance. The matching error is in disparity, and it stays roughly constant with distance: Intel work with 0.08 of a pixel, and the peer reviewed version of the same formula treats it as a constant of about a tenth. Depth is inversely proportional to disparity, so a fixed error in pixels is worth more millimetres the further away the surface is. Twice as far, four times the spread, with the same camera and the same match. It does not look like a hole or a spike. It looks like a surface that is solid and stable close up, gets noisier and softer the further back it goes, and breathes between frames at the far end of the range. StereoLabs put numbers on the same law for their own cameras: about one percent of the distance at close range, rising to about nine percent at the maximum. There is no filter for this one. Intel's own advice is the only mitigation, and it is to get as close to the subject as you can. A longer baseline or a longer focal length buys the same thing, but those are decisions you make when you choose the camera rather than while you are shooting.
The balance here is deliberate. Each class gets its real strengths and its real failures, and none of it is an argument for the technology my own apps happen to lean on.
Which one is in the camera you are looking at
The depth sensor comparison has a technology column for every camera worth comparing, alongside resolution, range, field of view and which of my apps support it. It also takes a position on what I would use for what, which this page deliberately does not: the best technology depends on the distance you work at and on the job, and that is a buying question rather than a physics one.