Tracking in Partially Known Scenes

We evaluated the tracking system with the integrated reconstruction of feature points on several image sequences. In the first sequence, as illustrated in Figure 7.1, the camera pose estimation is tested with a desktop scenario. A toy truck is used as a reference object and the tracking system is initialized with a generated line model of the toy truck. At the beginning the extrinsic camera parameters are set in such a way that the projection of the generated line model is in the center of the image. If the projected lines are close enough to the real object in the image, the line model is aligned and the first point features are initialized both on the model and on other areas of the image. During the point feature tracking phase the line model is just used for augmenting the scene. After the tracking is initialized, the camera pose is estimated by using only point features on the object, since these are the only points with a known 3D coordinate. Other features are detected, triangulated and refined during the further tracking. When the camera moves away from the truck, it is still possible to estimate the camera pose correctly. After the truck reappears in the image, features obtained in previous tracking steps are re-acquired, and no drift is visible in the augmentation.

For a clearly arranged visualization of the different steps of the tracking system we com-posed a result video with four image frames, which all point out different properties of the tracked features. In Figure 7.1 such an arrangement is illustrated.

All the four images represent the state of all the current features at the same time step.

In image (a) the results of the 2D feature tracking step with KLT features is visualized.

The green rectangles are successfully tracked features, for the red ones the tracking failed.

Mostly features which are located on object borders differ too much from the initially extracted template and can thus not be tracked successfully.

In (b) the 3D model of the reference object is shown from a different point of view together with the 3D covariances of the reconstructed feature points, which are represented by ellipsoids. It can be observed that the features have an initially high uncertainty along the viewing direction of the camera. While moving the camera through the scene, the feature positions are refined with an extended Kalman filter. The uncertainty of the feature positions is thereby decreased, which is visualized by shrinking ellipsoids.

7.5. Experimental Evaluation

(a) (b)

(c) (d)

Figure 7.1.: Illustration of the feature tracking and reconstruction process. All images represent the states of the features at the same camera frame. In (a) the results of the 2D feature tracking step are pointed out. In (b) the reference object and the 3D covariances of the reconstructed feature points are shown.

Figure (c) illustrates a projection of the 3D covariances in the image plane.

Features with a high absolute value of the covariance are colored red, features with a small uncertainty are colored green. In (d) the original image is aug-mented with the line model of the reference object and a virtual character standing on the table.

7. Tracking in Unknown Scenes

Figure 7.2.: Tracking results showing the ability of handling occlusion and even the total removal of the object from the scene.

Image (c) of Figure 7.1 illustrates the same uncertainty regions of the feature points as a projection in the image plane. The absolute value of the uncertainty is color coded in that way that the color value is shifted with increasing precision from red to green.

Finally in (d) the original image is overlaid with the line model with which the tracking was initialized and an additional virtual character standing on the table. The purpose of this frame is to evaluate, if the camera pose is estimated properly and that virtual objects are always placed correctly in the scene.

Figure 7.2 shows the tracking results of another sequence of the same scenario with an image size of 320x240 pixels. This time the initialization object is occluded and removed from the scene. Since enough other features have been triangulated and refined success-fully, it is still possible to keep tracking. The line model, which is used for augmentation, sticks at the same position in the real world.

In another sequence an industrial control unit is used as a reference object. In Figure 7.3 some frames of this sequence can be seen. Again the tracking is initialized with a line-model, which is generated out of a given VRML-model. After the initialization feature points are extracted in the whole image, but only those points which are located on the known geometry can be used instantly for the camera tracking. The other features are triangulated and refined when the camera is moved through the scene. Again it can be observed that due to the refinement process the uncertainties of the reconstructed feature points shrink during the tracking. When a person moves into the scene, some feature are occluded and the 2D tracking step for these features fails. In the first column of Figure 7.3

7.5. Experimental Evaluation

(a) (b) (c)

Figure 7.3.: Tracking results demonstrating the robustness against occlusion. In (a) the KLT features are shown, in (b) the covariances of the reconstructed features, and in (c) the feature with their 2D unceratinties can be seen.

all those occluded features are colored red. Since enough valid 2D features are available in every frame, the pose can be estimated successfully despite the occlusion.