3D Data Processing, Visualization, and Transmission, Third International Symposium on

The paper introduces a data collection system and a processing pipeline for automatic geo-registered 3D reconstruction of urban scenes from video. The system collects multiple video streams, as well as GPS and INS measurements in order to place the reconstructed models in geo- registered coordinates. Besides high quality in terms of both geometry and appearance, we aim at real-time performance. Even though our processing pipeline is currently far from being real-time, we select techniques and we design processing modules that can achieve fast performance on multiple CPUs and GPUs aiming at real-time performance in the near future. We present the main considerations in designing the system and the steps of the processing pipeline. We show results on real video sequences captured by our system. View full abstract»

3D city modeling using computer vision is very challenging. A typical city contains objects which are a nightmare for some vision algorithms, while other algorithms have been designed to identify exactly these parts but, in their turn, suffer from other weaknesses which limit their application. For instance, moving cars with metallic surfaces can degrade the results of a 3D city reconstruction algorithm which is primarily based on the assumption of a static scene with diffuse reflection properties. On the other hand, a specialized object recognition algorithm could be able to detect cars, but also yields too many false positives without the availability of additional scene knowledge. In this paper, the design of a cognitive loop which intertwines both aforementioned algorithms is demonstrated for 3D city modeling, proving that the whole can be much more than the simple sum of its parts. A cognitive loop is the mutual transfer of higher knowledge between algorithms, which enables the combination of algorithms to overcome the weaknesses of any single algorithm. We demonstrate the promise of this approach on a real-world city modeling task using video data recorded by a survey vehicle. Our results show that the cognitive combination of algorithms delivers convincing city models which improve upon the degree of realism that is possible from a purely reconstruction-based approach. View full abstract»

This paper presents a method for obtaining accurate dense elevation and appearance models of terrain using a single camera on-board an aerial platform. Applications of this method include geographical information systems, robot path planning, immersion and visualization, and surveying for scientific purposes such as watershed analysis. When given geo-registered images, the method can compute terrain maps on-line in real time. This algorithm, called the recursive multi-frame planar parallax algorithm, is a recursive extension of Irani et al.'s multi-frame planar parallax framework and in theory, with perfectly registered imagery, it will produce range data with error expected to increase between linearly and with the square root of the range, depending on image properties and whether other constants such as framerate and vehicle velocity are held constant. This is an improvement over stereo systems whose expected errors are proportional to the square of the range. We show experimental evidence on synthetic imagery and on a real video sequence taken in an experiment for autonomous helicopter landing. View full abstract»

This paper presents a new technique for estimating a multi- view reconstruction given pair-wise Euclidean reconstructions up to rotations, translations and scales. The partial reconstructions are glued by the following three step procedure: (i) Camera rotations consistent with all reconstructions are estimated linearly, (ii) All the pair-wise reconstructions are modified according to the new rotations and refined by bundle adjustment while keeping the corresponding rotations same. (Hi) The refined rotations are used to estimate both camera translations and 3D points using Second Order Cone Programming by minimizing the Linfin- norm. We introduce a new criterion for evaluating importance of an epipolar geometry in influence on the overall 3D geometry. The estimated importance is used to reweight data in the above algorithm to better handle unequiponderantly captured images. The performance of the proposed method is demonstrated on difficult wide base-line image sets. View full abstract»

In this paper we present a prototype system for image based localization in urban environments. Given a database of views of city street scenes tagged by GPS locations, the system computes the GPS location of a novel query view. We first use a wide-baseline matching technique based on SIFT features to select the closest views in the database. Often due to a large change of viewpoint and presence of repetitive structures, a large percentage of matches (> 50%) are not correct correspondences. The subsequent motion estimation between the query view and the reference view, is then handled by a novel and efficient robust estimation technique capable of dealing with large percentage of outliers. This stage is also accompanied by a model selection step among the fundamental matrix and the homography. Once the motion between the closest reference views is estimated, the location of the query view is then obtained by triangulation of translation directions. Approximate solutions for cases when triangulation cannot be obtained reliably are also described. The presented system is tested on the dataset used in ICCV 2005 Computer Vision Contest and is shown to have higher accuracy than previous reported results. View full abstract»

We present a probabilistic framework for correspondence and egomotion. First, we suggest computing probability distributions of correspondence. This has the advantage of being robust to points subject to the aperture effect and repetitive structure, while giving up no information at feature points. Additionally, correspondence probability distributions can be computed for every point in the scene. Next, we generate a probability distribution over the motions, from these correspondence probability distributions, through a probabilistic notion of the epipolar constraint. Finding the maximum in this distribution is shown to be a generalization of least-squared epipolar minimization. We will show that because our technique allows so much correspondence information to be extracted, more accurate ego- motion estimation is possible. View full abstract»

We present a new algorithm for improving an available {conservative) estimate of the shape of an object using constraints from ray-tracing. In particular, we exploit incoherences between the lit portions of the object - detected on a set of acquired images - and the shadows that the current estimate casts on itself Whenever a contradiction is found the current estimate is modified in order to remove the inconsistency. Sufficient conditions for the correctness of the algorithm and a discussion of their validity are provided. Finally, we describe a simple implementation of the method and present some preliminary experimental results from computer simulations. View full abstract»

This paper describes a non-intrusive method to estimate the gaze direction of a person by using stereo cameras. First, facial features are tracked with an adapted particle filtering algorithm using factorized likelihoods to estimate the 3D head pose. Next the 3D gaze vector is calculated by estimating the eyeball center and the cornea center of both eyes. For the intended application of visual perception research, we also propose a new screen registration scheme to accurately locate a planar screen in world coordinates within 2 mm error. We combine the 3D screen location with the 3D gaze vectors from both eyes to establish a point of focus on the screen. The experimental results indicate that an average error of the gaze direction of about 4.6deg can be achieved and an average error of about 4 mm for the focus point location at a viewing distance of 50 cm. View full abstract»

When an observer moves through a rigid 3D scene, points that are near to the observer move with a different image velocity than points that are far away. The difference between image velocity vectors is the direction of motion parallax. This direction vector points towards the observer's translation direction. Hence estimates of the direction of motion parallax are useful for estimating the observer's translation direction. Standard ways to compute the direction of motion parallax either rely on precomputed optical flow, or rely on motion compensation to remove the local image shift caused by observer rotation. Here we present a simple Fourier-based method for estimating the direction of motion parallax directly, that does not require optical flow and motion compensation. The method is real-time and performs accurately for image regions in which multiple motions are present. View full abstract»

Extended Kalman filters (EKF) have been proposed to estimate ego-motion and to recursively update scene structure in the form of 3-D positions of selected prominent features from motion and stereo sequences. Previous methods typically accommodate no more than a few dozen features for real-time processing. To maintain motion estimation accuracy, this calls for high contrast images to compute image feature locations with precision. Within manmade environments, various prominent corner points exist that can be extracted and tracked with required accuracy. However, prominent features are more difficult to localize precisely in natural scenes. Statistically, more feature points become necessary to maintain the same level of motion estimation accuracy and robustness. However, this imposes a computational burden beyond the capability of EKF-based techniques for real-time processing. A sequential dual EKF estimator utilizing stereo data is proposed for improved computation efficiency. Two important issues, unbiased estimation and stochastic stability are addressed. Furthermore, the dynamic feature set is handled in a more effective, efficient and robust way. Experimental results to demonstrate the merits of the new theoretical and algorithmic developments are presented. View full abstract»

This paper introduces a novel imaging system composed of an array of spherical mirrors and a single high-resolution digital camera. We describe the mechanical design and construction of a prototype, analyze the geometry of image formation, present a tailored calibration algorithm, and discuss the effect that design decisions had on the calibration routine. This system is presented as a unique platform for the development of efficient multi-view imaging algorithms which exploit the combined properties of camera arrays and non-central projection catadioptric systems. Initial target applications include data acquisition for image-based rendering and 3D scene reconstruction. The main advantages of the proposed system include: a relatively simple calibration procedure, a wide field of view, and a single imaging sensor which eliminates the need for color calibration and guarantees time synchronization. View full abstract»

This paper demonstrates that, for axial non-central optical systems, the equation of a 3D line can be estimated using only four points extracted from a single image of the line. This result, which is a direct consequence of the lack of vantage point, follows from a classic result in enumerative geometry: there are exactly two lines in 3-space which intersect four given lines in general position. We present a simple algorithm to reconstruct the equation of a 3D line from four image points. This algorithm is based on computing the Singular Value Decomposition (SVD) of the matrix of Plucker coordinates of the four corresponding rays. We evaluate the conditions for which the reconstruction fails, such as when the four rays are nearly coplanar. Preliminary experimental results using a spherical catadioptric camera are presented. We conclude by discussing the limitations imposed by poor calibration and numerical errors on the proposed reconstruction algorithm. View full abstract»

Thanks to their high performance and programmability, the latest graphics cards can now be used for scientific purpose. They are indeed very efficient parallel single instruction multiple data (SIMD) machines. This new trend is called general purpose computation on graphics processing unit (GPGPU). Regarding the stereo problem, variational methods based on deformable models provide dense, smooth and accurate results. Nevertheless, they prove to be slower than usual disparity-based approaches. In this paper, we present a dense stereo algorithm, handling occlusions, using three cameras as inputs and entirely implemented on a graphics processing unit (GPU). Experimental speedups prove that our approach is efficient and perfectly adapted to the GPU, leading to nearly video frame rate reconstruction. View full abstract»

Many applications in computer graphics need high level shape descriptions, in order to benefit from a global understanding of shapes. Topological approaches enable pertinent surface decompositions, providing structural descriptions of 3D polygonal meshes; but in practice, their use raises several difficulties. In this paper, we present a novel method for the construction of invariant high level Reeb graphs, topological entities that give a good overview of the shape structure. With this aim, we propose an accurate and straightforward feature point extraction algorithm for the computation of an invariant and meaningful quotient function. Moreover, we propose a new graph construction algorithm, based on an analysis of the connectivity evolutions of discrete level lines. This algorithm brings a practical solution for the suppression of non-significant critical points over piecewise continuous functions, providing meaningful Reeb graphs. Presented method gives accurate results, with satisfactory execution times and without input parameter. The geometrical invariance of resulting graphs and their robustness to variation in model pose and mesh sampling make them good candidates for several applications, like shape deformation (experimented in this paper), recognition, compression, indexing, etc. View full abstract»

We present a high performance reconstruction approach, which generates true 3D models from multiple views with known camera parameters. The complete pipeline from depth map generation over depth image integration to the final 3D model visualization is performed on programmable graphics processing units (GPUs). The proposed pipeline is suitable for long image sequences and uses a plane- sweep depth estimation procedure optionally employing robust image similarity functions to generate a set of depth images. The subsequent volumetric fusion step combines these depth maps into an impicit surface representation of the final model, which can be directly displayed using GPU- based ray casting methods. Depending on the number of input views and the desired resolution of the final model the computing times range from several seconds to a few minutes. The quality of the obtained models is illustrated with real-world datasets. View full abstract»

Camera models are essential infrastructure in computer vision, computer graphics, and visualization. The most frequently used camera models are based on the single- viewpoint constraint. Removing this constraint brings the advantage of improved flexibility in camera design. However, prior camera models that eliminate the single- viewpoint constraint are inefficient. We describe an approximate model for coherent general cameras, which projects efficiently with user chosen accuracy. The rays of the general camera are partitioned into simple cameras that approximate the camera locally. The simple cameras are modeled with k- ray cameras, a novel class of non-pinhole cameras. The rays of a k-ray camera interpolate between k construction rays. We analyze several variants of k-ray cameras. The resulting compound camera model is efficient because the number of simple cameras is orders of magnitude lower than the original number of rays camera, and because each simple camera offers closed-form projection. View full abstract»

Applications such as robot navigation and augmented reality require high-accuracy dense disparity maps in real-time and online. Due to time constraint, most realtime stereo applications rely on local winner-take-all optimization in the disparity computation process. These local approaches are generally outperformed by offline global optimization based algorithms. However, recent research shows that, through carefully selecting and aggregating the matching costs of neighboring pixels, the disparity maps produced by a local approach can be more accurate than those generated by many global optimization techniques. We are therefore motivated to investigate whether these cost aggregation approaches can be adopted in real-time stereo applications and, if so, how well they perform under the real-time constraint. The evaluation is conducted on a real-time stereo platform, which utilizes the processing power of programmable graphics hardware. Several recent cost aggregation approaches are also implemented and optimized for graphics hardware so that real-time speed can be achieved. The performances of these aggregation approaches in terms of both processing speed and result quality are reported. View full abstract»

In this paper, the depth cue due to the assumption of texture uniqueness is reviewed. The spatial direction over which a similarity measure is optimized, in order to establish a stereo correspondence, is considered and methods to increase the precision and accuracy of stereo reconstructions are presented. An efficient implementation of the above methods is offered, based on optimizations that evaluate potential correspondences hierarchically, in the spatial and angular dimensions. Furthermore, the expansion of the above techniques in a multi-view framework where calibration errors cause the misregistration of individually obtained reconstructions are considered, and a treatment of the data is proposed for the elimination of duplicate reconstructions of a single surface point. Finally, a processing step is proposed for the increase of reconstruction precision and post-processing of the final result. The above contributions are integrated in a generic and parallelizable implementation of the uniqueness constraint to observe speedup and increase in the fidelity of surface reconstruction. View full abstract»

We present an alternate approach to the problem of structure from motion (SfM) with noisy point measurements. With no information available about the joint density of three-dimensional points, the assumption of independence is the only reasonable one. With this assumption alone, the process of the factorization of the observed projections of inaccurately measured 3 dimensional points into motion and shape matrices is a blind source separation (demixing) problem in the presence of noise. This approach is very general, allowing the extension of all previous work on source separation to be applied to SfM in an information theory context. A significant reduction in the error of the estimation of the motion and shape matrices over other methods is possible. View full abstract»

A prerequisite to calibrated camera pose estimation is the construction of a camera neighborhood adjacency graph, a connected graph defining the pose neighbors of the camera set. Pose neighbors to a camera C are images containing sufficient overlap in image content with the image from C that they can be used to correctly estimate the pose of C using structure-from-motion techniques. In a video stream, the camera neighborhood adjacency graph is often a simple connected path; frame poses are only estimated relative to their immediate neighbors. We propose a novel method to build more complex camera adjacency graphs that are suitable for estimating the pose of large numbers of wide- and narrow-baseline images. We employ Content-Based Image Retrieval techniques to identify similar images likely to be graph neighbors. We also develop an optimization to improve graph accuracy that is based on an observation of common camera motions taken when photographing with the intent of structure-from-motion. Our results substantiate the use of our method for determining neighbors for pose estimation. View full abstract»