1 Computer Vision cmput 428/615 Basic 2D and 3D geometry and Camera models Martin Jagersand2 The equation of projection Intuitively: How do we develop...

Author:
Adelia Campbell

0 downloads 86 Views 4MB Size

Basic 2D and 3D geometry and Camera models Martin Jagersand

The equation of projection How do we develop a consistent mathematical framework for projection calculations? Mathematically: Intuitively: • Cartesian coordinates: x y (x, y, z) → ( f , f ) z z • Projectively: x = PX

Challenges in Computer Vision: What images don’t provide

lengths

depth

Distant objects are smaller

Visual ambiguity

•Will the scissors cut the paper in the middle?

Ambiguity

•Will the scissors cut the paper in the middle? NO!

Visual ambiguity

•Is the probe contacting the wire?

Ambiguity

•Is the probe contacting the wire? NO!

Visual ambiguity

•Is the probe contacting the wire?

Ambiguity

•Is the probe contacting the wire? NO!

History of Perspective Prehistoric:

Roman

Perspective: Da Vinci

Visualizing perspective: Dürer

Perspectograph 1500’s

Parallel lines meet Image plane

common to draw image plane in front of the focal point

Centre of projection

3D world

Perspective Imaging Properties Challenges with measurements in multiple images: • Distances/angles change • Ratios of dist/angles change • Parallel lines intersect 90

What is preserved? Invariants: • Points map to points • Intersections are preserved • Lines map to lines • Collinearity preserved • Ratios of ratios (cross ratio) • Horizon

horizon

What is a good way to represent imaged geometry?

Vanishing points • each set of parallel lines (=direction) meets at a different point – The vanishing point for this direction – How would you show this?

• Sets of parallel lines on the same plane lead to collinear vanishing points. – The line is called the horizon for that plane

Geometric properties of projection • • • •

Points go to points Lines go to lines Planes go to whole image Polygons go to polygons

• Degenerate cases – line through focal point to point – plane through focal point to line

Polyhedra project to polygons • (because lines project to lines)

Junctions are constrained • This leads to a process called “line labelling” – one looks for consistent sets of labels, bounding polyhedra – disadv - can’t get the lines and junctions to label from real images

Back to projection

• Cartesian coordinates:

x y (x, y, z) → ( f , f ) z z

We will develop a framework to express projection as x=PX, where x is 2D image projection, P a projection matrix and X is 3D world point.

Basic geometric transformations:

Translation • A translation is a straight line movement of an object from one postion to another. A point (x,y) is transformed to the point (x’,y’) by adding the translation distances Tx and Ty: x’ = x + Tx y’ = y + Ty z’ = z + Tz

Coordinate rotation •Example: Around y-axis

p′ =

Z’

′

x y′ z′

cos = 0 à sin

P X’ X

0 1 0

sin x 0 y = R yp cos z

Euler angles •Note: Successive rotations. Order matters.

R = R zR y R x =

cos 0 sin 1 0 0 cos ϕ à sinϕ 0 1 0 0 cos ψ à sinψ sinϕ cos ϕ 0 0 à sin 0 cos 0 sinψ cos ψ 0 0 1

Rotation and translation •Translation t’ in new o’ coordinates

Z’

P

cos ÷ p′ = 0 à sin ÷

X’ X

0 1 0

sin ÷ 0 p + t′ cos ÷

Basic transformations

Scaling • A scaling transformation alters the scale of an object. Suppose a point (x,y) is transformed to the point (x',y') by a scaling with scaling factors Sx and Sy, then: x' = x Sx y' = y Sy z' = z Sz • A uniform scaling is produced if Sx = Sy = Sz .

Basic transformations

Scaling The previous scaling transformation leaves the origin unaltered. If the point (xf,yf) is to be the fixed point, the transformation is: x' = xf + (x - xf) Sx y' = yf + (y - yf) Sy This can be rearranged to give: x' = x Sx + (1 - Sx) xf y' = y Sy + (1 - Sy) yf

Affine Geometric Transforms

In general, a point in n-D space transforms by P’ = rotate(point) + translate(point) In 2-D space, this can be written as a matrix equation:

x ' Cos(θ ) − Sin (θ ) x tx = + y ' Sin (θ ) Cos(θ ) y ty In 3-D space (or n-D), this can generalized as a matrix equation: p’ = R p + T

or

p = Rt (p’ – T)

A Simple 2-D Example

p = (0,1)’

p = (1,0)’

Suppose we rotate the coordinate system through 45 degrees (note that this is measured relative to the rotated system!

x ' Cos(π / 4) − Sin (π / 4) 1 = y ' Sin (π / 4) Cos(π / 4) 0 x ' Cos(π / 4) = y ' Sin (π / 4)

x' Cos(π / 4) − Sin (π / 4) 0 = y ' Sin (π / 4) Cos(π / 4) 1 x' − Sin (π / 4) = y ' Cos(π / 4)

Matrix representation and Homogeneous coordinates • Often need to combine several transformations to build the total transformation. • So far using affine transforms need both add and multiply • Good if all transformations could be represented as matrix multiplications then the combination of transformations simply involves the multiplication of the respective matrices • As translations do not have a 2 x 2 matrix representation, we introduce homogeneous coordinates to allow a 3 x 3 matrix representation.

How to translate a 2D point:

•Old way: •New way:

Relationship between 3D homogeneous and inhomogeneous • The Homogeneous coordinate corresponding to the point (x,y,z) is the triple (xh, yh, zh, w) where: xh = wx yh = wy zh = wz We can (initially) set w = 1. • Suppose a point P = (x,y,z,1) in the homogeneous coordinate system is mapped to a point P' = (x',y',z’,1) by a transformations, then the transformation can be expressed in matrix form.

Matrix representation and Homogeneous coordinates • For the basic transformations we have: – Translation x′ 1 y′ 0 ′ P = ′ = z 0 w 0

– Scaling

x′ sx y′ 0 ′ P = ′ = z 0 w 0

0 1 0 0 0 sy 0 0

0 0 1 0

Tx x Ty y Tz z w 1 0 0 sz 0

x 0 0 y z 0 w 1

Geometric Transforms

Using the idea of homogeneous transforms, we can write:

T R p p' = 0 0 0 1 R and T both require 3 parameters.

R=

cos 0 sin 1 0 0 cos ϕ à sin ϕ 0 1 0 0 cos ψ à sin ψ sin ϕ cos ϕ 0 0 à sin 0 cos 0 sin ψ cos ψ 0 0 1

Geometric Transforms

If we compute the matrix inverse, we find that

R' − R' T p' p = 1 0 0 0 R and T both require 3 parameters. These correspond to the 6 extrinsic parameters needed for camera calibration

Rotation about a Specified Axis •It is useful to be able to rotate about any axis in 3D space •This is achieved by composing 7 elementary transformations (next slide)

Rotation through θ about Specified Axis y

y

P2 P1 x

x z

z initial position y

z

y

translate P1 to origin y

x rotate through requ’d angle, θ

x z

rotate so that P2 lies on z-axis (2 rotations) y P2

x z

rotate axis z to orig orientation

P1 x translate back

Comparison: •Homogeneous coordinates – Rotations and translations are represented in a uniform way – Successive transforms are composed using matrix products: y = Pn*..*P2*P1*x

•Affine coordinates – Non-uniform representations: y = Ax + b – Difficult to keep track of separate elements

Camera models and projections Geometry part 2. •Using geometry and homogeneous transforms to describe:

x

– Perspective projection – Weak perspective projection – Orthographic projection

y

z y x

The equation of projection • Cartesian coordinates: – We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) – Ignore the third coordinate, and get

x y (x, y, z) → ( f , f ) z z

The camera matrix • Homogenous coordinates for 3D – four coordinates for 3D point – equivalence relation (X,Y,Z,T) is the same as (k X, k Y, k Z,k T)

• Turn previous expression into HC’s – HC’s for 3D point are (X,Y,Z,T) – HC’s for point in image are (U,V,W)

U 1 0 V = 0 1 W 0 0

0 0 1

f

X 0 Y 0 Z 0 T

U V (U ,V ,W ) → ( , ) = (u, v ) W W

Camera parameters • Issue – camera may not be at the origin, looking down the z-axis – extrinsic parameters – one unit in camera coordinates may not be the same as one unit in world coordinates – intrinsic parameters - focal length, principal point, aspect ratio, angle between axes, etc. X U Transforma tion Transforma tion Transforma tion Y representi ng representi ng V = representi ng W intrinsic parameters projection model extrinsic parameters Z T

Note: f moved from proj to intrinsics!

Intrinsic Parameters Intrinsic Parameters describe the conversion from metric to pixel coordinates (and the reverse) xmm = - (xpix – ox) sx ymm = - (ypix – oy) sy

x − f / sx y = 0 w 0 pix

or

0 − f / sy 0

ox x o y y = M int p 1 w mm

Note: Focal length is a property of the camera and can be incorporated as above

Example: A real camera • Laser range finder

• Camera

Relative location Camera-Laser • Camera

• Laser

R=10deg T=(16,6,-9)’

In homogeneous coordinates • Rotation:

• Translation

cos à 10 0 sinà 10 R= 0 1 0 à sinà 10 0 cos à 10

1 0 T= 0 0

0 1 0 0

0 0 1 0

16 6 à9 1

Full projection model • Camera internal parameters

p camera =

0.985 0 0.174 0

0 1 0 0

1278.6657 0 0

à 0.174 0 0.985 0

0 1 00 0 0 1 0

Extrinsic rot and translation

• Camera projection

0 1659.5688 0 0 1 0 0

0 0 1 0

256 240 1

1 0 0

0 1 0

0 0 1

0 0 0

16 0.6612 22262 6 à 10.55 = 16755 à9 108.0 97.47 1 1

Camera parameters • Issue – camera may not be at the origin, looking down the z-axis – extrinsic parameters – one unit in camera coordinates may not be the same as one unit in world coordinates – intrinsic parameters - focal length, principal point, aspect ratio, angle between axes, etc. X U Transforma tion Transforma tion Transforma tion Y representi ng representi ng V = representi ng W intrinsic parameters projection model extrinsic parameters Z T

Note: f moved from proj to intrinsics!

Result • Camera image

• Laser measured 3D structure

Hierarchy of different camera models xorth xwp xparap xpersp

Image plane

Object plane

X0(origin)

Camera center

Perspective: non-linear

Weak perspective: linear approx

Orthographic: lin, no scaling

Para-perspective: lin

Orthographic projection

u=x v= y

The fundamental model for orthographic projection

X U 1 0 0 0 Y V = 0 1 0 0 Z W 0 0 0 1 T

Perspective and Orthographic Projection

perspective

Orthographic (parallel)

Weak perspective

• Issue – perspective effects, but not over the scale of individual objects – collect points into a group at about the same depth, then divide each point by the depth of its group – Adv: easy – Disadv: wrong

u = Tx v = Ty T = f /Z

The fundamental model for weak perspective projection

U 1 0 0 V = 0 1 0 W 0 0 0

X 0 Y 0 Z f / Z * T

Note Z* is a fixed value, usually mean distance to scene

Weak perspective projection for an arbitrary camera pose R,t Weak perspective projection 1T t1 α x r 2T P∞ = αy t2 (7dof) r 1 0 1 / k

Full Affine linear camera Affine camera (8dof) 1T α r s t1 x m11 m12 m13 t1 2T m21 m22 m23 t2 P = PA = αy r t A 2 0 0 0 1 1 0 1 / k 1 0 0 0 PA = [3 × 3 affine]0 1 0 0[4 × 4 affine] 0 0 0 1 1. Affine camera=camera with principal plane coinciding with Π∞ 2. Affine camera maps parallel lines to parallel lines 3. No center of projection, but direction of projection PAD=0 (point on Π∞)

Hierarchy of camera models xorth xwp xparap xpersp

Object plane

Image plane

X0(origin)

Camera center

Perspective: Ppersp

f =

Orthographic: Porth

f

i = j 0T

i tx j t y 1 k t z tx t y 1

Weak perspective:

k Pwp =

i k j 1 0T

tx t y 1

Para-perspective: First order approximation of perspective

Camera Models • Internal calibration: • Weak calibration: • Affine calibration: • Stratification of stereo vision:

- characterizes the reconstructive certainty of weakly, affinely, and internally calibrated stereo rigs

C sim C aff C proj C inj

up to a similarity (scaled Euclidean transformation)

up to an affine transformation of task space

up to a projective transformation of task space

reconstruction up to a bijection of task space

Visual Invariance

≡ sim ≡ aff ≡ proj ≡ inj

≡ aff ≡ proj ≡ inj

≡ proj ≡ inj

≡ inj

Perspective Camera Model Structure Assume R and T express camera in world coordinates, then ' R − R' T w c p p = 1 0 0 0

Combining with a perspective model (and neglecting internal parameters) yields

− R' x R' x T − R' y R' y T c u = M wp = Rz − RzT f f

w p

Note the M is defined only up to a scale factor at this point! If M is viewed as a 3x4 matrix defined up to scale, it is called the projection matrix.

Perspective Camera Model Structure Assume R and T express camera in world coordinates, then ' R − R' T w c p p = 1 0 0 0

Combining with a weak perspective model (and neglecting internal parameters) yields

− R' x R' x T − R' y R' y T c u = M wp = 0 Rz ( P − T ) f Where

P

w p

is the nominal distance to the viewed object

Other Models • The affine camera is a generalization of weak perspective. • The projective camera is a generalization of the perspective camera. • Both have the advantage of being linear models on real and projective spaces, respectively. • But in general will recover structure up to an affine or projective transform only. (ie distorted structure)

Camera Internal Calibration Recall: Intrinsic Parameters Intrinsic Parameters describe the conversion from metric to pixel coordinates (and the reverse) xmm = - (xpix – ox) sx ymm = - (ypix – oy) sy or

x − 1 / sx y = 0 w 0 pix

0 − 1/ sy 0

o x x o y y = M int p 1 w mm

CAMERA INTERNAL CALIBRATION Compute Sx Focal length = 1/ Sx

Known distance d

rk i = ( xi − ox ) s x d r = ( xi +1 − xi ) s x d

known regular offset r A simple way to get scale parameters; we can compute the optical center as the numerical center and therefore have the intrinsic parameters

Camera calibration • Issues: – what are intrinsic parameters of the camera? – what is the camera matrix? (intrinsic+extrinsic)

• General strategy: – view calibration object – identify image points – obtain camera matrix by minimizing error – obtain intrinsic parameters from camera matrix

• Error minimization: – Linear least squares – easy problem numerically – solution can be rather bad – Minimize image distance – more difficult numerical problem – solution usually rather good, but can be hard to find – start with linear least squares – Numerical scaling is an issue

Stereo Vision

• GOAL: Passive 2camera system for triangulating 3D position of points in space to generate a depth map of a world scene. • Humans use stereo vision to obtain depth

Stereo depth calculation: Simple case, aligned cameras DISPARITY= (XL - XR) Z

Similar triangles: Z = (f/XL) X Z= (f/XR) (X-d)

Solve for X: (f/XL) X = (f/XR) (X-d) X = (XL d) / (XL - XR)

Solve for Z: Z =

d*f (XL - XR)

XL f

(0,0)

XR

(d,0)

X

Epipolar constraint

Special case: parallel cameras – epipolar lines are parallel and aligned with rows

Stereo measurement example:

• Left image Resolution = 1280 x 1024 pixels f = 1360 pixels

• Right image Baseline d = 1.2m Q: How wide is the hallway

How wide is the hallway? General strategy •Similar triangles: W v = Z f

•Need depth Z •Then solve for W

W

Z v

f

How wide is the hallway? Steps in solution: 1. Compute focal length f in meters from pixels 2. Compute depth Z using stereo formula (aligned camera planes) Z =

d*f (XL - XR)

3. Compute width: W =Z

v f

Focal length: Here screen projection is metric image plane.

f = 1360 pixels 1360 f = * 0.224 = 0.238m 1280

0.224m is 1280 pixels

How wide… Depth calculation XL = 0.144m

XR = 0.074m

Disparity: XL – XR = 0.07m (Note in the disparity calculation the choice of reference (here the edge) doesn’t matter. But in the case of say X-coordinate calculation it should be w.r.t. the center of the image as in the stereo formula derivation

• Depth Z=

1.2 * 0.238 = 4.1m 0.07

How wide…? Answer: • Similar triangles: W =Z

v f

W

• The width of the hallway is: W = 4.1*

0.135 = 2.3m 0.238

V = 0.135m

Z

v

f

Our partners will collect data and use cookies for ad personalization and measurement. Learn how we and our ad partner Google, collect and use data. Agree & Close