What is a Morphable or Active Appearance Model ?
On Image Alignment
Image Alignment is about getting
$$ generated\_image(x; \pp) = input\_image(x) $$
true, with respect to parameters \(\pp\).
Written as residual
$$ R(x; \pp) = generated\_image(x; \pp) - input\_image(x) \overset{!}{=} 0 $$
it is solved using the Sum of Least Squares method:
$$
\sum_{x}^{pixel} R(x; \pp)^2
$$
Lucas and Kanade use a constant template.
They warp the input to the coordinate frame of the template \(T(x)\) with respect to $\pp$.
On warping an image
$$
\mathcal{I}(x; \pp) = I( W(x; \pp))
$$
How to solve the LSQ optimization problem ?
Consider optimizing $ f: \mathbb{R}^n \rightarrow \mathbb{R} $
We use a variation of the steepest descent method:
\( x_{k+1} = x_k + \alpha d_k \)
Gradient Descent
\( x_{k+1} = x_k + \alpha d_k \)
\( d_k = -\JAC(x_k) \)
(With $\JAC$ being Jacobian of $f$ )
Newtons method
Use a second order approximation of $f(x)$
\( d(x^k) = -\HES^{-1} (x_k) \JAC(x_k) \)
Gauss-Newton approximation
"Simply" approximating $\HES \approx 2 \JAC^T \JAC$
only works with with LSQ - problems (we are lucky here)
high chance to fail if guess is far away from solution
This is what we use !
Lets apply this to image alignment.
We modify the residual function $R(x; \pp)$ to be optimized by updating with $\dp$ and $\pp$ being a good estimate:
$$
R(x; \pp + \dp)
$$
linearize (first order Taylor)
with respect to $\dp$ derive $\sum_{x} R(x; \pp + \dp)^2$
closed form solution $\dp = \dots$
What is missing for being a AAM ?
1. Deformation
Essentially $W(x; \pp)$ is just more complex
(contrary to just having affine transformations)
AAMs use the piecewise affine transformation.
Transform from $S_0$ to $S(\pp)$:
2. Texture variation
Constant $T(x)$ is replaced by variable $A(x; \lmbda)$
so we get a new residual
$$
R(x; \pp, \lmbda) = \mathcal{I}(x; \pp) - A(x; \lmbda)
$$
Just apply the same algorithm as before to get a closed form solution for both update parameters:
$$
\dlmbda = \dots
\\
\dp = \dots
$$
How are $ S(\pp) $ and $ A(x; \lmbda)$ structured ?
$$
S(\pp) = S_0 + \sum_{i}^{} p_i S_i
\\
A(x; \lmbda) = A_0(x) + \sum_{i}^{} \lambda_i A_i(x)
$$
How are $ S(\pp) $ and $ A(x; \lmbda)$ obtained ?
Just a short word on how the warp function for AAMs is constructed / learned.
... is probably the most prominent implementation
Composite warp
Additive approach (what we have seen so far)
$ W(x, \pp_{i+1}) = W(x, \pp_{i} + \dp) $
Compositional approach
$$
\begin{align}
W(x, \pp_{i+1}) &= W(x, \pp_i) \circ W(x, \dp)\\
&= W(W(x, \pp_i), \dp)
\end{align}
$$
Side-by-side comparison
Inverting roles
from forward compositional
$$
I(W(x; \pp) \circ \color{red}{W(x; \dp)}) = T(x)
$$
to inverse compositional
$$
\begin{align}
I(W(x; \pp)) &= T(\color{red}{W(x; \dp)^{-1}})\\
&= T(\color{red}{W(x; - \dp)})
\end{align}
$$
Note how input image \( I(x) \) and constant Template \( T(x) \) change roles.
Benefits
$\dp = \dots \nabla I(x) \dots \JAC_w(\pp) \dots \HES(\pp)$
| Algorithm | Gradient | Jacobian | Hessian $\HES$ |
|---|---|---|---|
| Fwd. Add. | $\nabla I(x)$ | $\JAC_W(\pp)$ | $\dots \nabla I(x) \JAC_W(\pp) \dots$ |
| Fwd. Cmp. | $\nabla \mathcal{I}(x)$ | $\color{red}{\JAC_W(0)}$ | $\dots \nabla \mathcal{I}(x) \JAC_W(0) \dots$ |
| Inv. Cmp. | $\nabla T(x)$ | $\color{red}{\JAC_W(0)}$ | $\dots \color{red}{\nabla T(x) \JAC_W(0)} \dots$ |
const.
volumetric data: voxel vs pixel
Definition 3D image: $$ I:\mathbb{R}^3 \rightarrow \mathbb{R} $$
$I^{3D}(\mathbf{x} = (x,y,z)^T) = density$
E.g obtained through MRI and CT scans or ToF/SL Cams
So we also use 3D versions of
$$ T^{3D}(x) $$
$$ W^{3D}(x; \pp) $$
"All we have done [is] but replace 3D quantities with 3D versions" [1]
Differences to 2D AAM
Aims to align 3D object (well ... its surface) to a 2D image
Model is warped in $\mathbb{R}^3$ and evaluated in $\mathbb{R}^3$
Therefore just a 2D input image is needed, still advantages of 3D Model
Can we easily apply the efficient ICIA ?
No! But lets try anyway ...
Some Definitions
Residual $R(x)$:
$$ R(x) = I(P(W^{3D}(x; \pp))) - T(P(x)) $$
Where $P: \mathbb{R}^3 \rightarrow \mathbb{R}^2$ is the projection function (e.g. using a projection matrix)
Therefore $I(P(x))$ and $T(P(x))$ are 3D images
We will try to apply the inv. comp. algorithm to these as we did with 3DMM
Can a 2DAAM do the same ?
Definitely !
But ...
unfortunately they can do too much
Reduction from 3D to 2D shape
$$
\mathbf{s^{3D}} = s_0^{3D} + \sum_{i=1}^{m^{3D}} p_i^{3D} s_i^{3D}
$$
Weak Projection (orthonormal; z ~ scaling)
$$ P = \begin{pmatrix} i_x & i_y & i_z \\ j_x & j_y & j_z \end{pmatrix} $$
Expand P
$$
P =
i_x \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}
+ i_y \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}
\dots
+ j_z \begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}
$$
And combine it with s
$$
\begin{align}
s &= P (s_0^{3D} + \sum_{i=1}^{m^{3D}} p_i^{3D} s_i^{3D}) \\
&= s_0 + \sum_{i=1}^{m} p_i s_i
\end{align}
$$
Therefore
$$ m = 6 \times (m^{3D} + 1), $$
2D AAM can do more than is actually possible than by projecting a morphed 3D object.
Idea: Constrain the AAM
$$ F(\mathbf{(p,q); P; p^{3D}}) = K || N(\mathbf{s}(\mathbf{p}); \mathbf{q}) \ - P(\mathbf{s^{3D}(p^{3D})}) ||^2 $$
Minimize
$$ || R(\mathbf{(p,q); \lambda})||^2 + F(\mathbf{(p,q); P; p^{3D}}) $$
Finding a approximate 2DAAM for the 3DMM
Consider the 2DAMM again; the following is enforced:
$$ W(x; 0) = x $$
s(0) and A(0) could be called reference frame
Plain old 2DAMMs just one reference frame
So we need a 2D reference frame.
The projection from 3D space to 2D gives infinite possibilities - we just take a few (e.g. randomly uniform)
Instead of
$$ W(x; \pp) \leftarrow W(x; \pp) \circ W(x; \dp)^{-1} $$
An incremental warp on the reference frame is used:
$$ W(x; \pp) \leftarrow W(x; \pp) \ \circ W(x; \pp^{\dagger}) \ \circ W(x; \pp^{\dagger} + \dp)^{-1} $$
Summary
References
[1] S. Baker and I. Matthews. Lucas-Kanade 20 years on: A unifying framework. International Journal of Computer Vision, 56(3):221–255, 2004. Previously appeared as “Lucas-Kanade 20 years on: A Unifying Framework: Part 5”, CMU Robotics Institute Technical Report CMU-RI-TR-02-16, July 2002.
[2] S. Romdhani and T. Vetter. Eff cient, robust and accurate f tting of a 3D morphable model. In Proceedings of the IEEE International Conference on Computer Vision, pages 59–66, 2003.
[3] J. Xiao, S. Baker, I. Matthews,and T. Kanade. Real-time combined 2D+3D activeappearance models. InProceedingsoftheIEEEConference onComputerVisionandPatternRecognition, 200
[4] I. Matthews and S. Baker. Active Appearance Models revisited. International Journal of Computer Vision, 60(2), 2004.
Thank you for your attention