Maximum Principle

This note details methods to solve continuous time optimization problems using the Hamiltonian and the Maximum Principle. These are just a typeset version of the notes I made while studying, using the material from lecture notes and figuring out the details (primarily typo resolution). Apologies if I made any errors of my own.

Setup¶

The continuous time optimization problem we wish to solve is

where $x \in \R^m$ is a state vector and $u \in \R^n$ is a control vector. For any function $f: X \times Y \to \R^k$, $f_x(x,\,y)$ denotes the partial derivative of $f$ with respect to $x$. If $x$ is a vector in $\R^n$, then $f_x(x,\,y)$ is a $n \times k$ matrix. For any function of time, $\dot{f}(t)$ is its time derivative. If $f$ is a column vector, so is $\dot{f}(t)$.

Pontryagin’s Maximum Principle¶

Consider the Lagrangian approach to solving problem (1). For each period $t$, we have a constraint $g(x(t),\,u(t)) - \dot{x}(t) = 0$. We can rewrite this as $e^{-\rho t}(g(x(t),\,u(t)) - \dot{x}(t)) = 0$. Then, the Lagrangian for this problem is defined as

The time-varying Lagrange multipliers, $\l(t)$, are referred to as the co-state variable. Pontryagin’s maximum principle provides an alternative approach to solving this optimization problem. To do so, we must first define the Hamiltonian function, $H: \R^m \times \R^n \times \R^{m} \to \R$, defined as:

The maximum principle characterizes a collection of first-order necessary conditions that must be satisfied by the optimal solution to problem (1).

Caution: differentiating the maximized Hamiltonian¶

Using the first order conditions, we defined $u^*(x,\,\l)$ and the maximized Hamiltonian $H^*(x,\,\l) = H(x,\,u^*(x,\,\l),\,\l)$. We start from $H_u(x,\,u^*(x,\,\l),\,\l) = h_u(x,\,u^*(x,\,\l)) + g_u(x,\,u^*(x,\,\l))\l = 0$. $H_u$ is an $n \times 1$ vector, so $H_{ux}$ and $H_{u\l}$ are $n \times m$ matrices. The (dimension-corrected) formulae for these matrices, as in the lecture notes, are

Unfortunately, $g_u$ was already an $m \times n$ matrix, so the term $g_{uu}$ or $g_{ux}$ is not well-defined as a matrix in the usual sense. The derivative of $g_{u}\l$, which is an $n\times 1$ vector, is still well-defined as an $m \times n$ matrix, so the operation itself is not inappropriate. These problems will not arise when the state or control are scalars, and I’m sure there is some correct way of defining these higher-order derivatives.

Steady state and linearization¶

With the expressions for the derivatives defined as they are here, we can reconcile the steady-state formulae for the system. From the dynamical system governing the evolution of the costate and state variables, the steady state is defined as

From the first equation, we have

Example: Neoclassical growth w/ fixed labor¶

Gives the FOCs

The steady state can then be shown to be characterized by

The dynamical system of $(\l,\,k)$ can be written, using the optimal decision rule, as

with the corresponding phase diagram below.

Then, to linearize the system around the steady state, we use a Taylor expansion and the Envelope theorem (to use derivatives from the maximized Hamiltonian). The non-linear system is

The linearized version is then

Analyzing shocks using phase diagrams¶

Consider a news shock about temporarily low productivity in the future in the neoclassical growth model above. The response, due to the smoothing motive, would be a fall in consumption today, which leads to a corresponding increase in the co-state. We can use the phase diagram then to analyze the path of the economy back to the steady state.

After the initial upward jump in the costate, the costate and capital stock increase, following the evolution equations for the neoclassical growth model. However, at the point when the shock arrives, the production function changes from $f$ to $\tilde{f}$ which has a lower marginal product of capital uniformly. When the productivity change takes effect, capital stock begins to decline as the $\dot{k}$ is determined by the new production function, with the $\dot{k} = 0$ locus being the dotted line in the phase diagram. As the shock dissipates, the economy returns to the saddle path, and returns to the steady state thereafter.

In this model, we can alternatively analyze the impact of the shock by looking at a system of $(c, k)$ instead of $(\l, k)$ due to a one-to-one correspondence between $c$ and $\l$. Using $U'(c) = \l$, write $\dot{\l} = U''(c)\dot{c}$. Then,

Then, consumption falls when $f'(k) < \rho + \d$ and increases otherwise. The same news shock, with analogous effects, can be represented in the $(c, k)$ phase diagram below.

Tobin’s Q¶

Linear adjustment costs¶

The problem at hand is

where $f$ is strictly increasing, strictly concave, and satisfies the Inada conditions. The constraint $0 \leq i \leq f(k)$ is the (linear) adjustment cost: investment can be neither negative nor larger than output. The Hamiltonian is

Since $H$ is linear in the control with slope $H_i = q - 1$, maximizing over $i \in [0,\,f(k)]$ yields a bang-bang policy with three cases,

The remaining first order conditions are

The $\dot{k} = 0$ locus is the horizontal line $q = 1$ (where interior $i = \d k$ is admissible), and the $\dot{q} = 0$ locus is $q = f'(k)/(\rho + \d)$, which is downward sloping by concavity of $f$. The phase diagram is below.

The optimal policy and its recursive solution¶

Consider the stationary policy $i = u(k)$ defined by

where $k^*$ solves $f'(k^*) = \rho + \d$. Along the saddle path of the phase diagram, $q$ and $k$ move together so that

which is exactly the sign pattern that makes $u$ optimal: $H_i < 0$ calls for $i = 0$, $H_i > 0$ for $i = f(k)$, and $H_i = 0$ leaves $i$ interior. The system can therefore be solved recursively. First solve the autonomous law of motion

forward from $k(0)$.^[1] Given the path $k(t)$, the costate solves the linear equation $\dot{q} - (\rho + \d)q = -f'(k)$. Multiplying by the integrating factor $e^{-(\rho+\d)t}$,

Integrating from $t$ to $T$,

Letting $T \to \infty$, the transversality condition $e^{-(\rho+\d)T}q(T) \to 0$ leaves $-e^{-(\rho+\d)t}q(t) = -\int_{t}^{\infty}e^{-(\rho+\d)s}f'(k(s))\,ds$, i.e.

so $q(t)$ is the present value of future marginal products of capital, discounted at $\rho + \d$.

Smooth adjustment costs¶

Now consider the problem

with no restriction on $i$. The quantity $i\,\phi(i/k)$ is the total adjustment cost incurred over and above the purchase price $i$ of the investment good. We assume

the last condition guaranteeing convexity of the adjustment technology. The Hamiltonian is

with first order conditions

where we used $H_k = f'(k) + (i/k)^2\phi'(i/k) - q\d$.

The investment rate¶

Let $x = i/k$. The first order condition $H_i = 0$ reads

Differentiating,

by the convexity assumption, so $\theta$ is strictly increasing and invertible. Writing $x = \gamma(q) \equiv \theta^{-1}(q)$, the inverse function theorem gives

In terms of $x$, the dynamics read

and substituting $x = \gamma(q)$ gives the system in $(q,\,k)$,

An alternative form and its interpretation¶

Multiplying $q = 1 + \phi(x) + x\phi'(x)$ through by $x$,

Substituting into $\dot{q} = (\rho+\d)q - x^2\phi'(x) - f'(k)$,

The bracketed term $f'(k) - x(1 + \phi(x))$ is the marginal benefit of investing: the marginal product of capital net of the per-unit-of-capital cost of investment (purchase plus adjustment). This linear equation in $q$ solves forward to

which one verifies by differentiating in $t$. Thus $q(t)$ is the present value of the marginal benefits of capital, discounted at the effective rate $\rho + \d - x$. The adjustment to the discount rate appears because $\dot{k}/k = x - \d$ is the proportional growth rate of capital.

The $\dot{q} = 0$ locus¶

Setting $\dot{q} = 0$ gives

Provided $\rho + \d - \gamma(q) \neq 0$, the implicit function theorem lets us solve for $q = \eta(k)$.^[2] Totally differentiating in $k$ (with $q = \eta(k)$), the chain rule gives

so that, collecting the $\gamma'(q)$ terms,

Now $2\gamma\phi'(\gamma) + \gamma^2\phi''(\gamma) = \gamma\bp{2\phi'(\gamma) + \gamma\phi''(\gamma)} = \gamma\,\theta'(\gamma)$, and since $\gamma'(q) = 1/\theta'(\gamma(q))$ the product collapses to $\gamma\,\theta'(\gamma)\cdot\tfrac{1}{\theta'(\gamma)} = \gamma$. Hence

Steady state¶

At a steady state $\dot{k} = 0$ requires $\gamma(q^*) = \d$, so

using $\phi(\d) = \phi'(\d)\d = 0$. Then $\dot{q} = 0$ gives $q^*(\rho + \d) - \d^2\phi'(\d) = f'(k^*)$, and since $\d^2\phi'(\d) = \d(\d\phi'(\d)) = 0$,

Evaluating the slope of the $\dot{q} = 0$ locus at the steady state, where $\gamma(\eta(k^*)) = \gamma(1) = \d$,

The phase diagram is below.

Linearization¶

Linearizing the nonlinear system

around $(q,\,k) = (1,\,k^*)$, using $\gamma(1) = \d$ and $\gamma'(1) = 1/\theta'(\d) = 1/(2\phi'(\d) + \d\phi''(\d))$. For the costate, the $q$-derivative of $\gamma(q)^2\phi'(\gamma(q))$ evaluated at $q = 1$ is, exactly as in the $\eta'$ computation,

so that

For the state, $\dot{k} = k(\gamma(q) - \d)$ has $\partial\dot{k}/\partial k|_* = \gamma(1) - \d = 0$ and $\partial\dot{k}/\partial q|_* = k^*\gamma'(1)$, so

or, writing $\hat{q} = q - 1$ and $\hat{k} = k - k^*$,

Eigenvalue and speed of convergence¶

The characteristic equation is

Since $f''(k^*) < 0$ the constant term is negative, so the roots have opposite signs (a saddle). The stable (negative) root is

Defining the curvature of the production function

and using $f'(k^*) = \rho + \d$, so that

we can write

Let $\zeta = 2\phi'(\d) + \d\phi''(\d)$ measure the curvature of the adjustment technology. Then $|\l_-|$ is increasing in $\varepsilon$ and decreasing in $\zeta$: convergence is faster when the production function is more concave (a larger loss from being away from the steady state) and slower when adjustment costs are steeper. Diagonalizing $\Hc = \Phi^{-1}\Theta\Phi$ and setting $z = \Phi[\hat{q},\,\hat{k}]'$ gives $\dot{z} = \Theta z$, so the stable mode decays at rate $|\l_-|$ and convergence is faster the larger $|\l_-|$ is.

Slope of the saddle path¶

Let $q = s(k)$ describe the saddle path and $s' = s'(k^*)$. Since $s' = \dot{q}/\dot{k}$ but both vanish at the steady state, L’H^opital’s rule gives

using $\partial\dot{k}/\partial k = 0$ at the steady state. Hence $s'$ solves the quadratic

and we take the negative root,

Comparative statics of the saddle path¶

If $-f''$ increases (i.e. $\varepsilon \uparrow$), then $s' \downarrow$: the saddle path steepens and, as we saw, capital converges faster. As $f'' \to 0$ ($\varepsilon \to 0$), $s' \to 0$ and convergence becomes extremely slow: a change in the capital stock brings no change in productivity, so $q$ stays near 1 and there is almost no force pulling capital toward $k^*$.

The dependence on $\zeta$ takes more care. To organize the algebra, write

so that $R^2 - \rho^2 = 4(\rho + \d)\varepsilon/\zeta$ and $dR/d\zeta = \tfrac{1}{2R}\cdot\bp{-4(\rho+\d)\varepsilon/\zeta^2} = -2(\rho+\d)\varepsilon/(\zeta^2 R)$. By the product rule,

The first term is precisely $s'/\zeta$. Factoring it out,

and substituting $s' = \zeta(\rho - R)/2k^*$, i.e. $\zeta/s' = 2k^*/(\rho - R)$, the bracketed remainder simplifies using $2(\rho+\d)\varepsilon/\zeta = (R^2 - \rho^2)/2$:

Therefore

since $s' < 0$ and $R > \rho > 0$. The saddle path steepens as $\zeta$ rises.

The two limits are instructive. Writing $\rho - R = (\rho^2 - R^2)/(\rho + R) = -4(\rho + \d)\varepsilon/(\zeta(\rho + R))$,

As $\zeta \to 0$, $R \to \infty$ and $s' \to 0$: with negligible adjustment costs capital jumps essentially instantaneously to $k^*$ while $q$ stays at 1, so the saddle path is horizontal. As $\zeta \to \infty$, $R \to \rho$, and using $(\rho + \d)\varepsilon = -k^*f''(k^*)$,

the saddle path approaches the $\dot{q} = 0$ locus: when adjustment is prohibitively costly the economy crawls along the $\eta$ locus, and convergence (governed by $|\l_-| \to 0$) is very slow even though the path itself is steep.

The value function¶

Since $q(k) = V'(k)$ and $q > 0$, the value function is increasing. Along the saddle path $q(k) = s(k)$ with $s' < 0$, so $V''(k) = q'(k) = s'(k) < 0$ and $V$ is concave. The ratio $V(k)/k$ is the market (book) value of capital per unit installed — the firm’s stock price — i.e. the average value of capital, while $q(k) = V'(k)$ is the marginal value of an additional unit. Thus $V(k)/k$ is average $q$ and $q(k)$ is marginal $q$.

The linear case¶

Finally suppose $f(k) = \pi k$. This arises from an underlying neoclassical production function $F(k,\,l)$ that is homogeneous of degree one, strictly quasi-concave, and satisfies the Inada conditions. Given the capital stock, the firm chooses labor to maximize sales net of the wage bill,

with first order condition $F_l(k,\,l) = w$. Because $F$ is homogeneous of degree one, its partials are homogeneous of degree zero, so $F_l(k,\,l) = F_l(1,\,l/k)$, and the optimal labor-capital ratio $l/k = \lambda(w)$ solves $w = F_l(1,\,\lambda(w))$. Euler’s theorem gives $F(k,\,l) = F_k(k,\,l)k + F_l(k,\,l)l$, and applying degree-zero homogeneity to each partial,

Evaluating at $l/k = \lambda(w)$ and using Euler’s theorem once more,^[3]

Thus $\max_l F(k,\,l) - wl = \pi k$ with $\pi = F(1,\,\lambda(w)) - w\lambda(w)$ independent of $k$: the indirect profit function is linear in capital.