20.1 The Nature of Optimal Control
In static optimization, the task is to find a single value for each choice variable, such that a stated objective function will be maximized or minimized. This is devoid of the time dimension. However, in dynamic optimization problems we will always have in mind a planning period, from initial \(t = 0\) to a terminal \(t = T\), trying to find the best course of action to take during that entire period. Thus the solution is not just a variable, but a complete time path
For example, suppose we want to maximize profit over a time period. At any point of time \(t\), we have to chose the value of some control variable, \(u(t)\), which will then affect the value of some state variable, \(y(t)\), via the equation of motion. Finally, \(y(t)\) will determine the profit \(\pi (t)\). Since the objective is to maximize profit over the entire period, the objective function should take the form of a definite integral of \(\pi\) from \(t = 0\) to \(t = T\). The simplest problem of optimal control looks like this:
Maximize \(\int ^T _0 F(t, y, u) dt\)
subject to \(\frac{fy}{dt} \equiv y' = f(t, y, u)\)
and \(y(0) = A; y(T) = \) free
\(u(t) \in U\); for all \( t \in [0, T]\)
The first line states the objective function, which is an integral whose integrand \(F(t, y, u)\) stipulates how the choice of the control variable \(u \) at time \(t\), along with the resulting \(y\) at time \(t\), determines our maximization. The second line is the equation of motion for the state variable \(y\). This equation provides the mechanism whereby our choice of control variable \(u \) can be translated into a specific pattern of movement of the state variable \(y\). This normally is done by the first-order differential equation. In the third line, the initial value of \(y\) at \(t = 0\) is a constant \(A\), but the terminal state \(y(T)\) is left unrestricted. Finally, the fourth line indicates that the permissible choices of \(u \) are limited to a control region \(U\).
Illustration with a Simple Macroeconomic Model
Consider an economy producing output using capital and a fixed amount of labor according to the production function: \(Y = Y(K, L)\). Since output is used for consumption or investment and we ignore depreciation, investment will be the change in capital stock over time: \(I = Y - C = Y(K, L) - C = \frac{dk}{dt}\). If our objective is to maximize some form of social utility over a fixed planning period, the problem becomes:
Maximize \(\int ^T _0 U(C) dt\)
subject to \(\frac{dK}{dt} = Y( K, L) - C\)
and \(K(0) = K_0\) \(K(T) = K_T\)
The problem is to choose the optimal control path \(C(t)\) such that its impact on output \(Y\) and capital \(K\), and the repercussions therefrom upon \(C\) itself will together maximize the aggregate utility over the planning period.
Pontryagin's Maximum Principle
The key to optimal control theory is a first-order necessary condition known as the maximum principle. It is akin to the Lagrangian function and multiplier variable, but for optimal control problems are known as the Hamiltonian function and costate variable.
The Hamiltonian
In our maximization equation, there are three variables: time (\(t\)), the state variable (\(y\)), and the control variable (\(u\)). We now introduce the costate variable, denoted by \(\lambda(t)\). Like the Lagrange multiplier, the costate variable measures the shadow price of the state variable. Introduced via the Hamiltonian function, it becomes:
\(H(t, y, u, \lambda) \equiv F(t, y, u) + \lambda(t) f(t, y, u)\)
The Maximum Principle
The maximum principle requires us to choose \(u\) so as to maximize the Hamiltonian \(H\) at every point of time. Since there are other variables, the maximum principle also stipulates how \(y\) and \(\lambda\) should change over time, via the equation of motion for the state variable \(y\) as well as an equation of motion for the costate variable \(\lambda\) . The state equation always comes as part of the problem statement itself, so it is:
\(y' = f(t, y, u) \Longrightarrow \frac{\partial H}{\partial \lambda}\)
Since \(\lambda\) does not appear in the problem statement its equation of motion enters into the picture purely as an optimization condition. The costate equation is:
\(\lambda ' (\equiv \frac{d \lambda}{dt}) = -\frac{\partial H}{\partial y}\)
This presents a system of two differential equations, thus we need two boundary conditions to definitize the two arbitrary constants that will arise in the process of solution. If the terminal state is not fixed, then a transversality condition must be included to fill the gap.
Example 1:
Find the shortest path from given point A to a given straight line. If the lines are curved, their length will be the aggregate of small path segments, which can be considered as the hypotenuse of triangles formed by the small movements of dt and dy (slope). Thus we can use the Pythagoras: \(dh^2 = dt^2 + dy^2\). Dividing both sides by \(dt^2\) and taking square root: \(\frac{dh}{dt} = [1 + (\frac{dy}{dt})^2]^{1/2} = [1 + (y')^2]^{1/2}\). The total length of the path can then be found by integrating with respect to \(t\). By letting \(y' = u\) be the control variable, we have: \(\frac{dh}{dt} = (1 + u^2)^{1/2}\). Thus, the shortest-path problem is:
Maximize \(\int ^T _0 -(1 + u^2)^{1/2}\)
subject to \(y' = u\)
and \(y(0) = A\) \(y(T) =\) free
The Hamiltonian for the problem is:
\(H = -(1 + u^2)^{1/2} + \lambda u\)
Since \(H\) is differentiable in \(u\), and \(u\) is unrestricted, the following first-order condition can be used to maximize \(H\):
\(\frac{\partial H}{\partial u} = -\frac{1}{2}(1 + u^2)^{-1/2}(2u) + \lambda = 0\)
or, \(u(t) = \lambda(1 - \lambda ^2) ^{-1/2}\)
Demonstrating that the solution does maximize the Hamiltonian. Now, we have to find a solution to the costate variable:
\(\lambda ' = -\frac{\partial H}{\partial y} = 0\)
Since \(H\) is independent of \(y\), thus \(\lambda\) is a constant. To definitize this constant, make use of the transversality condition: \(\lambda (t) = 0\). Finally, using the equation of motion for the state variable, \(y' = u = 0\). WE can conclude that the shortest path is found to be a straight line with zero slope.
20.2 Alternative Terminal Conditions
What happens to the maximum principle when the terminal condition is different from a constant beginning and free terminal value? Maximization requires:
\(H(t, y, u*, \lambda) \geq H(t, y, u, \lambda)\)
\(y' = \frac{\partial H}{\partial \lambda}\)
\(\lambda ' = -\frac{\partial H}{\partial y}\)
with the transversality condition: \(\lambda(T) = 0\)
With alternative terminal conditions, we just modify the transversality condition. A practical approach is to use the Kuhn-Tucker conditions:
\(\lambda(T) \geq 0 ; y_T \geq y_{min} ; (y_T - y_{min}) \lambda (T) = 0\)
20.3 Autonomous Problems
Problems in which \(t\) is absent form the objective function and state equation are called autonomous problems. In such problems, since the Hamiltonian does not contain \(t\), we can use phase-diagram analysis. Or, if it enters as part of the discount factor, \(e^{-rt}\), we can have an objective function:
\(\int^T _0 G(y, u) e^{-rt} dt\)
This can also allow a current-value Hamiltonian and then current-value Lagrange multiplier defined as:
\(H_c \equiv He^{rt} = G(y, u) + \mu f(y, u)\)
\(\mu \equiv \lambda e^{rt}\)
20.4 Economic Applications
Lifetime Utility Maximization
Suppose a consumer has the utility function \(U(C(t))\), where \(C(t)\) is consumption at time \(t\). It's concave, thus having the following properties: \(U' > 0 ; U'' < 0\).
The consumer is also endowed with an initial stock of wealth, or capital \(K_0\), with income stream derived from the stock of capital according to: \(Y = rK\) with \(r\) as the interest rate. Any income not consumed is added to the capital stock as investment. Thus: \(K' \equiv I = Y - C = rK - C\).
The consumer's lifetime utility maximization problem is to
Maximize \(\int ^T _0 U(C(t))e^{\delta t} dt\)
subject to \(K' = rK(t) - C(t)\)
and \(K(0) = K_0 ; K(T) \geq 0\)
Where the consumer's personal rate of time preference is \(\delta\). The Hamiltonian is:
\(H = U(C(t)) e^{- \delta t} + \lambda (t) [rK(t) - C(t)]\)
Where \(C\) is the control variable and \(K\) is that state variable. Since \(U(C(t))\)is concave we know that the maximization of H can be achieved by simply setting the derivative of the Hamiltonian with respect to consumption at zero. Thus we have:
\(\frac{\partial H}{\partial C} = U' (C) e^{- \delta t} - \lambda = 0\)
\(K' = rK(t) - C(t)\)
\(\lambda ' = -\frac{\partial H}{\partial K} = -r \lambda\)
Stating that the discounted marginal utility should be equated to the present shadow price of an additional unit of capital.
Class Notes
Chad Jones's dissertation: gave birth to semi-endogenous growth theory
Romer said new ideas were born this way: \(\dot A = zA_t R_t \Longrightarrow \frac{\dot A}{A} = zR_t\) (Growth rate of tech. progress)
\(g_A = zR_t\): linear relationship between # of researchers and nation's growth rate (scale effect)
Chad Jones's First Big Contribution: \(\dot A = zA_t^\alpha R_t^\beta\)
1. Assume there's some state growth rate, \(g_A\)
2. Divide both sides by A: \(g_A = \frac{\dot A}{A} = zA^{\alpha -1}R^\beta\)
3. Assume \(\alpha > 0\) "shoulder of giants" effect. \(\alpha < 0\) would mean fishing-out, "Great Stagnation"
4. Take logs and derivatives (turn into growth rates):
\(\ln g_A = \ln z + (\alpha - 1) \ln A + \beta \ln R\)
\(0 = 0 + (\alpha - 1) \frac{\dot A}{A} + \beta \frac{\dot R}{R}\)
\((1 - \alpha)g_A = \beta g_R\)
\(g_A = (\frac{\beta}{1 - \alpha}) g_R\)
\(\therefore\) growth rate of technology, which is growth rate of economy, is proportional to growth rate of researchers
*Interesting note: research and idea growth only in 5 countries (USA, UK, Germany, Japan, France/China)
**Other interesting note: Zipf's law: In probability, assertion that the frequencies f of certain events are inversely proportional to their rank r. In cities this means that the second city will have half the population as the first, and on down with the 100th biggest city have 100th the population. Also works for endowments, with the size of endowments following this pattern.