Functional Gradient Descent through Directional Derivatives

Many problems in science and engineering can be understood as a risk minimization procedure over a suitable linear space, such as regression tasks in statistics, or finding solutions to partial differential equations and inverse problems. The natural spaces where the solutions to these problems live are often infinite dimensional, which leads to tractability problems. In response, many solution approaches involve, in one way or another, a reformulation within a parametric, finite-dimensional setting. In boundary value problems for PDEs, for example, finite element methods use a weak formulation of the PDE over a suitable discretization of the domain to arrive at a finite-dimensional linear system. Even more modern ideas, such as Physics Informed Neural Networks (PINNs), involve representing the solution to the PDE through parametric function in the form of a neural network, although the number of parameters can be quite high. In this work, we wish to directly tackle the problem of risk minimization in an infinite dimensional space, and our strategy will be to perform (stochastic) gradient descent within this space. The main difficulty comes from the fact that, in our problems of interest, it is not possible to compute the (stochastic) gradient exactly, so we must employ approximation strategies.