FOCE ELS
FOCE ELS is essentially equivalent to the NONMEM FOCE methodology with interaction and is based on minimizing an extended least squares objective function that represents the FOCE approximation to the negative log of the marginal likelihood as a function of (
, 
, 
). Unlike FOCE L-B, which requires optimization of approximate marginal log likelihoods for a sequence of linearized mixed effects problems, FOCE ELS conceptually involves only a single optimization of a top-level approximate marginal log likelihood function.
However, each evaluation of the objective function in the top-level minimization requires a conditional step in the form of an inner optimization of the joint log likelihood for each subject with respect to 
at the current (, , ) values. The inner optimizations, which are the same calculations that are performed in step 1 of FOCE L-B, are nested within the overall optimization process but many more of them are required than in FOCE L-B. Therefore, FOCE ELS is usually considerably slower than FOCE L-B.
Iterations of FOCE L-B are defined in terms of iterations of the underlying quasi-Newton optimization algorithm as applied to the top-level minimization and generally consist of a gradient evaluation followed by a line search along a direction computed from the gradient and previous iterates. Unlike FOCE L-B, progress is monotonic from iteration to iteration in the absence of numerical difficulties that require internal restarts.