| Title: | Alternating Optimization |
| Version: | 1.2.3 |
| Description: | Implementation of an iterative process that optimizes a function by alternately performing restricted optimization over parameter subsets. Instead of solving one joint optimization problem, alternating optimization breaks it into smaller sub-problems. This approach can make optimization feasible when joint optimization is too difficult. |
| URL: | https://loelschlaeger.de/ao/, https://github.com/loelschlaeger/ao/ |
| BugReports: | https://github.com/loelschlaeger/ao/issues |
| License: | GPL-3 |
| Encoding: | UTF-8 |
| Language: | en-US |
| RoxygenNote: | 7.3.3 |
| Imports: | checkmate, cli, future.apply, oeli (≥ 0.7.5), optimizeR (≥ 1.2.1), progressr, R6, stats, utils |
| Suggests: | devtools, ggplot2, knitr, rmarkdown, testthat (≥ 3.0.0) |
| Config/testthat/edition: | 3 |
| VignetteBuilder: | knitr |
| Depends: | R (≥ 4.1.0) |
| NeedsCompilation: | no |
| Packaged: | 2026-05-11 14:44:03 UTC; Lennart Oelschläger |
| Author: | Lennart Oelschläger
|
| Maintainer: | Lennart Oelschläger <oelschlaeger.lennart@gmail.com> |
| Repository: | CRAN |
| Date/Publication: | 2026-05-11 17:10:09 UTC |
ao: Alternating Optimization
Description
Implementation of an iterative process that optimizes a function by alternately performing restricted optimization over parameter subsets. Instead of solving one joint optimization problem, alternating optimization breaks it into smaller sub-problems. This approach can make optimization feasible when joint optimization is too difficult.
Author(s)
Maintainer: Lennart Oelschläger oelschlaeger.lennart@gmail.com (ORCID)
Other contributors:
Siddhartha Chib chib@wustl.edu [contributor]
See Also
Useful links:
Report bugs at https://github.com/loelschlaeger/ao/issues
Examples
# Example 1: Minimization of Himmelblau's function --------------------------
himmelblau <- function(x) (x[1]^2 + x[2] - 11)^2 + (x[1] + x[2]^2 - 7)^2
ao(f = himmelblau, initial = c(0, 0))
# Example 2: Maximization of 2-class Gaussian mixture log-likelihood --------
normal_mixture_loglik_uc = function(mu, logsd, eta, data) {
sd <- exp(logsd)
e <- exp(eta[1])
den <- 1 + e
q1 <- e / den
q2 <- 1 / den
l1 <- log(q1) + dnorm(data, mu[1], sd[1], log = TRUE)
l2 <- log(q2) + dnorm(data, mu[2], sd[2], log = TRUE)
m <- pmax(l1, l2)
sum(m + log(exp(l1 - m) + exp(l2 - m)))
}
set.seed(123)
data <- datasets::faithful$eruptions
fit <- ao(
f = normal_mixture_loglik_uc,
initial = c(mean(data) + c(-1, 1), rep(log(sd(data)), 2), 0),
target = c("mu", "logsd", "eta"),
npar = c(2, 2, 1),
data = data,
partition = "random",
base_optimizer = optimizeR::Optimizer$new("ucminf::ucminf"),
minimize = FALSE,
add_details = FALSE
)
(muhat <- fit$estimate_split$mu)
(sdhat <- exp(fit$estimate_split$logsd))
e <- exp(fit$estimate_split$eta)
den <- 1 + e
(qhat <- c(e / den, 1 / den))
# Example 3: Constrained Optimization in the Setting of Example 2 -----------
# target arguments:
# - class means mu (2, unrestricted)
# - class standard deviations sd (2, must be non-negative)
# - class proportion lambda (only 1 for identification, must be in [0, 1])
normal_mixture_loglik <- function(mu, sd, lambda, data) {
c1 <- lambda * dnorm(data, mu[1], sd[1])
c2 <- (1 - lambda) * dnorm(data, mu[2], sd[2])
sum(log(c1 + c2))
}
set.seed(123)
ao(
f = normal_mixture_loglik,
initial = runif(5),
target = c("mu", "sd", "lambda"),
npar = c(2, 2, 1),
data = datasets::faithful$eruptions,
partition = list("sequential", "random", "none"),
minimize = FALSE,
lower = c(-Inf, -Inf, 0, 0, 0),
upper = c(Inf, Inf, Inf, Inf, 1),
add_details = FALSE
)
Process Object
Description
This object specifies an AO process.
Active bindings
npar[
integer()]
The length(s) of the target argument(s).partition[
character(1)|list()]
Defines the parameter partition. It can be-
"sequential"for treating each parameter separately, -
"random"for a random partition in each iteration, -
"none"for no partition (which is equivalent to joint optimization), or a
listof vectors of parameter indices, specifying a custom partition for the AO process.
-
new_block_probability[
numeric(1)]
Only relevant ifpartition = "random".The probability of creating a new parameter block when creating a random partition.
Values close to 0 result in larger parameter blocks, values close to 1 result in smaller parameter blocks.
minimum_block_number[
integer(1)]
Only relevant ifpartition = "random".The minimum number of blocks in random partitions.
verbose[
logical(1)]
Print tracing details during the AO process?minimize[
logical(1)]
Minimize during the AO process?If
FALSE, maximization is performed.iteration_limit[
integer(1)|Inf]
The maximum number of iterations through the parameter partition before the AO process is terminated.Can also be
Inffor no iteration limit.seconds_limit[
numeric(1)]
The time limit in seconds before the AO process is terminated.Can also be
Inffor no time limit.Note that this stopping criterion is only checked after a sub-problem is solved and not within solving a sub-problem, so the actual process time can exceed this limit.
tolerance_value[
numeric(1)]
A non-negative tolerance value. The AO process terminates if the absolute difference between the current function value and the value fromtolerance_historyiterations earlier is smaller thantolerance_value.Can be
0for no value threshold.tolerance_parameter[
numeric(1)]
A non-negative tolerance value. The AO process terminates if the distance between the current estimate and the estimate fromtolerance_historyiterations earlier is smaller thantolerance_parameter.Can be
0for no parameter threshold.By default, the distance is measured using the Euclidean norm, but another norm can be specified via the
tolerance_parameter_normfield.tolerance_parameter_norm[
function]
The norm that measures the distance between two estimates. If the distance is smaller thantolerance_parameter, the AO process is terminated.It must be of the form
function(x, y)for two vector inputsxandy, and return a singlenumericvalue. By default, the Euclidean normfunction(x, y) sqrt(sum((x - y)^2))is used.tolerance_history[
integer(1)]
The number of iterations to look back to determine whethertolerance_valueortolerance_parameterhas been reached.add_details[
logical(1)]
Add details about the AO process to the output?iteration[
integer(1)]
The current iteration number.block[
integer()]
The currently active parameter block, represented as parameter indices.output[
list(), read-only]
The output of the AO process, which is alistwith the following elements:-
estimateis the parameter vector at termination. -
valueis the function value at termination. -
detailsis adata.framewith full information about the AO process. For each iteration (columniteration) it contains the function value (columnvalue), parameter values (columns starting withpfollowed by the parameter index), the active parameter block (columns starting withbfollowed by the parameter index, where1stands for a parameter contained in the active parameter block and0if not), and computation times in seconds (columnseconds). Only available ifadd_details = TRUE. -
secondsis the overall computation time in seconds. -
stopping_reasonis a message explaining why the AO process terminated.
-
Methods
Public methods
Method new()
Creates a new object of this R6 class.
Usage
Process$new( target = NULL, npar = integer(), partition = "sequential", new_block_probability = 0.3, minimum_block_number = 1, verbose = FALSE, minimize = TRUE, iteration_limit = Inf, seconds_limit = Inf, tolerance_value = 1e-06, tolerance_parameter = 1e-06, tolerance_parameter_norm = function(x, y) sqrt(sum((x - y)^2)), tolerance_history = 1, add_details = TRUE )
Arguments
target[
character()|NULL]
The name(s) of the argument(s) over whichfis optimized.This can only be
numericarguments.If
NULL(default), the first argument offis optimized.npar[
integer()]
The length(s) of the target argument(s).partition[
character(1)|list()]
Defines the parameter partition. It can be-
"sequential"for treating each parameter separately, -
"random"for a random partition in each iteration, -
"none"for no partition (which is equivalent to joint optimization), or a
listof vectors of parameter indices, specifying a custom partition for the AO process.
-
new_block_probability[
numeric(1)]
Only relevant ifpartition = "random".The probability of creating a new parameter block when creating a random partition.
Values close to 0 result in larger parameter blocks, values close to 1 result in smaller parameter blocks.
minimum_block_number[
integer(1)]
Only relevant ifpartition = "random".The minimum number of blocks in random partitions.
verbose[
logical(1)]
Print tracing details during the AO process?minimize[
logical(1)]
Minimize during the AO process?If
FALSE, maximization is performed.iteration_limit[
integer(1)|Inf]
The maximum number of iterations through the parameter partition before the AO process is terminated.Can also be
Inffor no iteration limit.seconds_limit[
numeric(1)]
The time limit in seconds before the AO process is terminated.Can also be
Inffor no time limit.Note that this stopping criterion is only checked after a sub-problem is solved and not within solving a sub-problem, so the actual process time can exceed this limit.
tolerance_value[
numeric(1)]
A non-negative tolerance value. The AO process terminates if the absolute difference between the current function value and the value fromtolerance_historyiterations earlier is smaller thantolerance_value.Can be
0for no value threshold.tolerance_parameter[
numeric(1)]
A non-negative tolerance value. The AO process terminates if the distance between the current estimate and the estimate fromtolerance_historyiterations earlier is smaller thantolerance_parameter.Can be
0for no parameter threshold.By default, the distance is measured using the Euclidean norm, but another norm can be specified via the
tolerance_parameter_normfield.tolerance_parameter_norm[
function]
The norm that measures the distance between two estimates. If the distance is smaller thantolerance_parameter, the AO process is terminated.It must be of the form
function(x, y)for two vector inputsxandy, and return a singlenumericvalue. By default, the Euclidean normfunction(x, y) sqrt(sum((x - y)^2))is used.tolerance_history[
integer(1)]
The number of iterations to look back to determine whethertolerance_valueortolerance_parameterhas been reached.add_details[
logical(1)]
Add details about the AO process to the output?
Method print_status()
Prints a status message.
Usage
Process$print_status(message, message_type = 8, verbose = self$verbose)
Arguments
message[
character(1)]
A status message.message_type[
integer(1)]
The message type, one of the following:-
1forcli::cli_h1() -
2forcli::cli_h2() -
3forcli::cli_h3() -
4forcli::cli_alert_success() -
5forcli::cli_alert_info() -
6forcli::cli_alert_warning() -
7forcli::cli_alert_danger() -
8forcli::cat_line()
-
verbose[
logical(1)]
Print tracing details during the AO process?
Method initialize_details()
Initializes the details part of the output.
Usage
Process$initialize_details(initial_parameter, initial_value)
Arguments
initial_parameter[
numeric()]
The starting parameter values for the AO process.initial_value[
numeric(1)]
The function value at the initial parameters.
Method update_details()
Updates the details part of the output.
Usage
Process$update_details( value, parameter_block, seconds, error, error_message, block = self$block )
Arguments
value[
numeric(1)]
The updated function value.parameter_block[
numeric()]
The updated parameter values for the active parameter block.seconds[
numeric(1)]
The time in seconds for solving the sub-problem.error[
logical(1)]
Did solving the sub-problem result in an error?error_message[
character(1)]
An error message iferror = TRUE.block[
integer()]
The currently active parameter block, represented as parameter indices.
Method get_partition()
Get a parameter partition.
Usage
Process$get_partition()
Method get_details()
Get the details part of the output.
Usage
Process$get_details(
which_iteration = NULL,
which_block = NULL,
which_column = c("iteration", "value", "parameter", "block", "seconds")
)Arguments
which_iteration[
integer()]
Selects the iteration(s).Can also be
NULLto select all iterations.which_block[
character(1)|integer()]
Selects the parameter block in the partition and can be one of-
"first"for the first parameter block, -
"last"for the last parameter block, an
integervector of parameter indices,or
NULLfor all parameter blocks.
-
which_column[
character()]
Selects the columns in thedetailspart of the output and can be one or more of-
"iteration", -
"value", -
"parameter", -
"block", and
"seconds".
-
Method get_value()
Get the function value in different steps of the AO process.
Usage
Process$get_value( which_iteration = NULL, which_block = NULL, keep_iteration_column = FALSE, keep_block_columns = FALSE )
Arguments
which_iteration[
integer()]
Selects the iteration(s).Can also be
NULLto select all iterations.which_block[
character(1)|integer()]
Selects the parameter block in the partition and can be one of-
"first"for the first parameter block, -
"last"for the last parameter block, an
integervector of parameter indices,or
NULLfor all parameter blocks.
-
keep_iteration_column[
logical(1)]
Keep the column containing the information about the iteration in the output?keep_block_columns[
logical(1)]
Keep the column containing the information about the active parameter block in the output?
Method get_value_latest()
Get the function value in the latest step of the AO process.
Usage
Process$get_value_latest()
Method get_value_best()
Get the optimum function value in the AO process.
Usage
Process$get_value_best()
Method get_parameter()
Get the parameter values in different steps of the AO process.
Usage
Process$get_parameter( which_iteration = self$iteration, which_block = NULL, keep_iteration_column = FALSE, keep_block_columns = FALSE )
Arguments
which_iteration[
integer()]
Selects the iteration(s).Can also be
NULLto select all iterations.which_block[
character(1)|integer()]
Selects the parameter block in the partition and can be one of-
"first"for the first parameter block, -
"last"for the last parameter block, an
integervector of parameter indices,or
NULLfor all parameter blocks.
-
keep_iteration_column[
logical(1)]
Keep the column containing the information about the iteration in the output?keep_block_columns[
logical(1)]
Keep the column containing the information about the active parameter block in the output?
Method get_parameter_latest()
Get the parameter value in the latest step of the AO process.
Usage
Process$get_parameter_latest(parameter_type = "full")
Arguments
parameter_type[
character(1)]
Selects the parameter type and can be one of-
"full"(default) to get the full parameter vector, -
"block"to get the parameter values for the current block, i.e., the parameters with the indicesself$block -
"fixed"to get the parameter values which are currently fixed, i.e., all except for those with the indicesself$block
-
Method get_parameter_best()
Get the optimum parameter value in the AO process.
Usage
Process$get_parameter_best(parameter_type = "full")
Arguments
parameter_type[
character(1)]
Selects the parameter type and can be one of-
"full"(default) to get the full parameter vector, -
"block"to get the parameter values for the current block, i.e., the parameters with the indicesself$block -
"fixed"to get the parameter values which are currently fixed, i.e., all except for those with the indicesself$block
-
Method get_seconds()
Get the optimization time in seconds in different steps of the AO process.
Usage
Process$get_seconds( which_iteration = NULL, which_block = NULL, keep_iteration_column = FALSE, keep_block_columns = FALSE )
Arguments
which_iteration[
integer()]
Selects the iteration(s).Can also be
NULLto select all iterations.which_block[
character(1)|integer()]
Selects the parameter block in the partition and can be one of-
"first"for the first parameter block, -
"last"for the last parameter block, an
integervector of parameter indices,or
NULLfor all parameter blocks.
-
keep_iteration_column[
logical(1)]
Keep the column containing the information about the iteration in the output?keep_block_columns[
logical(1)]
Keep the column containing the information about the active parameter block in the output?
Method get_seconds_total()
Get the total optimization time in seconds of the AO process.
Usage
Process$get_seconds_total()
Method check_stopping()
Checks if the AO process can be terminated.
Usage
Process$check_stopping()
Alternating Optimization
Description
Alternating optimization (AO) is an iterative process for optimizing a real-valued function jointly over all its parameters by alternating restricted optimization over parameter partitions.
Usage
ao(
f,
initial,
target = NULL,
npar = NULL,
gradient = NULL,
hessian = NULL,
...,
partition = "sequential",
new_block_probability = 0.3,
minimum_block_number = 1,
minimize = TRUE,
lower = NULL,
upper = NULL,
iteration_limit = Inf,
seconds_limit = Inf,
tolerance_value = 1e-06,
tolerance_parameter = 1e-06,
tolerance_parameter_norm = function(x, y) sqrt(sum((x - y)^2)),
tolerance_history = 1,
base_optimizer = optimizeR::Optimizer$new("stats::optim", method = "L-BFGS-B", control
= list(maxit = 10)),
verbose = FALSE,
hide_warnings = TRUE,
add_details = TRUE
)
Arguments
f |
[ The first argument of If |
initial |
[ This can also be a |
target |
[ This can only be If |
npar |
[ Must be specified if more than one target argument is specified via
the Can be |
gradient |
[ The function signature of Ignored if |
hessian |
[ The function signature of Ignored if |
... |
Additional arguments to be passed to |
partition |
[
This can also be a |
new_block_probability |
[ The probability of creating a new parameter block when creating a random partition. Values close to 0 result in larger parameter blocks, values close to 1 result in smaller parameter blocks. |
minimum_block_number |
[ The minimum number of blocks in random partitions. |
minimize |
[ If |
lower, upper |
[ Ignored if |
iteration_limit |
[ Can also be |
seconds_limit |
[ Can also be Note that this stopping criterion is only checked after a sub-problem is solved and not within solving a sub-problem, so the actual process time can exceed this limit. |
tolerance_value |
[ Can be |
tolerance_parameter |
[ Can be By default, the distance is measured using the Euclidean norm, but another
norm can be specified via the |
tolerance_parameter_norm |
[ It must be of the form |
tolerance_history |
[ |
base_optimizer |
[ By default, the This can also be a |
verbose |
[ Not supported when using multiple processes, see details. |
hide_warnings |
[ |
add_details |
[ |
Details
Multiple processes
AO can suffer from local optima. To increase the likelihood of finding a better optimum, you can specify:
multiple starting parameters
multiple parameter partitions
multiple base optimizers
Use the initial, partition, and/or base_optimizer arguments to provide
a list of possible values for each parameter. Each combination of initial
values, parameter partitions, and base optimizers will create a separate AO
process.
Output value
In the case of multiple processes, the output values refer to the best AO process with respect to function value.
If add_details = TRUE, the following elements are added:
-
estimatesis alistof optimal parameters in each process. -
valuesis alistof optimal function values in each process. -
detailscombines details of the single processes and has an additional columnprocesswith an index for the different processes. -
seconds_eachgives the computation time in seconds for each process. -
stopping_reasonsgives the termination message for each process. -
processesgives details on how the processes were specified.
Parallel computation
By default, processes run sequentially. However, since they are independent,
they can be parallelized. To enable parallel computation, use the
{future} framework. For example, run the
following before the ao() call:
future::plan(future::multisession, workers = 4)
Progress updates
When using multiple processes, setting verbose = TRUE to print tracing
details during AO is not supported. However, you can still track the progress
using the {progressr} framework.
For example, run the following before the ao() call:
progressr::handlers(global = TRUE)
progressr::handlers(
progressr::handler_progress(":percent :eta :message")
)
Value
A list with the following elements:
-
estimateis the parameter vector at termination. -
estimate_splitisestimatesplit bytarget(only when applicable). -
valueis the function value at termination. -
detailsis adata.framewith information about the AO process: For each iteration (columniteration) it contains the function value (columnvalue), parameter values (columns starting withpfollowed by the parameter index), the active parameter block (columns starting withbfollowed by the parameter index, where1stands for a parameter contained in the active parameter block and0if not), and computation times in seconds (columnseconds). Only available ifadd_details = TRUE. -
secondsis the overall computation time in seconds. -
stopping_reasonis a message explaining why the AO process terminated.
In the case of multiple processes, the output changes slightly, see details.
Examples
# Example 1: Minimization of Himmelblau's function --------------------------
himmelblau <- function(x) (x[1]^2 + x[2] - 11)^2 + (x[1] + x[2]^2 - 7)^2
ao(f = himmelblau, initial = c(0, 0))
# Example 2: Maximization of 2-class Gaussian mixture log-likelihood --------
normal_mixture_loglik_uc = function(mu, logsd, eta, data) {
sd <- exp(logsd)
e <- exp(eta[1])
den <- 1 + e
q1 <- e / den
q2 <- 1 / den
l1 <- log(q1) + dnorm(data, mu[1], sd[1], log = TRUE)
l2 <- log(q2) + dnorm(data, mu[2], sd[2], log = TRUE)
m <- pmax(l1, l2)
sum(m + log(exp(l1 - m) + exp(l2 - m)))
}
set.seed(123)
data <- datasets::faithful$eruptions
fit <- ao(
f = normal_mixture_loglik_uc,
initial = c(mean(data) + c(-1, 1), rep(log(sd(data)), 2), 0),
target = c("mu", "logsd", "eta"),
npar = c(2, 2, 1),
data = data,
partition = "random",
base_optimizer = optimizeR::Optimizer$new("ucminf::ucminf"),
minimize = FALSE,
add_details = FALSE
)
(muhat <- fit$estimate_split$mu)
(sdhat <- exp(fit$estimate_split$logsd))
e <- exp(fit$estimate_split$eta)
den <- 1 + e
(qhat <- c(e / den, 1 / den))
# Example 3: Constrained Optimization in the Setting of Example 2 -----------
# target arguments:
# - class means mu (2, unrestricted)
# - class standard deviations sd (2, must be non-negative)
# - class proportion lambda (only 1 for identification, must be in [0, 1])
normal_mixture_loglik <- function(mu, sd, lambda, data) {
c1 <- lambda * dnorm(data, mu[1], sd[1])
c2 <- (1 - lambda) * dnorm(data, mu[2], sd[2])
sum(log(c1 + c2))
}
set.seed(123)
ao(
f = normal_mixture_loglik,
initial = runif(5),
target = c("mu", "sd", "lambda"),
npar = c(2, 2, 1),
data = datasets::faithful$eruptions,
partition = list("sequential", "random", "none"),
minimize = FALSE,
lower = c(-Inf, -Inf, 0, 0, 0),
upper = c(Inf, Inf, Inf, Inf, 1),
add_details = FALSE
)
Generate random partition
Description
This helper function generates a random parameter partition, which is used for the randomized AO procedure.
Usage
generate_random_partition(x, p, min)
Arguments
x |
[ |
p |
[ |
min |
[ |
Value
A list, a random partition of x.
Author(s)
Siddhartha Chib
Merge optimization results
Description
This helper function merges the results of multiple AO processes.
Usage
merge_results(
results,
minimize = TRUE,
add_details = TRUE,
processes = data.frame()
)
Arguments
results |
[ |
minimize |
[ If |
add_details |
[ |
processes |
[ |
Value
A list, see section "Output value" on the ao page.
Split estimate by target
Description
This helper function splits the solution by target parameters (if provided), which is used for the output.
Usage
split_by_target(estimate, target = NULL, npar)
Arguments
estimate |
[ |
target |
[ This can only be If |
npar |
[ Must be specified if more than one target argument is specified via
the Can be |
Value
A (named) list, a partition of estimate according to npar.
Author(s)
Siddhartha Chib