torch_numopt.algorithms.levenberg_marquardt module#

Levenberg-Marquardt optimizer (trust-region variant).

The Levenberg-Marquardt algorithm interpolates between the Gauss-Newton method and gradient descent by adaptively adjusting a damping parameter (mu). It is particularly effective for nonlinear least-squares problems and is implemented here as a trust-region optimizer with Fletcher’s damping strategy.

class LevenbergMarquardt(params, mu=1, mu_max=10000000000.0, damping='fletcher', solver='cholesky', block_hessian=True, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=10, shrink_factor=0.1)[source]#

Bases: TrustRegionOptimizer

Levenberg-Marquardt optimizer (trust-region variant).

This optimizer solves the least-squares problem by adaptively combining Gauss-Newton and gradient descent via a damping parameter (mu). The step is computed by solving (JᵀJ + mu I) p = -g. The damping is adjusted based on the ratio rho.

Parameters:

paramsParams: Parameter tensors.
mufloat, default=1e-2: Initial damping parameter.
mu_maxfloat, default=1e10: Maximum allowed damping.
accept_tolfloat, default=0: Threshold for rho to accept the step.
dampingstr, default=”fletcher”: Damping strategy for the curvature estimator (e.g., “identity” or “fletcher”).
solverstr, default=”cholesky”: Linear solver for the system.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`new_model_radius`(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

new_model_radius(objective, radius, loss, params, grad_params, new_loss, step_dir)[source]#

Update the trust-region radius based on the ratio rho.

The ratio rho measures the agreement between the actual reduction and the predicted reduction. If rho is small, the model is poor, so shrink the radius; if rho is large and the step is at the boundary, expand it.

Parameters:

objectiveObjectiveFunction: Objective function.
radiusfloat: Current trust-region radius.
radius_initfloat: Initial radius (used as upper bound for expansion).
losstorch.Tensor: Loss value at current parameters.
paramsParams: Current parameters.
grad_paramsParams: Gradient at current parameters.
new_losstorch.Tensor: Loss value at proposed new parameters.
step_dirParams: Proposed step direction.

Returns:

tuple (rho, new_radius)

rhofloat: Ratio of actual to predicted reduction.
new_radiusfloat: Updated trust-region radius.

apply_gradients(objective, params, grad_params)[source]#

Update parameters using the current gradient and curvature.

This method computes a step direction, determines a step length (via learning-rate initialization or line-search/trust-region), and applies the update. It also updates stored previous iterates.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters (in-place updated).
grad_paramsParams: Current gradient.

class InexactLevenbergMarquardt(params, mu=1, mu_max=10000000000.0, solver='cg-trunc', block_hessian=False, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=10, shrink_factor=0.1)[source]#

Bases: LevenbergMarquardt

Inexact Levenberg-Marquardt optimizer (using a truncated conjugate-gradient solver).

Parameters:

paramsParams: Parameter tensors.
mufloat, default=1e-2: Initial damping parameter.
mu_decfloat, default=0.1: Factor by which mu is multiplied when the step is successful (reduction).
mu_maxfloat, default=1e10: Maximum allowed damping.
accept_tolfloat, default=0: Threshold for rho to accept the step.
solverstr, default=”cg-trunc”: Linear solver for the system.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`new_model_radius`(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step