torch_numopt.algorithms.gauss_newton module#

Gauss-Newton optimization algorithms for least-squares problems.

The Gauss-Newton method approximates the Hessian as JᵀJ, where J is the Jacobian of the residual vector. This module provides three variants: a vanilla version with a fixed learning rate, a line-search version, and a trust-region version. The block-diagonal approximation is available for memory efficiency.

class GaussNewton(params, lr_init=0.001, lr_method=None, solver='solve', damping=None, mu=1, block_hessian=True)[source]#

Bases: NumericalOptimizer

Gauss-Newton optimizer (no line search or trust region).

Uses the Gauss-Newton approximation of the Hessian and solves the linear system to obtain the step direction, then applies a fixed learning rate.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1e-3: Initial learning rate.
lr_methodstr or None, default=None: Learning rate initialization method.
solverstr, default=”solve”: Linear solver for the system.
dampingstr or None, default=None: Damping strategy.
mufloat, default=1: Damping coefficient.
block_hessianbool, default=True: If True, use block-diagonal Gauss-Newton; else full.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GaussNewtonLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo', solver='solve', damping=None, mu=1, block_hessian=True)[source]#

Bases: LineSearchOptimizer

Gauss-Newton optimizer with line search for step-length selection.

This method computes the step direction by solving the Gauss-Newton system (JᵀJ) p = -g, where J is the Jacobian of the residuals and g is the gradient. The resulting direction is then scaled by a step length determined by a line search (backtracking, interpolation, or bisection) that satisfies a chosen condition (Armijo, Wolfe, etc.).

This is particularly useful for non-linear least-squares problems where the residual vector is known and the Hessian can be approximated as JᵀJ.

Parameters:

paramsParams: Parameter tensors to optimize.
lr_initfloat, default=1: Initial guess for the learning rate (starting point for the line search).
lr_methodstr or None, default=None: Learning-rate initialization method (see NumericalOptimizer).
c1float, default=1e-4: Sufficient decrease parameter (Armijo condition).
c2float, default=0.9: Curvature condition parameter (Wolfe conditions).
taufloat, default=0.1: Step-size reduction factor for backtracking.
max_iterint, default=20: Maximum number of line-search iterations.
tolfloat, default=1e-8: Tolerance for stopping (e.g., minimum step size).
line_search_methodstr, default=”backtrack”: Line-search algorithm. Options: “backtrack”, “interpolate”, “bisect”.
line_search_condstr, default=”armijo”: Stopping condition. Options: “greedy”, “armijo”, “wolfe”, “strong-wolfe”, “goldstein”.
solverstr, default=”solve”: Linear solver used to invert the Gauss-Newton system.
dampingstr or None, default=None: Damping strategy (“identity” or “fletcher”) to improve conditioning.
mufloat, default=1: Damping coefficient.
block_hessianbool, default=True: If True, use block-diagonal Gauss-Newton (each parameter group forms its own block) to save memory; otherwise, compute the full matrix.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GaussNewtonTR(params, lr_init=1.0, trust_region_method='exact', solver='solve', damping=None, mu=1, block_hessian=False, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)[source]#

Bases: TrustRegionOptimizer

Gauss-Newton optimizer with a trust-region subproblem solver.

This method builds a quadratic model using the Gauss-Newton approximation of the Hessian (JᵀJ) and solves the trust-region subproblem

minimize m(p) = f + gᵀp + ½ pᵀ(JᵀJ)p subject to ||p|| ≤ Δ.

The step is computed by a trust-region solver (e.g., Cauchy point, dogleg, exact, or Steihaug-Toint) that respects the trust-region radius.

Trust-region Gauss-Newton is robust and often converges faster than the line-search variant, especially in regions where the quadratic model is not accurate.

Parameters:

paramsParams: Parameter tensors to optimize.
radius_initfloat, default=1.0: Initial trust-region radius.
trust_region_methodstr, default=”exact”: Trust-region solver method (see create_trust_region_solver). Common choices: “cauchy”, “dogleg”, “exact”, “steihaug-toint”.
solverstr, default=”solve”: Linear solver used internally (for methods that require solving a system).
dampingstr or None, default=None: Damping strategy to improve conditioning.
mufloat, default=1: Damping coefficient.
block_hessianbool, default=False: If True, use block-diagonal Gauss-Newton; otherwise, compute the full matrix. Note: block-diagonal is often sufficient and saves memory.
accept_tolfloat, default=0.1: Threshold for the ratio rho (actual vs. predicted reduction) above which the step is accepted.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`new_model_radius`(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step