torch_numopt.algorithms package

Submodules

• torch_numopt.algorithms.adahessian module
  • AdaHessianMixin
    • AdaHessianMixin.get_step_direction()
  • AdaHessian
  • AdaHessianLS
• torch_numopt.algorithms.conjugate_gradient module
  • ConjugateGradientMixin
    • ConjugateGradientMixin.get_step_direction()
  • ConjugateGradient
  • ConjugateGradientLS
• torch_numopt.algorithms.gauss_newton module
  • GaussNewton
  • GaussNewtonLS
  • GaussNewtonTR
• torch_numopt.algorithms.gradient_descent module
  • GradientDescent
  • GradientDescentLS
  • GradientDescentTR
  • GradientDescentLipschitz
• torch_numopt.algorithms.lbfgs module
  • LBFGSMixin
    • LBFGSMixin.get_step_direction()
    • LBFGSMixin.apply_gradients()
  • LBFGS
  • LBFGSLS
• torch_numopt.algorithms.levenberg_marquardt module
  • LevenbergMarquardt
    • LevenbergMarquardt.new_model_radius()
    • LevenbergMarquardt.apply_gradients()
  • InexactLevenbergMarquardt
• torch_numopt.algorithms.newton module
  • Newton
  • NewtonLS
  • NewtonTR
  • NewtonCG
  • NewtonCGLS
  • NewtonCGTR

Module contents

Concrete optimizer implementations.

This package contains ready-to-use optimizers that inherit from the base classes in numerical_optimizer. They combine curvature estimators with step- selection strategies (line search or trust region) to provide complete optimization algorithms.

Available optimizers: - GradientDescent (and variants with line search / trust region) - ConjugateGradient (with line search) - Newton (exact Hessian, with line search, trust region, or CG) - GaussNewton (Gauss-Newton approximation) - LevenbergMarquardt (trust-region with adaptive damping) - LBFGS (limited-memory BFGS) - AdaHessian (diagonal Hessian with momentum)

class GradientDescent(params, lr_init=0.001, lr_method=None)

Bases: NumericalOptimizer

Vanilla gradient descent with a fixed or adaptively initialized learning rate.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1e-3

Initial learning rate.

lr_methodstr, optional

Learning-rate initialization method (see NumericalOptimizer).

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GradientDescentLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo')

Bases: LineSearchOptimizer

Gradient descent with a line search to determine the step length.

This optimizer computes the steepest descent direction and then performs a line search (backtracking, interpolation, or bisection) to find a step size that satisfies the chosen condition (Armijo, Wolfe, Goldstein, etc.).

Parameters:

paramsParams

Parameter tensors to optimize.

lr_initfloat, default=1

Initial guess for the learning rate (used as the starting point for the line search).

lr_methodstr or None, default=None

Method for initializing the learning rate before the line search (see NumericalOptimizer for options). If None, uses lr_init directly.

c1float, default=1e-4

Sufficient decrease parameter (Armijo condition).

c2float, default=0.9

Curvature condition parameter (Wolfe conditions).

taufloat, default=0.1

Step-size reduction factor for backtracking.

max_iterint, default=20

Maximum number of line-search iterations.

tolfloat, default=1e-8

Tolerance for stopping (e.g., minimum step size).

line_search_methodstr, default=”backtrack”

Line-search algorithm. Options: “backtrack”, “interpolate”, “bisect”.

line_search_condstr, default=”armijo”

Stopping condition. Options: “greedy”, “armijo”, “wolfe”, “strong-wolfe”, “goldstein”.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GradientDescentTR(params, lr_init=1.0, trust_region_method='cauchy', *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)

Bases: TrustRegionOptimizer

Gradient descent with a trust-region (Cauchy point) step.

This optimizer builds a quadratic model using the gradient (identity curvature) and solves the trust-region subproblem using the Cauchy point, which is the minimizer of the model along the steepest descent direction within the trust region.

Parameters:

paramsParams

Parameter tensors to optimize.

radius_initfloat, default=1.0

Initial trust-region radius.

trust_region_methodstr, default=”cauchy”

Trust-region solver method. For gradient descent, only “cauchy” is recommended, but other methods (e.g., “dogleg”, “exact”) can be used if a curvature estimator is provided (here it uses identity, so they degenerate to Cauchy point anyway).

accept_tolfloat, default=0.1

Threshold for the ratio rho (actual vs. predicted reduction) above which the step is accepted.

curvature_estimatorCurvatureEstimator, optional

Not used directly (identity is forced), kept for compatibility with the base class.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
new_model_radius(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GradientDescentLipschitz(params, lr_init=0.001)

Bases: NumericalOptimizer

Gradient descent with a learning rate estimated from the Lipschitz constant.

This optimizer uses a heuristic (the “lipschitz” method) to estimate the learning rate at each step based on the change in gradient and parameters from the previous iteration. It avoids the need for manual tuning and often converges faster than fixed-rate gradient descent.

Parameters:

paramsParams

Parameter tensors to optimize.

lr_initfloat, default=1e-3

Initial learning rate (used only for the first iteration).

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class ConjugateGradient(params, lr_init=1.0, lr_method='lipschitz', cg_method='PRP+')

Bases: ConjugateGradientMixin, NumericalOptimizer

Non-linear conjugate gradient optimizer with fixed learning rate.

Uses the Fletcher-Reeves, Polak-Ribière, etc. formulas to compute the search direction without explicit curvature.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1.0

Initial learning rate.

lr_methodstr or None, default=”lipschitz”

Learning rate initialization method.

cg_methodstr, default=”PRP+”

Conjugate gradient formula: “FR”, “PR”, “PRP+”, “HS”, “DY”.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
get_step_direction
profile_hook_step

class ConjugateGradientLS(params, lr_init=1.0, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo', cg_method='PRP+')

Bases: ConjugateGradientMixin, LineSearchOptimizer

Non-linear conjugate gradient with line search.

Combines conjugate gradient direction with a line-search to determine the step length.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1.0

Initial learning rate.

lr_methodstr or None, default=None

Learning rate initialization method.

c1float, default=1e-4

Sufficient decrease parameter (Armijo).

c2float, default=0.9

Curvature condition parameter (Wolfe).

taufloat, default=0.1

Step reduction factor for backtracking.

max_iterint, default=20

Maximum iterations for line search.

tolfloat, default=1e-8

Tolerance (e.g., minimum step).

line_search_methodstr, default=”backtrack”

Line-search method (see create_line_search_solver).

line_search_condstr, default=”armijo”

Line-search stopping condition.

cg_methodstr, default=”PRP+”

Conjugate gradient formula.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
get_step_direction
profile_hook_step

class Newton(params, lr_init=1, lr_method=None, damping=None, mu=1, solver='solve', block_hessian=True)

Bases: NumericalOptimizer

Newton method with exact Hessian (full or block) and fixed learning rate.

Uses the exact Hessian (or block-diagonal) to compute the Newton step.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning rate initialization method.

dampingstr or None, default=None

Damping strategy.

mufloat, default=1

Damping coefficient.

solverstr, default=”solve”

Linear solver for the system.

block_hessianbool, default=True

If True, use block-diagonal Hessian.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, damping=None, mu=1, line_search_method='backtrack', line_search_cond='armijo', solver='solve', block_hessian=True)

Bases: LineSearchOptimizer

Newton method with exact Hessian and line search.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning rate initialization method.

c1, c2, tau, max_iter, tolline search parameters.

dampingstr or None, default=None

Damping strategy.

mufloat, default=1

Damping coefficient.

line_search_methodstr, default=”backtrack”

Line-search method.

line_search_condstr, default=”armijo”

Line-search condition.

solverstr, default=”solve”

Linear solver.

block_hessianbool, default=True

If True, use block-diagonal Hessian.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonTR(params, lr_init=1.0, trust_region_method='exact', damping=None, mu=1, solver='solve', block_hessian=False, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)

Bases: TrustRegionOptimizer

Newton method with exact Hessian and trust region.

Uses a trust-region solver (e.g., exact or Steihaug-Toint) to compute the step.

Parameters:

paramsParams

Parameter tensors.

radius_initfloat, default=1.0

Initial trust-region radius.

trust_region_methodstr, default=”exact”

Trust-region solver method.

dampingstr or None, default=None

Damping strategy.

mufloat, default=1

Damping coefficient.

solverstr, default=”solve”

Linear solver for the system.

block_hessianbool, default=False

If True, use block-diagonal Hessian.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
new_model_radius(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonCG(params, lr_init=1, lr_method=None, damping=None, mu=1, solver='cg-trunc')

Bases: NumericalOptimizer

Newton-CG method (inexact Newton) using conjugate gradient to solve the linear system.

Uses exact Hessian but solves the system iteratively with CG.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning rate initialization method.

dampingstr or None, default=None

Damping strategy.

mufloat, default=1

Damping coefficient.

solverstr, default=”cg-trunc”

Iterative solver (must be in iterative_solver_set).

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonCGLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, damping=None, mu=1, line_search_method='backtrack', line_search_cond='armijo', solver='cg-trunc')

Bases: LineSearchOptimizer

Newton-CG with line search.

Combines the iterative CG solution of the Newton system with a line search to determine the step length.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning-rate initialization method.

c1, c2, tau, max_iter, tolline-search parameters.

dampingstr or None, default=None

Damping strategy.

mufloat, default=1

Damping coefficient.

line_search_methodstr, default=”backtrack”

Line-search method.

line_search_condstr, default=”armijo”

Stopping condition.

solverstr, default=”cg-trunc”

Iterative solver.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonCGTR(params, lr_init=1.0, damping=None, mu=1, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)

Bases: TrustRegionOptimizer

Newton-CG with trust region (Steihaug-Toint).

Uses the Steihaug-Toint CG-trust-region method, which solves the trust-region subproblem iteratively with a CG approach that automatically handles negative curvature and the trust-region boundary.

Parameters:

paramsParams

Parameter tensors.

radius_initfloat, default=1.0

Initial trust-region radius.

dampingstr or None, default=None

Damping strategy.

mufloat, default=1

Damping coefficient.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
new_model_radius(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GaussNewton(params, lr_init=0.001, lr_method=None, solver='solve', damping=None, mu=1, block_hessian=True)

Bases: NumericalOptimizer

Gauss-Newton optimizer (no line search or trust region).

Uses the Gauss-Newton approximation of the Hessian and solves the linear system to obtain the step direction, then applies a fixed learning rate.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1e-3

Initial learning rate.

lr_methodstr or None, default=None

Learning rate initialization method.

solverstr, default=”solve”

Linear solver for the system.

dampingstr or None, default=None

Damping strategy.

mufloat, default=1

Damping coefficient.

block_hessianbool, default=True

If True, use block-diagonal Gauss-Newton; else full.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GaussNewtonLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo', solver='solve', damping=None, mu=1, block_hessian=True)

Bases: LineSearchOptimizer

Gauss-Newton optimizer with line search for step-length selection.

This method computes the step direction by solving the Gauss-Newton system (JᵀJ) p = -g, where J is the Jacobian of the residuals and g is the gradient. The resulting direction is then scaled by a step length determined by a line search (backtracking, interpolation, or bisection) that satisfies a chosen condition (Armijo, Wolfe, etc.).

This is particularly useful for non-linear least-squares problems where the residual vector is known and the Hessian can be approximated as JᵀJ.

Parameters:

paramsParams

Parameter tensors to optimize.

lr_initfloat, default=1

Initial guess for the learning rate (starting point for the line search).

lr_methodstr or None, default=None

Learning-rate initialization method (see NumericalOptimizer).

c1float, default=1e-4

Sufficient decrease parameter (Armijo condition).

c2float, default=0.9

Curvature condition parameter (Wolfe conditions).

taufloat, default=0.1

Step-size reduction factor for backtracking.

max_iterint, default=20

Maximum number of line-search iterations.

tolfloat, default=1e-8

Tolerance for stopping (e.g., minimum step size).

line_search_methodstr, default=”backtrack”

Line-search algorithm. Options: “backtrack”, “interpolate”, “bisect”.

line_search_condstr, default=”armijo”

Stopping condition. Options: “greedy”, “armijo”, “wolfe”, “strong-wolfe”, “goldstein”.

solverstr, default=”solve”

Linear solver used to invert the Gauss-Newton system.

dampingstr or None, default=None

Damping strategy (“identity” or “fletcher”) to improve conditioning.

mufloat, default=1

Damping coefficient.

block_hessianbool, default=True

If True, use block-diagonal Gauss-Newton (each parameter group forms its own block) to save memory; otherwise, compute the full matrix.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GaussNewtonTR(params, lr_init=1.0, trust_region_method='exact', solver='solve', damping=None, mu=1, block_hessian=False, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)

Bases: TrustRegionOptimizer

Gauss-Newton optimizer with a trust-region subproblem solver.

This method builds a quadratic model using the Gauss-Newton approximation of the Hessian (JᵀJ) and solves the trust-region subproblem

minimize m(p) = f + gᵀp + ½ pᵀ(JᵀJ)p subject to ||p|| ≤ Δ.

The step is computed by a trust-region solver (e.g., Cauchy point, dogleg, exact, or Steihaug-Toint) that respects the trust-region radius.

Trust-region Gauss-Newton is robust and often converges faster than the line-search variant, especially in regions where the quadratic model is not accurate.

Parameters:

paramsParams

Parameter tensors to optimize.

radius_initfloat, default=1.0

Initial trust-region radius.

trust_region_methodstr, default=”exact”

Trust-region solver method (see create_trust_region_solver). Common choices: “cauchy”, “dogleg”, “exact”, “steihaug-toint”.

solverstr, default=”solve”

Linear solver used internally (for methods that require solving a system).

dampingstr or None, default=None

Damping strategy to improve conditioning.

mufloat, default=1

Damping coefficient.

block_hessianbool, default=False

If True, use block-diagonal Gauss-Newton; otherwise, compute the full matrix. Note: block-diagonal is often sufficient and saves memory.

accept_tolfloat, default=0.1

Threshold for the ratio rho (actual vs. predicted reduction) above which the step is accepted.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
new_model_radius(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class LevenbergMarquardt(params, mu=1, mu_max=10000000000.0, damping='fletcher', solver='cholesky', block_hessian=True, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=10, shrink_factor=0.1)

Bases: TrustRegionOptimizer

Levenberg-Marquardt optimizer (trust-region variant).

This optimizer solves the least-squares problem by adaptively combining Gauss-Newton and gradient descent via a damping parameter (mu). The step is computed by solving (JᵀJ + mu I) p = -g. The damping is adjusted based on the ratio rho.

Parameters:

paramsParams

Parameter tensors.

mufloat, default=1e-2

Initial damping parameter.

mu_maxfloat, default=1e10

Maximum allowed damping.

accept_tolfloat, default=0

Threshold for rho to accept the step.

dampingstr, default=”fletcher”

Damping strategy for the curvature estimator (e.g., “identity” or “fletcher”).

solverstr, default=”cholesky”

Linear solver for the system.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
new_model_radius(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

new_model_radius(objective, radius, loss, params, grad_params, new_loss, step_dir)

Update the trust-region radius based on the ratio rho.

The ratio rho measures the agreement between the actual reduction and the predicted reduction. If rho is small, the model is poor, so shrink the radius; if rho is large and the step is at the boundary, expand it.

Parameters:

objectiveObjectiveFunction

Objective function.

radiusfloat

Current trust-region radius.

radius_initfloat

Initial radius (used as upper bound for expansion).

losstorch.Tensor

Loss value at current parameters.

paramsParams

Current parameters.

grad_paramsParams

Gradient at current parameters.

new_losstorch.Tensor

Loss value at proposed new parameters.

step_dirParams

Proposed step direction.

Returns:

tuple (rho, new_radius)

rhofloat

Ratio of actual to predicted reduction.

new_radiusfloat

Updated trust-region radius.

apply_gradients(objective, params, grad_params)

Update parameters using the current gradient and curvature.

This method computes a step direction, determines a step length (via learning-rate initialization or line-search/trust-region), and applies the update. It also updates stored previous iterates.

Parameters:

objectiveObjectiveFunction

Objective function.

paramsParams

Current parameters (in-place updated).

grad_paramsParams

Current gradient.

class InexactLevenbergMarquardt(params, mu=1, mu_max=10000000000.0, solver='cg-trunc', block_hessian=False, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=10, shrink_factor=0.1)

Bases: LevenbergMarquardt

Inexact Levenberg-Marquardt optimizer (using a truncated conjugate-gradient solver).

This optimizer solves the least-squares problem by adaptively combining Gauss-Newton and gradient descent via a damping parameter (mu). The step is computed by solving (JᵀJ + mu I) p = -g with inexact methods. The damping is adjusted based on the ratio rho.

Parameters:

paramsParams

Parameter tensors.

mufloat, default=1e-2

Initial damping parameter.

mu_decfloat, default=0.1

Factor by which mu is multiplied when the step is successful (reduction).

mu_maxfloat, default=1e10

Maximum allowed damping.

accept_tolfloat, default=0

Threshold for rho to accept the step.

solverstr, default=”cg-trunc”

Linear solver for the system.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
new_model_radius(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class LBFGS(params, lr_init=1.0, lr_method=None, memory_size=10)

Bases: LBFGSMixin, NumericalOptimizer

Limited-memory BFGS optimizer with fixed learning rate.

Maintains a history of past updates (s, y) to approximate the inverse Hessian.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1.0

Initial learning rate.

lr_methodstr or None, default=None

Learning rate initialization method.

memory_sizeint, default=10

Number of past updates to store.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
apply_gradients
get_step_direction
profile_hook_step

class LBFGSLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, memory_size=10, line_search_method='interpolate', line_search_cond='wolfe')

Bases: LBFGSMixin, LineSearchOptimizer

L-BFGS with line search.

After computing the L-BFGS direction, a line search is performed to find an appropriate step length. This is the recommended way to use L-BFGS.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning-rate initialization method.

c1, c2, tau, max_iter, tolline-search parameters.

memory_sizeint, default=10

Number of stored (s, y) pairs.

line_search_methodstr, default=”interpolate”

Line-search method (interpolate is often good for L-BFGS).

line_search_condstr, default=”wolfe”

Condition (Wolfe conditions are typical for L-BFGS).

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
apply_gradients
get_step_direction
profile_hook_step

class AdaHessian(params, lr_init=0.01, lr_method=None, beta1=0.9, beta2=0.999, k=1, eps=1e-08, n_samples=10, skip_iters=0)

Bases: AdaHessianMixin, NumericalOptimizer

AdaHessian optimizer (diagonal Hessian with momentum).

Uses Hutchinson diagonal Hessian approximation and momentum for both gradient and Hessian diagonal, similar to Adam but using second-order information.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning rate initialization method.

beta1float, default=0.9

Exponential decay rate for the first moment estimate (gradient).

beta2float, default=0.999

Exponential decay rate for the second moment estimate (Hessian diagonal).

kfloat, default=1

Exponent for the Hessian diagonal in the step calculation (0.5 for AdaHessian).

epsfloat, default=1e-4

Small constant for numerical stability.

n_samplesint, default=5

Number of Hutchinson samples for diagonal estimation.

skip_itersint, default=0

Number of times in which we reuse the previously computed diagonal approximation (No skipping by default).

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
get_step_direction
profile_hook_step

class AdaHessianLS(params, lr_init=1, lr_method=None, beta1=0.9, beta2=0.999, k=1, eps=1e-08, n_samples=10, skip_iters=0, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo')

Bases: AdaHessianMixin, LineSearchOptimizer

AdaHessian with line search.

Same as AdaHessian, but instead of a fixed learning rate it performs a line search to determine the step length.

Works well in practice, but theoretically it’s not well supported.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning-rate initialization method.

beta1, beta2, k, eps, n_samples, skip_iterssame as in AdaHessian.

c1, c2, tau, max_iter, tolline-search parameters.

line_search_methodstr, default=”backtrack”

Line-search method.

line_search_condstr, default=”armijo”

Line-search condition.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
get_step_direction
profile_hook_step

class DiagonalNewton(params, lr_init=0.001, lr_method=None, n_samples=10, skip_iters=0)

Bases: NumericalOptimizer

Diagonal Newton approximation (using Hutchinson’s method).

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning-rate initialization method.

n_samples, skip_iters: Hutchinson approximation parameters.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class DiagonalNewtonLS(params, lr_init=1, lr_method=None, n_samples=10, skip_iters=0, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo')

Bases: LineSearchOptimizer

Diagonal Newton approximation (using Hutchinson’s method) with line search.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1

Initial learning rate.

lr_methodstr or None, default=None

Learning-rate initialization method.

n_samples, skip_iters: Hutchinson approximation parameters.

c1, c2, tau, max_iter, tolline-search parameters.

line_search_methodstr, default=”backtrack”

Line-search method.

line_search_condstr, default=”armijo”

Line-search condition.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step