torch_numopt.algorithms.newton module#

Newton-type methods using exact Hessian (full or block).

These optimizers compute the exact second-order derivatives and use them to form a quadratic model. They offer fast local convergence but may be expensive for large models.

class Newton(params, lr_init=1, lr_method=None, damping=None, mu=1, solver='solve', block_hessian=True)[source]#

Bases: NumericalOptimizer

Newton method with exact Hessian (full or block) and fixed learning rate.

Uses the exact Hessian (or block-diagonal) to compute the Newton step.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1: Initial learning rate.
lr_methodstr or None, default=None: Learning rate initialization method.
dampingstr or None, default=None: Damping strategy.
mufloat, default=1: Damping coefficient.
solverstr, default=”solve”: Linear solver for the system.
block_hessianbool, default=True: If True, use block-diagonal Hessian.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, damping=None, mu=1, line_search_method='backtrack', line_search_cond='armijo', solver='solve', block_hessian=True)[source]#

Bases: LineSearchOptimizer

Newton method with exact Hessian and line search.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1: Initial learning rate.
lr_methodstr or None, default=None: Learning rate initialization method.
c1, c2, tau, max_iter, tolline search parameters.
dampingstr or None, default=None: Damping strategy.
mufloat, default=1: Damping coefficient.
line_search_methodstr, default=”backtrack”: Line-search method.
line_search_condstr, default=”armijo”: Line-search condition.
solverstr, default=”solve”: Linear solver.
block_hessianbool, default=True: If True, use block-diagonal Hessian.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonTR(params, lr_init=1.0, trust_region_method='exact', damping=None, mu=1, solver='solve', block_hessian=False, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)[source]#

Bases: TrustRegionOptimizer

Newton method with exact Hessian and trust region.

Uses a trust-region solver (e.g., exact or Steihaug-Toint) to compute the step.

Parameters:

paramsParams: Parameter tensors.
radius_initfloat, default=1.0: Initial trust-region radius.
trust_region_methodstr, default=”exact”: Trust-region solver method.
dampingstr or None, default=None: Damping strategy.
mufloat, default=1: Damping coefficient.
solverstr, default=”solve”: Linear solver for the system.
block_hessianbool, default=False: If True, use block-diagonal Hessian.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`new_model_radius`(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonCG(params, lr_init=1, lr_method=None, damping=None, mu=1, solver='cg-trunc')[source]#

Bases: NumericalOptimizer

Newton-CG method (inexact Newton) using conjugate gradient to solve the linear system.

Uses exact Hessian but solves the system iteratively with CG.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1: Initial learning rate.
lr_methodstr or None, default=None: Learning rate initialization method.
dampingstr or None, default=None: Damping strategy.
mufloat, default=1: Damping coefficient.
solverstr, default=”cg-trunc”: Iterative solver (must be in iterative_solver_set).

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonCGLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, damping=None, mu=1, line_search_method='backtrack', line_search_cond='armijo', solver='cg-trunc')[source]#

Bases: LineSearchOptimizer

Newton-CG with line search.

Combines the iterative CG solution of the Newton system with a line search to determine the step length.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1: Initial learning rate.
lr_methodstr or None, default=None: Learning-rate initialization method.
c1, c2, tau, max_iter, tolline-search parameters.
dampingstr or None, default=None: Damping strategy.
mufloat, default=1: Damping coefficient.
line_search_methodstr, default=”backtrack”: Line-search method.
line_search_condstr, default=”armijo”: Stopping condition.
solverstr, default=”cg-trunc”: Iterative solver.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class NewtonCGTR(params, lr_init=1.0, damping=None, mu=1, *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)[source]#

Bases: TrustRegionOptimizer

Newton-CG with trust region (Steihaug-Toint).

Uses the Steihaug-Toint CG-trust-region method, which solves the trust-region subproblem iteratively with a CG approach that automatically handles negative curvature and the trust-region boundary.

Parameters:

paramsParams: Parameter tensors.
radius_initfloat, default=1.0: Initial trust-region radius.
dampingstr or None, default=None: Damping strategy.
mufloat, default=1: Damping coefficient.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`get_step_direction`(objective, grad_params)	Compute the un-scaled step direction by solving the system `H * p = -grad` (or an approximation).
`load_state_dict`(state_dict)	Load the optimizer state.
`new_model_radius`(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step