torch_numopt.algorithms.lbfgs module#

Limited-memory BFGS (L-BFGS) optimizers.

L-BFGS is a quasi-Newton method that approximates the inverse Hessian using a limited history of past updates (s, y pairs). It is memory-efficient and works well for medium-scale optimization problems. This module provides both a fixed learning rate variant and a line-search variant (recommended).

class LBFGSMixin(*args, memory_size=10, **kwargs)[source]#

Bases: object

Methods

apply_gradients
get_step_direction

get_step_direction(objective, grad_params)[source]#

apply_gradients(objective, params, grad_params)[source]#

class LBFGS(params, lr_init=1.0, lr_method=None, memory_size=10)[source]#

Bases: LBFGSMixin, NumericalOptimizer

Limited-memory BFGS optimizer with fixed learning rate.

Maintains a history of past updates (s, y) to approximate the inverse Hessian.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1.0: Initial learning rate.
lr_methodstr or None, default=None: Learning rate initialization method.
memory_sizeint, default=10: Number of past updates to store.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
apply_gradients
get_step_direction
profile_hook_step

class LBFGSLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, memory_size=10, line_search_method='interpolate', line_search_cond='wolfe')[source]#

Bases: LBFGSMixin, LineSearchOptimizer

L-BFGS with line search.

After computing the L-BFGS direction, a line search is performed to find an appropriate step length. This is the recommended way to use L-BFGS.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1: Initial learning rate.
lr_methodstr or None, default=None: Learning-rate initialization method.
c1, c2, tau, max_iter, tolline-search parameters.
memory_sizeint, default=10: Number of stored (s, y) pairs.
line_search_methodstr, default=”interpolate”: Line-search method (interpolate is often good for L-BFGS).
line_search_condstr, default=”wolfe”: Condition (Wolfe conditions are typical for L-BFGS).

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
apply_gradients
get_step_direction
profile_hook_step