torch_numopt.algorithms.conjugate_gradient module#

Non-linear conjugate gradient methods.

These methods combine the gradient with previous search directions to achieve faster convergence than gradient descent, without requiring explicit curvature.

class ConjugateGradientMixin(*args, cg_method='PRP+', **kwargs)[source]#

Bases: object

Mixin that provides the conjugate gradient direction computation.

Supports formulas: FR (Fletcher-Reeves), PR (Polak-Ribière), PRP+ (positive version), HS (Hestenes-Stiefel), DY (Dai-Yuan).

Parameters:

cg_methodstr, default=”PRP+”: Formula name.

Methods

get_step_direction

get_step_direction(objective, grad_params)[source]#

class ConjugateGradient(params, lr_init=1.0, lr_method='lipschitz', cg_method='PRP+')[source]#

Bases: ConjugateGradientMixin, NumericalOptimizer

Non-linear conjugate gradient optimizer with fixed learning rate.

Uses the Fletcher-Reeves, Polak-Ribière, etc. formulas to compute the search direction without explicit curvature.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1.0: Initial learning rate.
lr_methodstr or None, default=”lipschitz”: Learning rate initialization method.
cg_methodstr, default=”PRP+”: Conjugate gradient formula: “FR”, “PR”, “PRP+”, “HS”, “DY”.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
get_step_direction
profile_hook_step

class ConjugateGradientLS(params, lr_init=1.0, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo', cg_method='PRP+')[source]#

Bases: ConjugateGradientMixin, LineSearchOptimizer

Non-linear conjugate gradient with line search.

Combines conjugate gradient direction with a line-search to determine the step length.

Parameters:

paramsParams: Parameter tensors.
lr_initfloat, default=1.0: Initial learning rate.
lr_methodstr or None, default=None: Learning rate initialization method.
c1float, default=1e-4: Sufficient decrease parameter (Armijo).
c2float, default=0.9: Curvature condition parameter (Wolfe).
taufloat, default=0.1: Step reduction factor for backtracking.
max_iterint, default=20: Maximum iterations for line search.
tolfloat, default=1e-8: Tolerance (e.g., minimum step).
line_search_methodstr, default=”backtrack”: Line-search method (see create_line_search_solver).
line_search_condstr, default=”armijo”: Line-search stopping condition.
cg_methodstr, default=”PRP+”: Conjugate gradient formula.

Methods

`add_param_group`(param_group)	Add a param group to the `Optimizer` s param_groups.
`apply_gradients`(objective, params, grad_params)	Update parameters using the current gradient and curvature.
`load_state_dict`(state_dict)	Load the optimizer state.
`register_load_state_dict_post_hook`(hook[, ...])	Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature::.
`register_load_state_dict_pre_hook`(hook[, ...])	Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature::.
`register_state_dict_post_hook`(hook[, prepend])	Register a state dict post-hook which will be called after `state_dict()` is called.
`register_state_dict_pre_hook`(hook[, prepend])	Register a state dict pre-hook which will be called before `state_dict()` is called.
`register_step_post_hook`(hook)	Register an optimizer step post hook which will be called after optimizer step.
`register_step_pre_hook`(hook)	Register an optimizer step pre hook which will be called before optimizer step.
`state_dict`()	Return the state of the optimizer as a `dict`.
`step`(objective)	Perform one optimization step.
`zero_grad`([set_to_none])	Reset the gradients of all optimized `torch.Tensor` s.

OptimizerPostHook
OptimizerPreHook
get_step_direction
profile_hook_step