torch_numopt.algorithms.gradient_descent module

Gradient descent optimizers (first-order).

These optimizers use only the gradient (identity curvature). They are the simplest methods and serve as a baseline.

class GradientDescent(params, lr_init=0.001, lr_method=None)

Bases: NumericalOptimizer

Vanilla gradient descent with a fixed or adaptively initialized learning rate.

Parameters:

paramsParams

Parameter tensors.

lr_initfloat, default=1e-3

Initial learning rate.

lr_methodstr, optional

Learning-rate initialization method (see NumericalOptimizer).

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GradientDescentLS(params, lr_init=1, lr_method=None, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08, line_search_method='backtrack', line_search_cond='armijo')

Bases: LineSearchOptimizer

Gradient descent with a line search to determine the step length.

This optimizer computes the steepest descent direction and then performs a line search (backtracking, interpolation, or bisection) to find a step size that satisfies the chosen condition (Armijo, Wolfe, Goldstein, etc.).

Parameters:

paramsParams

Parameter tensors to optimize.

lr_initfloat, default=1

Initial guess for the learning rate (used as the starting point for the line search).

lr_methodstr or None, default=None

Method for initializing the learning rate before the line search (see NumericalOptimizer for options). If None, uses lr_init directly.

c1float, default=1e-4

Sufficient decrease parameter (Armijo condition).

c2float, default=0.9

Curvature condition parameter (Wolfe conditions).

taufloat, default=0.1

Step-size reduction factor for backtracking.

max_iterint, default=20

Maximum number of line-search iterations.

tolfloat, default=1e-8

Tolerance for stopping (e.g., minimum step size).

line_search_methodstr, default=”backtrack”

Line-search algorithm. Options: “backtrack”, “interpolate”, “bisect”.

line_search_condstr, default=”armijo”

Stopping condition. Options: “greedy”, “armijo”, “wolfe”, “strong-wolfe”, “goldstein”.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GradientDescentTR(params, lr_init=1.0, trust_region_method='cauchy', *, accept_tol=0.1, contract_tol=0.25, expand_tol=0.75, growth_factor=2, shrink_factor=0.25, radius_max=1000.0)

Bases: TrustRegionOptimizer

Gradient descent with a trust-region (Cauchy point) step.

This optimizer builds a quadratic model using the gradient (identity curvature) and solves the trust-region subproblem using the Cauchy point, which is the minimizer of the model along the steepest descent direction within the trust region.

Parameters:

paramsParams

Parameter tensors to optimize.

radius_initfloat, default=1.0

Initial trust-region radius.

trust_region_methodstr, default=”cauchy”

Trust-region solver method. For gradient descent, only “cauchy” is recommended, but other methods (e.g., “dogleg”, “exact”) can be used if a curvature estimator is provided (here it uses identity, so they degenerate to Cauchy point anyway).

accept_tolfloat, default=0.1

Threshold for the ratio rho (actual vs. predicted reduction) above which the step is accepted.

curvature_estimatorCurvatureEstimator, optional

Not used directly (identity is forced), kept for compatibility with the base class.

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
new_model_radius(objective, radius, loss, ...)	Update the trust-region radius based on the ratio rho.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step

class GradientDescentLipschitz(params, lr_init=0.001)

Bases: NumericalOptimizer

Gradient descent with a learning rate estimated from the Lipschitz constant.

This optimizer uses a heuristic (the “lipschitz” method) to estimate the learning rate at each step based on the change in gradient and parameters from the previous iteration. It avoids the need for manual tuning and often converges faster than fixed-rate gradient descent.

Parameters:

paramsParams

Parameter tensors to optimize.

lr_initfloat, default=1e-3

Initial learning rate (used only for the first iteration).

Methods

add_param_group(param_group)	Add a param group to the Optimizer s param_groups.
apply_gradients(objective, params, grad_params)	Update parameters using the current gradient and curvature.
get_step_direction(objective, grad_params)	Compute the un-scaled step direction by solving the system H * p = -grad (or an approximation).
load_state_dict(state_dict)	Load the optimizer state.
register_load_state_dict_post_hook(hook[, ...])	Register a load_state_dict post-hook which will be called after load_state_dict() is called. It should have the following signature::.
register_load_state_dict_pre_hook(hook[, ...])	Register a load_state_dict pre-hook which will be called before load_state_dict() is called. It should have the following signature::.
register_state_dict_post_hook(hook[, prepend])	Register a state dict post-hook which will be called after state_dict() is called.
register_state_dict_pre_hook(hook[, prepend])	Register a state dict pre-hook which will be called before state_dict() is called.
register_step_post_hook(hook)	Register an optimizer step post hook which will be called after optimizer step.
register_step_pre_hook(hook)	Register an optimizer step pre hook which will be called before optimizer step.
state_dict()	Return the state of the optimizer as a dict.
step(objective)	Perform one optimization step.
zero_grad([set_to_none])	Reset the gradients of all optimized torch.Tensor s.

OptimizerPostHook
OptimizerPreHook
profile_hook_step