torch_numopt.line_search module#

Line-search algorithms for step-length determination.

This module provides various line-search strategies: backtracking, interpolation, and bisection. Each solver implements a specific method and can be combined with different stopping conditions (Armijo, Wolfe, Goldstein, etc.).

create_line_search_solver(method, condition, c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08)[source]#

Factory function to instantiate a line-search solver.

Parameters:

methodstr: One of "backtrack", "interpolate", "bisect".
conditionstr: Stopping condition: "greedy", "armijo", "wolfe", "strong-wolfe", "goldstein".
c1float, default=1e-4: Sufficient decrease parameter (Armijo).
c2float, default=0.9: Curvature condition parameter (Wolfe).
taufloat, default=0.1: Step-size reduction factor for backtracking.
max_iterint, default=20: Maximum number of iterations.
tolfloat, default=1e-8: Tolerance for stopping (e.g., minimum step size).

Returns:

LineSearchSolver: Instance of the requested solver.

class LineSearchSolver(condition='armijo', c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08)[source]#

Bases: ABC

Abstract base class for line-search solvers.

Subclasses must implement the line_search method to find an acceptable step length.

Parameters:

conditionstr, default=”armijo”: Stopping condition (see above).
c1float, default=1e-4: Sufficient decrease parameter.
c2float, default=0.9: Curvature condition parameter.
taufloat, default=0.1: Step reduction factor.
max_iterint, default=20: Maximum iterations.
tolfloat, default=1e-8: Tolerance (e.g., minimum step).

Methods

`accept_step`(params, new_params, step_dir, ...)	Check if the current step satisfies the chosen stopping condition.
`find_step_size`(params, step_dir, ...)	Perform the line search.

accept_step(params, new_params, step_dir, lr, loss, new_loss, grad_params)[source]#

Check if the current step satisfies the chosen stopping condition.

Parameters:

paramsParams: Current parameters.
new_paramsParams: Candidate parameters at step length lr.
step_dirParams: Search direction.
lrfloat: Step length.
losstorch.Tensor: Loss at params.
new_losstorch.Tensor: Loss at new_params.
grad_paramsParams: Gradient at params.

Returns:

bool: True if the step is acceptable.

abstract find_step_size(params, step_dir, grad_params, lr_init, objective)[source]#

Perform the line search.

Parameters:

paramsParams: Current parameters.
step_dirParams: Search direction.
grad_paramsParams: Gradient at current point.
lr_initfloat: Initial step length.
objectiveObjectiveFunction: Objective function.

Returns:

tuple (new_params, lr): The updated parameters and the chosen step length.

class BacktrackingLineSearch(condition='armijo', c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08)[source]#

Bases: LineSearchSolver

Backtracking line search with step reduction.

Starting from lr_init, the step size is repeatedly multiplied by tau until the chosen condition is satisfied or max_iter is reached.

Methods

`accept_step`(params, new_params, step_dir, ...)	Check if the current step satisfies the chosen stopping condition.
`find_step_size`(params, step_dir, ...)	Perform the line search.

find_step_size(params, step_dir, grad_params, lr_init, objective)[source]#

Perform the line search.

Parameters:

paramsParams: Current parameters.
step_dirParams: Search direction.
grad_paramsParams: Gradient at current point.
lr_initfloat: Initial step length.
objectiveObjectiveFunction: Objective function.

Returns:

tuple (new_params, lr): The updated parameters and the chosen step length.

class InterpolationLineSearch(condition='armijo', c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08)[source]#

Bases: LineSearchSolver

Line search using quadratic/cubic interpolation.

Uses the function and derivative values at two points to fit a polynomial and estimate the optimum step length.

Methods

`accept_step`(params, new_params, step_dir, ...)	Check if the current step satisfies the chosen stopping condition.
`find_step_size`(params, step_dir, ...)	Perform the line search.

find_step_size(params, step_dir, grad_params, lr_init, objective)[source]#

Perform the line search.

Parameters:

paramsParams: Current parameters.
step_dirParams: Search direction.
grad_paramsParams: Gradient at current point.
lr_initfloat: Initial step length.
objectiveObjectiveFunction: Objective function.

Returns:

tuple (new_params, lr): The updated parameters and the chosen step length.

class BisectionLineSearch(condition='armijo', c1=0.0001, c2=0.9, tau=0.1, max_iter=20, tol=1e-08)[source]#

Bases: LineSearchSolver

Bisection (binary search) line search.

It maintains an interval containing the optimal step and narrows it by checking the derivative sign.

Methods

`accept_step`(params, new_params, step_dir, ...)	Check if the current step satisfies the chosen stopping condition.
`find_step_size`(params, step_dir, ...)	Perform the line search.

find_step_size(params, step_dir, grad_params, lr_init, objective)[source]#

Perform the line search.

Parameters:

paramsParams: Current parameters.
step_dirParams: Search direction.
grad_paramsParams: Gradient at current point.
lr_initfloat: Initial step length.
objectiveObjectiveFunction: Objective function.

Returns:

tuple (new_params, lr): The updated parameters and the chosen step length.