torch_numopt base classes

Custom Optimizer class

class CustomOptimizer(params: Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]], defaults: dict[str, Any])

Bases: Optimizer, ABC

Class for Optimization methods using second derivative information.

abstract step(x: Tensor, y: Tensor, loss_fn: Module, closure: Callable | None = None) → Iterable

Method to update the parameters of the Neural Network.

Parameters:

• x (torch.Tensor) – Inputs of the Neural Network.

• y (torch.Tensor) – Targets of the Neural Network.

• loss_fn (nn.Module) – Loss function to be optimized.

• closure (Callable) – Kept for compatibility, unused.

update(loss: float)

Function to update the internal parameters of the optimization procedure.

loss: float

Loss of the Neural Network with the new parameters.

Line Search Optimizer class

class LineSearchOptimizer(model: Module, scaling_matrix: ScalingMatrixCalculator, line_search: LineSearchSolver, lr_init: float = 1, lr_method: str | None = None, solver='solve')

Bases: NumericalOptimizer, ABC

Base class for gradient-based optimization algorithms with line search.

Parameters:

• model (nn.Module)

• lr_init (float) – Maximum learning rate in backtracking line search, if the learning rate is set as constant, this will be the value used.

• lr_method (str) – Method to use to initialize the learning rate before applying line search.

• line_search_cond (str (optional))

• line_search_method (str (optional))

• c1 (float (optional))

• c2 (float (optional))

• tau (float (optional))

apply_gradients(eval_model: Callable, params: list, d_p_list: list, h_list: list)

Updates the parameters of the network using a direction and a step length.

Parameters:

• lr (float)

• eval_model (Callable)

• params (list)

• d_p_list (list)

• h_list (list, optional)

Second Order Optimizer class