torch_numopt package

Base classes

• torch_numopt base classes
  • Custom Optimizer class
  • Line Search Optimizer class
  • Second Order Optimizer class

Implemented algorithms

• torch_numopt implemented Algorithms
  • Gradient Descent with Line Search
  • Newton’s method with Line Search
  • Gauss-Newton algorithm with Line Search
  • Levenberg-Marquardt algorithm with Line Search
  • Conjugate Gradient algorithm with Line Search
  • AdaHessian
  • Quasi-Newton

utils

param_sizes(params: list)

Obtains the shape of every matrix in the list of parameters provided.

Parameters:

params (list) – List of matrices containing a list of parameters.

param_reshape_like(params_flat: Tensor, params: list)

Reshapes a vector into a list of matrices with the same shapes as the params parameter.

Parameters:

• params_flat (Tensor) – Vector with the parameters to reshape.

• params (list) – List of matrices with the desired shape.

Returns:

reshaped_params

Return type:

Tensor

param_flatten(params: list)

fix_stability(mat: Tensor)

Procedure to adjust a matrix by adding a very small value to the diagonal to avoid numerical instability problems.

Parameters:

mat (torch.Tensor) – Ill conditioned matrix.

Returns:

fixed_mat – (Hopefully) Well conditioned matrix.

Return type:

torch.Tensor

pinv_svd_trunc(mat: Tensor, thresh: float = 0.0001)

Procedure to calculate the pseudoinverse of a matrix by using truncated SVD in order to maintain numerical stability.

Parameters:

• mat (torch.Tensor) – Problematic matrix that we want to invert.

• thresh (float) – Threshold applied to the S matrix in the SVD procedure.

Returns:

inverted_mat – Pseudoinverse of the input matrix.

Return type:

torch.Tensor