torch_numopt.solve_system module#
Linear system solvers for curvature matrices.
This module provides functions to solve H p = -g (or similar) for various representations of the curvature (full matrix, block diagonal, diagonal, scalar). It includes direct solvers (Cholesky, LU, pseudo-inverse, least-squares) and iterative solvers (conjugate gradient, conjugate residual, truncated CG).
- solve_system(curvature_estimator, objective, rhs_params, solver=None, **kwargs)[source]#
High-level solver that selects the appropriate method based on curvature representation and the requested solver.
If the primary solver fails, it attempts fallbacks (e.g., least-squares).
- Return type:
Tuple[Tensor]- Parameters:
- curvature_estimatorCurvatureEstimator
Curvature estimator.
- objectiveObjectiveFunction
Objective function.
- rhs_paramsParams
Right-hand side vector (typically -gradient).
- solverstr or None, default=None
Solver name: one of
"pinv","pinv-trunc","solve","lsqrs","safe-lsqrs","cholesky"(direct), or"cg","cg-trunc","cr"(iterative). IfNone, a default is chosen.- **kwargs
Additional parameters passed to the specific solver.
- Returns:
- Params
Solution vector p.
- conjugate_gradient(curvature_estimator, objective, rhs, max_iter=100, atol=1e-08, tol=0.0001, min_iter=2)[source]#
Solve H p = rhs using the conjugate gradient (CG) method.
The Hessian (or its approximation) is accessed via the hvp method of the curvature estimator. This is an iterative method that only requires matrix- vector products, making it suitable for large-scale problems.
- Return type:
Tuple[Tensor]- Parameters:
- curvature_estimatorCurvatureEstimator
Provides the Hessian-vector product (Hvp).
- objectiveObjectiveFunction
Objective function (needed to pass parameters to Hvp).
- rhsParams
Right-hand side vector (typically -gradient).
- max_iterint, default=100
Maximum number of CG iterations.
- atolfloat, default=1e-8
Absolute tolerance for the residual norm.
- tolfloat, default=1e-4
Relative tolerance (residual norm <= tol * norm(rhs)).
- min_iterint, default=2
Minimum iterations before checking stopping criteria.
- Returns:
- Params
Solution vector p.
- truncated_cg(curvature_estimator, objective, rhs, max_iter=100, atol=1e-08, tol=0.0001, min_iter=2)[source]#
Truncated conjugate gradient method.
Similar to CG, but it stops early if a direction of negative curvature is encountered (pᵀHp ≤ 0).
- Return type:
Tuple[Tensor]- Parameters:
- curvature_estimatorCurvatureEstimator
Provides the Hessian-vector product (Hvp).
- objectiveObjectiveFunction
Objective function (needed to pass parameters to Hvp).
- rhsParams
Right-hand side vector (typically -gradient).
- max_iterint, default=100
Maximum number of CG iterations.
- atolfloat, default=1e-8
Absolute tolerance for the residual norm.
- tolfloat, default=1e-4
Relative tolerance (residual norm <= tol * norm(rhs)).
- min_iterint, default=2
Minimum iterations before checking stopping criteria.
- Returns:
- Params
Approximate solution (may be truncated).
- conjugate_residual(curvature_estimator, objective, rhs, max_iter=100, atol=1e-08, tol=0.0001, min_iter=2)[source]#
Conjugate residual method for solving H p = rhs.
Similar to CG but uses the residual norm in the construction of the search direction. It can be more robust for certain non-symmetric systems, though here it is used with symmetric H.
- Parameters:
- curvature_estimatorCurvatureEstimator
Provides the Hessian-vector product (Hvp).
- objectiveObjectiveFunction
Objective function (needed to pass parameters to Hvp).
- rhsParams
Right-hand side vector (typically -gradient).
- max_iterint, default=100
Maximum number of CG iterations.
- atolfloat, default=1e-8
Absolute tolerance for the residual norm.
- tolfloat, default=1e-4
Relative tolerance (residual norm <= tol * norm(rhs)).
- min_iterint, default=2
Minimum iterations before checking stopping criteria.
- Returns:
- Params
Solution vector p.