torch_numopt.curvature.exact_hessian module

Exact Hessian computation using torch.func.

This module provides the full Hessian matrix (or block-diagonal approximation) via automatic differentiation. It supports damping (identity or Fletcher) and batched evaluation.

class ExactHessianCalculator(damping=None, mu=0.0001)

Bases: CurvatureEstimator

Compute the exact Hessian matrix (full or block) of the objective.

The Hessian is obtained using torch.func.hessian, which computes the full second-order derivatives. For large models, this can be memory- intensive; use the block version for parameter groups.

Parameters:

dampingstr or None, default=None

Damping strategy: "identity" adds mu * I, "fletcher" adds mu * diag(H). If None, no damping is applied.

mufloat, default=1e-4

Damping coefficient.

Methods

full_scaling_matrix(objective, params)	Return the curvature matrix as a single dense tensor.
hvp(objective, params, step_dir)	Compute the Hessian-vector product H * v.
quadratic_form(objective, params, grad_params)	Compute the quadratic form vᵀ H v.
reset()	Reset any internal state (intended to be used for quasi-Newton methods).
scaling_matrix(objective, params)	Calculation of the exact hessian of the Neural network given a dataset.
update()	Updates the parameters of the curvature estimator.

scaling_matrix(objective, params)

Calculation of the exact hessian of the Neural network given a dataset.

Return type:

Tuple[Tensor]

Parameters:

x: torch.Tensor

Input dataset for calculating the loss.

y: torch.Tensor

Target dataset for calculating the loss.

loss_fn: torch.Module

Loss function for which to calculate the hessian.

vectorize: boolean

Use vectorization in pytorch’s implementation of the hessian calculation.

hvp(objective, params, step_dir)

Compute the Hessian-vector product H * v.

Return type:

Tuple[Tensor]

Parameters:

objectiveObjectiveFunction

Objective function.

paramsParams

Parameter tensors.

step_dirParams

Vector v (same structure as params).

Returns:

Params

Result of H * v.

quadratic_form(objective, params, grad_params)

Compute the quadratic form vᵀ H v.

Return type:

Tensor

Parameters:

objectiveObjectiveFunction

Objective function.

paramsParams

Parameter tensors.

step_dirParams

Vector v.

Returns:

torch.Tensor

Scalar value vᵀ H v.