torch_numopt.curvature.exact_block_hessian module#
Block-diagonal exact Hessian.
Instead of computing the full Hessian matrix, this estimator computes only the diagonal blocks (one per parameter group), ignoring cross-group second derivatives. This reduces memory and computational cost.
- class ExactBlockHessianCalculator(damping=None, mu=0.0001)[source]#
Bases:
CurvatureEstimatorExact Hessian approximated as a block-diagonal matrix.
Each block corresponds to a single parameter tensor (e.g., a weight matrix). This is often sufficient for many optimization problems and is cheaper than the full Hessian.
- Parameters:
- dampingstr or None, default=None
Damping strategy (see
ExactHessianCalculator).- mufloat, default=1e-4
Damping coefficient.
Methods
full_scaling_matrix(objective, params)Return the curvature matrix as a single dense tensor.
hvp(objective, params, step_dir)Compute the Hessian-vector product H * v.
quadratic_form(objective, params, grad_params)Compute the quadratic form vᵀ H v.
reset()Reset any internal state (intended to be used for quasi-Newton methods).
scaling_matrix(objective, params)Calculation of the exact hessian of the Neural network given a dataset.
update()Updates the parameters of the curvature estimator.
- scaling_matrix(objective, params)[source]#
Calculation of the exact hessian of the Neural network given a dataset.
- Return type:
Tuple[Tensor]- Parameters:
- x: torch.Tensor
Input dataset for calculating the loss.
- y: torch.Tensor
Target dataset for calculating the loss.
- loss_fn: torch.Module
Loss function for which to calculate the hessian.
- vectorize: boolean
Use vectorization in pytorch’s implementation of the hessian calculation.