torch_numopt.curvature.hutchinson_diagonal_approximation module#
Hutchinson diagonal Hessian approximation.
This estimator uses random Rademacher vectors and the Hessian-vector product to estimate the diagonal of the Hessian. It is unbiased and works well for large models where full Hessian computation is infeasible.
- class HutchinsonDiagonalApproximation(n_samples=1, skip_iters=0)[source]#
Bases:
CurvatureEstimatorDiagonal Hessian estimator via Hutchinson’s method.
The diagonal is estimated as the average of z ⊙ (H z) over random Rademacher vectors z.
- Parameters:
- n_samplesint, default=1
Number of random samples to average.
Methods
full_scaling_matrix(objective, params)Return the curvature matrix as a single dense tensor.
hvp(objective, params, step_dir)Compute the Hessian-vector product H * v.
quadratic_form(objective, params, step_dir)Compute the quadratic form vᵀ H v.
reset()Reset any internal state (intended to be used for quasi-Newton methods).
scaling_matrix(objective, params)Obtain the curvature matrix in its native representation.
update()Updates the parameters of the curvature estimator.
- scaling_matrix(objective, params)[source]#
Obtain the curvature matrix in its native representation.
- Return type:
Tuple[Tensor]- Parameters:
- objectiveObjectiveFunction
Objective function.
- paramsParams
Parameter tensors.
- Returns:
- iterable or torch.Tensor
Representation of the matrix (scalar, vector, tuple of blocks, or full tensor).
- hvp(objective, params, step_dir)[source]#
Compute the Hessian-vector product H * v.
- Return type:
Tuple[Tensor]- Parameters:
- objectiveObjectiveFunction
Objective function.
- paramsParams
Parameter tensors.
- step_dirParams
Vector v (same structure as params).
- Returns:
- Params
Result of H * v.