torch_numopt.utils package#
Submodules#
Module contents#
Utility functions for parameter operations, stability, and type conversions.
These helpers operate on tuples of tensors (Params) and provide common operations like addition, scaling, dot product, norm, reshaping, etc.
- param_dot(params_a, params_b)[source]#
Computes the dot product between two parameter groups treated as a single flat vector.
- Parameters:
- params_aParams
First iterable of tensors.
- params_bParams
Second iterable of tensors, same shapes as params_a.
- Returns:
- torch.Tensor
Scalar tensor equal to sum_{p_a, p_b} (p_a * p_b).sum().
- param_add(params_a, params_b)[source]#
Element-wise addition of two parameter groups.
- Parameters:
- params_aParams
First iterable of tensors.
- params_bParams
Second iterable of tensors, must have the same structure as params_a.
- Returns:
- tuple of torch.Tensor
A new tuple where each element is the sum of the corresponding elements of params_a and params_b.
- param_argnums(params)[source]#
Return a tuple of indices (0, 1, …, len(params)-1) for the parameter groups.
This is a convenience function used to pass argnums to torch.func when computing Hessians or Jacobians.
- Return type:
tuple
- param_copy(params)[source]#
Creates a deep copy of a parameter group (each tensor cloned).
- Parameters:
- paramsParams
Iterable of tensors to copy.
- Returns:
- tuple of torch.Tensor
New tuple containing detached clones of the original tensors.
- param_detach(params)[source]#
Detach (and clone) all tensors from the computation graph.
- Return type:
Tuple[Tensor]
- param_diff(params_a, params_b)[source]#
Element-wise subtraction of two parameter groups.
- Parameters:
- params_aParams
First iterable of tensors.
- params_bParams
Second iterable of tensors, same structure.
- Returns:
- tuple of torch.Tensor
New tuple where each element is params_a[i] - params_b[i].
- param_flatten(params)[source]#
Flattens an entire parameter group into a single 1-dimensional tensor.
- Parameters:
- paramsParams
Iterable of tensors (possibly nested). Each tensor is flattened and concatenated.
- Returns:
- torch.Tensor
1-D tensor containing all parameter values concatenated in order.
- param_is_finite(params)[source]#
Checks that all elements in a parameter group are finite.
- Parameters:
- paramsParams
Iterable of tensors to inspect.
- Returns:
- bool
True if every element of every tensor is finite, False otherwise.
- param_reshape_like(params_flat, params)[source]#
Reshapes a vector into a sequence of matrices with the same shapes as the params parameter.
- Parameters:
- params_flat: Tensor
Vector with the parameters to reshape.
- params: Params
Sequence of matrices with the desired shape.
- Returns:
- reshaped_params: Tensor
- param_mult(params_a, params_b)[source]#
Element-wise (Hadamard) product of two parameter groups.
- Parameters:
- params_aParams
First iterable of tensors.
- params_bParams
Second iterable of tensors, same shapes.
- Returns:
- tuple of torch.Tensor
New tuple where each element is params_a[i] * params_b[i].
- param_neg(params)[source]#
Negates every tensor in a parameter group.
- Parameters:
- paramsParams
Iterable of tensors.
- Returns:
- tuple of torch.Tensor
New tuple with each element negated (-p).
- param_norm(params)[source]#
Euclidean (L2) norm of a parameter group treated as a flat vector.
- Parameters:
- paramsParams
Iterable of tensors.
- Returns:
- torch.Tensor
Scalar tensor equal to sqrt(sum_{p} (p * p).sum()).
- param_numel(params)[source]#
Returns the total amount of parameters.
- Return type:
int- Parameters:
- paramsParams
Iterable of tensors.
- Returns:
- int
Count of the total number of parameters.
- param_scalar_prod(scalar, params)[source]#
Multiplies every tensor in a parameter group by a scalar.
- Parameters:
- scalarfloat
The scalar multiplier.
- paramsParams
Iterable of tensors to scale.
- Returns:
- tuple of torch.Tensor
New tuple containing each tensor multiplied by scalar.
- param_scaled_add(params_a, params_b, scale)[source]#
Computes params_a + scale * params_b element-wise.
- Parameters:
- params_aParams
Base parameter group.
- params_bParams
Parameter group to be scaled and added.
- scalefloat
Scaling factor for params_b.
- Returns:
- tuple of torch.Tensor
New tuple with the result of the scaled addition.
- param_sizes(params)[source]#
Obtains the shape of every matrix in the parameters provided.
- Parameters:
- params: Params
Sequence of matrices containing a sequence of parameters.
- param_transpose(params)[source]#
Transposes every 2-dimensional tensor in the parameter group.
- Parameters:
- paramsParams
Iterable of tensors; only 2D tensors are transposed, others are left unchanged.
- Returns:
- tuple of torch.Tensor
New tuple with transposed tensors where applicable.
- param_zero_like(params)[source]#
Create a tuple of zero tensors with the same shapes as the input parameters.
- Parameters:
- paramsParams
Iterable of tensors providing the shapes.
- Returns:
- tuple of torch.Tensor
A tuple of tensors, each filled with zeros and with the same shape and device as the corresponding input tensor.
- fix_stability(mat)[source]#
Procedure to adjust a matrix by adding a very small value to the diagonal to avoid numerical instability problems.
- Parameters:
- mat: torch.Tensor
Ill conditioned matrix.
- Returns:
- fixed_mat: torch.Tensor
(Hopefully) Well conditioned matrix.
- pinv_svd_trunc(mat, thresh=0.0001)[source]#
Procedure to calculate the pseudoinverse of a matrix by using truncated SVD in order to maintain numerical stability.
- Parameters:
- mat: torch.Tensor
Problematic matrix that we want to invert.
- thresh: float
Threshold applied to the S matrix in the SVD procedure.
- Returns:
- inverted_mat: torch.Tensor
Pseudoinverse of the input matrix.