torch_numopt.step_initializer module#

create_step_size_init(method, lr_init, curvature_estimator, min_lr=1e-18, max_lr=100, **kwargs)[source]#

class StepSizeInitializer(lr_init, curvature_estimator, min_lr=1e-18, max_lr=100)[source]#

Bases: ABC

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

abstract get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.

class ConstantStepSize(lr_init, curvature_estimator, min_lr=1e-18, max_lr=100)[source]#

Bases: StepSizeInitializer

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.

class KeepStepSize(lr_init, curvature_estimator, min_lr=1e-18, max_lr=100)[source]#

Bases: StepSizeInitializer

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.

class ScaledStepSize(lr_init, curvature_estimator, min_lr=1e-18, max_lr=100)[source]#

Bases: StepSizeInitializer

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.

class QuadraticStepSize(lr_init, curvature_estimator, min_lr=1e-18, max_lr=100)[source]#

Bases: StepSizeInitializer

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.

class InterpolateStepSize(lr_init, curvature_estimator, min_lr=1e-18, max_lr=100)[source]#

Bases: StepSizeInitializer

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.

class BarzilaiBorweinStepSize(*args, long_step=True, **kwargs)[source]#

Bases: StepSizeInitializer

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.

class LipschitzStepSize(lr_init, curvature_estimator, min_lr=1e-18, max_lr=100)[source]#

Bases: StepSizeInitializer

Methods

`__call__`(objective, params, grad_params, ...)	Generates the initial step size to be used adjusting it appropriately.
`get_initial_step`(objective, params, ...)	Generates the initial step size to be used.

get_initial_step(objective, params, grad_params, prev_grad, step_dir, prev_step_dir, prev_lr, delta_loss)[source]#

Generates the initial step size to be used.

Parameters:

objectiveObjectiveFunction: Objective function.
paramsParams: Current parameters.
grad_paramsParams: Gradient of the parameters.
prev_gradParams: Gradient on the previous iteration.
step_dirParams: Step direction.
prev_step_dirParams: Step direction on the previous iteration
prev_lrfloat: Previous step size

Returns:

float: Next step size.