On this page
LBFGS
class torch.optim.LBFGS(params, lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn=None)
[source]-
Implements L-BFGS algorithm, heavily inspired by minFunc.
Warning
This optimizer doesn’t support per-parameter options and parameter groups (there can be only one).
Warning
Right now all parameters have to be on a single device. This will be improved in the future.
Note
This is a very memory intensive optimizer (it requires additional
param_bytes * (history_size + 1)
bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.- Parameters
-
- lr (float) – learning rate (default: 1)
- max_iter (int) – maximal number of iterations per optimization step (default: 20)
- max_eval (int) – maximal number of function evaluations per optimization step (default: max_iter * 1.25).
- tolerance_grad (float) – termination tolerance on first order optimality (default: 1e-7).
- tolerance_change (float) – termination tolerance on function value/parameter changes (default: 1e-9).
- history_size (int) – update history size (default: 100).
- line_search_fn (str) – either ‘strong_wolfe’ or None (default: None).
add_param_group(param_group)
-
Add a param group to the
Optimizer
sparam_groups
.This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the
Optimizer
as training progresses.- Parameters
-
param_group (dict) – Specifies what Tensors should be optimized along with group specific optimization options.
load_state_dict(state_dict)
-
Loads the optimizer state.
- Parameters
-
state_dict (dict) – optimizer state. Should be an object returned from a call to
state_dict()
.
register_load_state_dict_post_hook(hook, prepend=False)
-
Register a load_state_dict post-hook which will be called after
load_state_dict()
is called. It should have the following signature:hook(optimizer) -> None
The
optimizer
argument is the optimizer instance being used.The hook will be called with argument
self
after callingload_state_dict
onself
. The registered hook can be used to perform post-processing afterload_state_dict
has loaded thestate_dict
.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided post
hook
will be fired before all the already registered post-hooks onload_state_dict
. Otherwise, the providedhook
will be fired after all the already registered post-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_load_state_dict_pre_hook(hook, prepend=False)
-
Register a load_state_dict pre-hook which will be called before
load_state_dict()
is called. It should have the following signature:hook(optimizer, state_dict) -> state_dict or None
The
optimizer
argument is the optimizer instance being used and thestate_dict
argument is a shallow copy of thestate_dict
the user passed in toload_state_dict
. The hook may modify the state_dict inplace or optionally return a new one. If a state_dict is returned, it will be used to be loaded into the optimizer.The hook will be called with argument
self
andstate_dict
before callingload_state_dict
onself
. The registered hook can be used to perform pre-processing before theload_state_dict
call is made.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided pre
hook
will be fired before all the already registered pre-hooks onload_state_dict
. Otherwise, the providedhook
will be fired after all the already registered pre-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_state_dict_post_hook(hook, prepend=False)
-
Register a state dict post-hook which will be called after
state_dict()
is called. It should have the following signature:hook(optimizer, state_dict) -> state_dict or None
The hook will be called with arguments
self
andstate_dict
after generating astate_dict
onself
. The hook may modify the state_dict inplace or optionally return a new one. The registered hook can be used to perform post-processing on thestate_dict
before it is returned.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided post
hook
will be fired before all the already registered post-hooks onstate_dict
. Otherwise, the providedhook
will be fired after all the already registered post-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_state_dict_pre_hook(hook, prepend=False)
-
Register a state dict pre-hook which will be called before
state_dict()
is called. It should have the following signature:hook(optimizer) -> None
The
optimizer
argument is the optimizer instance being used. The hook will be called with argumentself
before callingstate_dict
onself
. The registered hook can be used to perform pre-processing before thestate_dict
call is made.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided pre
hook
will be fired before all the already registered pre-hooks onstate_dict
. Otherwise, the providedhook
will be fired after all the already registered pre-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_step_post_hook(hook)
-
Register an optimizer step post hook which will be called after optimizer step. It should have the following signature:
hook(optimizer, args, kwargs) -> None
The
optimizer
argument is the optimizer instance being used.- Parameters
-
hook (Callable) – The user defined hook to be registered.
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemovableHandle
register_step_pre_hook(hook)
-
Register an optimizer step pre hook which will be called before optimizer step. It should have the following signature:
hook(optimizer, args, kwargs) -> None or modified args and kwargs
The
optimizer
argument is the optimizer instance being used. If args and kwargs are modified by the pre-hook, then the transformed values are returned as a tuple containing the new_args and new_kwargs.- Parameters
-
hook (Callable) – The user defined hook to be registered.
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemovableHandle
state_dict()
-
Returns the state of the optimizer as a
dict
.It contains two entries:
-
state: a Dict holding current optimization state. Its content
-
differs between optimizer classes, but some common characteristics hold. For example, state is saved per parameter, and the parameter itself is NOT saved.
state
is a Dictionary mapping parameter ids to a Dict with state corresponding to each parameter.
-
param_groups: a List containing all parameter groups where each
-
parameter group is a Dict. Each parameter group contains metadata specific to the optimizer, such as learning rate and weight decay, as well as a List of parameter IDs of the parameters in the group.
NOTE: The parameter IDs may look like indices but they are just IDs associating state with param_group. When loading from a state_dict, the optimizer will zip the param_group
params
(int IDs) and the optimizerparam_groups
(actualnn.Parameter
s) in order to match state WITHOUT additional verification.A returned state dict might look something like:
{ 'state': { 0: {'momentum_buffer': tensor(...), ...}, 1: {'momentum_buffer': tensor(...), ...}, 2: {'momentum_buffer': tensor(...), ...}, 3: {'momentum_buffer': tensor(...), ...} }, 'param_groups': [ { 'lr': 0.01, 'weight_decay': 0, ... 'params': [0] }, { 'lr': 0.001, 'weight_decay': 0.5, ... 'params': [1, 2, 3] } ] }
-
step(closure)
[source]-
Performs a single optimization step.
- Parameters
-
closure (Callable) – A closure that reevaluates the model and returns the loss.
zero_grad(set_to_none=True)
-
Resets the gradients of all optimized
torch.Tensor
s.- Parameters
-
set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests
zero_grad(set_to_none=True)
followed by a backward pass,.grad
s are guaranteed to be None for params that did not receive a gradient. 3.torch.optim
optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).
© 2024, PyTorch Contributors
PyTorch has a BSD-style license, as found in the LICENSE file.
https://pytorch.org/docs/2.1/generated/torch.optim.LBFGS.html