On this page
Adadelta
class torch.optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0, foreach=None, *, maximize=False, differentiable=False)
[source]-
Implements Adadelta algorithm.
For further details regarding the algorithm we refer to ADADELTA: An Adaptive Learning Rate Method.
- Parameters
-
- params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
- rho (float, optional) – coefficient used for computing a running average of squared gradients (default: 0.9)
- eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-6)
- lr (float, optional) – coefficient that scale delta before it is applied to the parameters (default: 1.0)
- weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)
- foreach (bool, optional) – whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to the intermediates being a tensorlist vs just one tensor. If memory is prohibitive, batch fewer parameters through the optimizer at a time or switch this flag to False (default: None)
- maximize (bool, optional) – maximize the params based on the objective, instead of minimizing (default: False)
- differentiable (bool, optional) – whether autograd should occur through the optimizer step in training. Otherwise, the step() function runs in a torch.no_grad() context. Setting to True can impair performance, so leave it False if you don’t intend to run autograd through this instance (default: False)
add_param_group(param_group)
-
Add a param group to the
Optimizer
sparam_groups
.This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the
Optimizer
as training progresses.- Parameters
-
param_group (dict) – Specifies what Tensors should be optimized along with group specific optimization options.
load_state_dict(state_dict)
-
Loads the optimizer state.
- Parameters
-
state_dict (dict) – optimizer state. Should be an object returned from a call to
state_dict()
.
register_load_state_dict_post_hook(hook, prepend=False)
-
Register a load_state_dict post-hook which will be called after
load_state_dict()
is called. It should have the following signature:hook(optimizer) -> None
The
optimizer
argument is the optimizer instance being used.The hook will be called with argument
self
after callingload_state_dict
onself
. The registered hook can be used to perform post-processing afterload_state_dict
has loaded thestate_dict
.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided post
hook
will be fired before all the already registered post-hooks onload_state_dict
. Otherwise, the providedhook
will be fired after all the already registered post-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_load_state_dict_pre_hook(hook, prepend=False)
-
Register a load_state_dict pre-hook which will be called before
load_state_dict()
is called. It should have the following signature:hook(optimizer, state_dict) -> state_dict or None
The
optimizer
argument is the optimizer instance being used and thestate_dict
argument is a shallow copy of thestate_dict
the user passed in toload_state_dict
. The hook may modify the state_dict inplace or optionally return a new one. If a state_dict is returned, it will be used to be loaded into the optimizer.The hook will be called with argument
self
andstate_dict
before callingload_state_dict
onself
. The registered hook can be used to perform pre-processing before theload_state_dict
call is made.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided pre
hook
will be fired before all the already registered pre-hooks onload_state_dict
. Otherwise, the providedhook
will be fired after all the already registered pre-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_state_dict_post_hook(hook, prepend=False)
-
Register a state dict post-hook which will be called after
state_dict()
is called. It should have the following signature:hook(optimizer, state_dict) -> state_dict or None
The hook will be called with arguments
self
andstate_dict
after generating astate_dict
onself
. The hook may modify the state_dict inplace or optionally return a new one. The registered hook can be used to perform post-processing on thestate_dict
before it is returned.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided post
hook
will be fired before all the already registered post-hooks onstate_dict
. Otherwise, the providedhook
will be fired after all the already registered post-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_state_dict_pre_hook(hook, prepend=False)
-
Register a state dict pre-hook which will be called before
state_dict()
is called. It should have the following signature:hook(optimizer) -> None
The
optimizer
argument is the optimizer instance being used. The hook will be called with argumentself
before callingstate_dict
onself
. The registered hook can be used to perform pre-processing before thestate_dict
call is made.- Parameters
-
- hook (Callable) – The user defined hook to be registered.
- prepend (bool) – If True, the provided pre
hook
will be fired before all the already registered pre-hooks onstate_dict
. Otherwise, the providedhook
will be fired after all the already registered pre-hooks. (default: False)
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemoveableHandle
register_step_post_hook(hook)
-
Register an optimizer step post hook which will be called after optimizer step. It should have the following signature:
hook(optimizer, args, kwargs) -> None
The
optimizer
argument is the optimizer instance being used.- Parameters
-
hook (Callable) – The user defined hook to be registered.
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemovableHandle
register_step_pre_hook(hook)
-
Register an optimizer step pre hook which will be called before optimizer step. It should have the following signature:
hook(optimizer, args, kwargs) -> None or modified args and kwargs
The
optimizer
argument is the optimizer instance being used. If args and kwargs are modified by the pre-hook, then the transformed values are returned as a tuple containing the new_args and new_kwargs.- Parameters
-
hook (Callable) – The user defined hook to be registered.
- Returns
-
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
-
torch.utils.hooks.RemovableHandle
state_dict()
-
Returns the state of the optimizer as a
dict
.It contains two entries:
-
state: a Dict holding current optimization state. Its content
-
differs between optimizer classes, but some common characteristics hold. For example, state is saved per parameter, and the parameter itself is NOT saved.
state
is a Dictionary mapping parameter ids to a Dict with state corresponding to each parameter.
-
param_groups: a List containing all parameter groups where each
-
parameter group is a Dict. Each parameter group contains metadata specific to the optimizer, such as learning rate and weight decay, as well as a List of parameter IDs of the parameters in the group.
NOTE: The parameter IDs may look like indices but they are just IDs associating state with param_group. When loading from a state_dict, the optimizer will zip the param_group
params
(int IDs) and the optimizerparam_groups
(actualnn.Parameter
s) in order to match state WITHOUT additional verification.A returned state dict might look something like:
{ 'state': { 0: {'momentum_buffer': tensor(...), ...}, 1: {'momentum_buffer': tensor(...), ...}, 2: {'momentum_buffer': tensor(...), ...}, 3: {'momentum_buffer': tensor(...), ...} }, 'param_groups': [ { 'lr': 0.01, 'weight_decay': 0, ... 'params': [0] }, { 'lr': 0.001, 'weight_decay': 0.5, ... 'params': [1, 2, 3] } ] }
-
step(closure=None)
[source]-
Performs a single optimization step.
- Parameters
-
closure (Callable, optional) – A closure that reevaluates the model and returns the loss.
zero_grad(set_to_none=True)
-
Resets the gradients of all optimized
torch.Tensor
s.- Parameters
-
set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests
zero_grad(set_to_none=True)
followed by a backward pass,.grad
s are guaranteed to be None for params that did not receive a gradient. 3.torch.optim
optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).
© 2024, PyTorch Contributors
PyTorch has a BSD-style license, as found in the LICENSE file.
https://pytorch.org/docs/2.1/generated/torch.optim.Adadelta.html