torch.set_deterministic

torch.set_deterministic(d)

Sets whether native PyTorch operations must use deterministic algorithms. When True, operations without deterministic algorithms will throw a :class:RuntimeError when called.

Warning

This feature is a beta feature, so it does not affect every nondeterministic operation yet. The following operations are affected by this flag.

The following normally-nondeterministic operations will act deterministically when d=True:

• torch.nn.Conv1d when called on CUDA tensor

• torch.nn.Conv2d when called on CUDA tensor

• torch.nn.Conv3d when called on CUDA tensor

• torch.nn.ConvTranspose1d when called on CUDA tensor

• torch.nn.ConvTranspose2d when called on CUDA tensor

• torch.nn.ConvTranspose3d when called on CUDA tensor

• torch.bmm() when called on sparse-dense CUDA tensors

The following normally-nondeterministic operations will throw a RuntimeError when d=True:

• torch.nn.AvgPool3d when called on a CUDA tensor that requires grad

• torch.nn.AdaptiveAvgPool2d when called on a CUDA tensor that requires grad

• torch.nn.AdaptiveAvgPool3d when called on a CUDA tensor that requires grad

• torch.nn.MaxPool3d when called on a CUDA tensor that requires grad

• torch.nn.AdaptiveMaxPool2d when called on a CUDA tensor that requires grad

• torch.nn.FractionalMaxPool2d when called on a CUDA tensor that requires grad

• torch.nn.FractionalMaxPool3d when called on a CUDA tensor that requires grad

• torch.nn.functional.interpolate() when called on a CUDA tensor that requires grad

  and one of the following modes is used: - linear - bilinear - bicubic - trilinear

• torch.nn.ReflectionPad1d when called on a CUDA tensor that requires grad

• torch.nn.ReflectionPad2d when called on a CUDA tensor that requires grad

• torch.nn.ReplicationPad1d when called on a CUDA tensor that requires grad

• torch.nn.ReplicationPad2d when called on a CUDA tensor that requires grad

• torch.nn.ReplicationPad3d when called on a CUDA tensor that requires grad

• torch.nn.NLLLoss when called on a CUDA tensor that requires grad

• torch.nn.CTCLoss when called on a CUDA tensor that requires grad

• torch.nn.EmbeddingBag when called on a CUDA tensor that requires grad

• torch.scatter_add_() when called on a CUDA tensor

• torch.index_add_() when called on a CUDA tensor

• torch.index_select() when called on a CUDA tensor that requires grad

• torch.repeat_interleave() when called on a CUDA tensor that requires grad

• torch.histc() when called on a CUDA tensor

• torch.bincount() when called on a CUDA tensor

A handful of CUDA operations are nondeterministic if the CUDA version is 10.2 or greater, unless the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8 is set. See the CUDA documentation for more details: https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility If one of these environment variable configurations is not set, a RuntimeError will be raised from these operations when called with CUDA tensors:

• torch.mm()

• torch.mv()

• torch.bmm()

Note that deterministic operations tend to have worse performance than non-deterministic operations.

Parameters

d (bool) – If True, force operations to be deterministic. If False, allow non-deterministic operations.