torchgan.losses¶
This losses subpackage is a collection of popular loss functions used in the training of GANs. Currently the following losses are supported:
These losses are tested with the current available trainers. So if you need
to implement you custom loss for using with the trainer
it is recommended
that you subclass the GeneratorLoss
and DiscriminatorLoss
.
Warning
The override_train_ops
gets only the arguments that were received by the
default train_ops
. Hence it might not be a wise to use this very often.
If this is used make sure to take into account the arguments and their order.
A better alternative is to subclass the Loss and define a custom train_ops
.
Warning
train_ops
are designed to be used internally through the Trainer
. Hence it
is highly recommended that this function is not directly used by external sources,
i.e. no call to this function is made outside the Trainer
.
Loss¶
GeneratorLoss¶
-
class
torchgan.losses.
GeneratorLoss
(reduction='mean', override_train_ops=None)[source]¶ Base class for all generator losses.
Note
All Losses meant to be minimized for optimizing the Generator must subclass this.
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
set_arg_map
(value)[source]¶ Updates the
arg_map
for passing a different value to thetrain_ops
.Parameters: value (dict) – A mapping of the argument name
in the method signature and the variable name in theTrainer
it corresponds to.Note
If the
train_ops
signature istrain_ops(self, gen, disc, optimizer_generator, device, batch_size, labels=None)
then we need to mapgen
togenerator
anddisc
todiscriminator
. In this case we make the following function callloss.set_arg_map({"gen": "generator", "disc": "discriminator"})
.
-
train_ops
(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]¶ Defines the standard
train_ops
used by most losses. Losses which have a different training procedure can eithersubclass
it (recommended approach) or make use ofoverride_train_ops
argument.The
standard optimization algorithm
for thegenerator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value = discriminator(fake)\)
- \(loss = loss\_function(value)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for generator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thegenerator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
- reduction (str, optional) – Specifies the reduction to apply to the output.
If
DiscriminatorLoss¶
-
class
torchgan.losses.
DiscriminatorLoss
(reduction='mean', override_train_ops=None)[source]¶ Base class for all discriminator losses.
Note
All Losses meant to be minimized for optimizing the Discriminator must subclass this.
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
set_arg_map
(value)[source]¶ Updates the
arg_map
for passing a different value to thetrain_ops
.Parameters: value (dict) – A mapping of the argument name
in the method signature and the variable name in theTrainer
it corresponds to.Note
If the
train_ops
signature istrain_ops(self, gen, disc, optimizer_discriminator, device, batch_size, labels=None)
then we need to mapgen
togenerator
anddisc
todiscriminator
. In this case we make the following function callloss.set_arg_map({"gen": "generator", "disc": "discriminator"})
.
-
train_ops
(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]¶ Defines the standard
train_ops
used by most losses. Losses which have a different training procedure can eithersubclass
it (recommended approach) or make use ofoverride_train_ops
argument.The
standard optimization algorithm
for thediscriminator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value_1 = discriminator(fake)\)
- \(value_2 = discriminator(real)\)
- \(loss = loss\_function(value_1, value_2)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for discriminator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
- reduction (str, optional) – Specifies the reduction to apply to the output.
If
Least Squares Loss¶
LeastSquaresGeneratorLoss¶
-
class
torchgan.losses.
LeastSquaresGeneratorLoss
(reduction='mean', c=1.0, override_train_ops=None)[source]¶ Least Squares GAN generator loss from “Least Squares Generative Adversarial Networks by Mao et. al.” paper
The loss can be described as
\[L(G) = \frac{(D(G(z)) - c)^2}{2}\]where
- \(G\) : Generator
- \(D\) : Disrciminator
- \(c\) : target generator label
- \(z\) : A sample from the noise prior
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - c (float, optional) – Target generator label.
- override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
forward
(dgz)[source]¶ Computes the loss for the given input.
Parameters: dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions. Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
LeastSquaresDiscriminatorLoss¶
-
class
torchgan.losses.
LeastSquaresDiscriminatorLoss
(reduction='mean', a=0.0, b=1.0, override_train_ops=None)[source]¶ Least Squares GAN discriminator loss from “Least Squares Generative Adversarial Networks by Mao et. al.” paper.
The loss can be described as:
\[L(D) = \frac{(D(x) - b)^2 + (D(G(z)) - a)^2}{2}\]where
- \(G\) : Generator
- \(D\) : Disrciminator
- \(a\) : Target discriminator label for generated image
- \(b\) : Target discriminator label for real image
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - a (float, optional) – Target discriminator label for generated image.
- b (float, optional) – Target discriminator label for real image.
- override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
forward
(dx, dgz)[source]¶ Computes the loss for the given input.
Parameters: - dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
- dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
Minimax Loss¶
MinimaxGeneratorLoss¶
-
class
torchgan.losses.
MinimaxGeneratorLoss
(reduction='mean', nonsaturating=True, override_train_ops=None)[source]¶ Minimax game generator loss from the original GAN paper “Generative Adversarial Networks by Goodfellow et. al.”
The loss can be described as:
\[L(G) = log(1 - D(G(z)))\]The nonsaturating heuristic is also supported:
\[L(G) = -log(D(G(z)))\]where
- \(G\) : Generator
- \(D\) : Discriminator
- \(z\) : A sample from the noise prior
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place of the default
train_ops
- nonsaturating (bool, optional) – Specifies whether to use the nonsaturating heuristic loss for the generator.
- reduction – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the mean of the output. Ifsum
the elements of the output will be summed.
-
forward
(dgz)[source]¶ Computes the loss for the given input.
Parameters: dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions. Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
MinimaxDiscriminatorLoss¶
-
class
torchgan.losses.
MinimaxDiscriminatorLoss
(label_smoothing=0.0, reduction='mean', override_train_ops=None)[source]¶ Minimax game discriminator loss from the original GAN paper “Generative Adversarial Networks by Goodfellow et. al.”
The loss can be described as:
\[L(D) = -[log(D(x)) + log(1 - D(G(z)))]\]where
- \(G\) : Generator
- \(D\) : Discriminator
- \(x\) : A sample from the data distribution
- \(z\) : A sample from the noise prior
Parameters: - label_smoothing (float, optional) – The factor by which the labels (1 in this case) needs to be smoothened. For example, label_smoothing = 0.2 changes the value of the real labels to 0.8.
- reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the mean of the output. Ifsum
the elements of the output will be summed. - override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
forward
(dx, dgz)[source]¶ Computes the loss for the given input.
Parameters: - dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
- dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
Boundary Equilibrium Loss¶
BoundaryEquilibriumGeneratorLoss¶
-
class
torchgan.losses.
BoundaryEquilibriumGeneratorLoss
(reduction='mean', override_train_ops=None)[source]¶ Boundary Equilibrium GAN generator loss from “BEGAN : Boundary Equilibrium Generative Adversarial Networks by Berthelot et. al.” paper
The loss can be described as
\[L(G) = D(G(z))\]where
- \(G\) : Generator
- \(D\) : Discriminator
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
forward
(dgz)[source]¶ Computes the loss for the given input.
Parameters: dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions. Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
BoundaryEquilibriumDiscriminatorLoss¶
-
class
torchgan.losses.
BoundaryEquilibriumDiscriminatorLoss
(reduction='mean', override_train_ops=None, init_k=0.0, lambd=0.001, gamma=0.75)[source]¶ Boundary Equilibrium GAN discriminator loss from “BEGAN : Boundary Equilibrium Generative Adversarial Networks by Berthelot et. al.” paper
The loss can be described as
\[L(D) = D(x) - k_t \times D(G(z))\]\[k_{t+1} = k_t + \lambda \times (\gamma \times D(x) - D(G(z)))\]where
- \(G\) : Generator
- \(D\) : Discriminator
- \(k_t\) : Running average of the balance point of G and D
- \(\lambda\) : Learning rate of the running average
- \(\gamma\) : Goal bias hyperparameter
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place ofthe default
train_ops
- init_k (float, optional) – Initial value of the balance point
k
. - lambd (float, optional) – Learning rate of the running average.
- gamma (float, optional) – Goal bias hyperparameter.
-
forward
(dx, dgz)[source]¶ Computes the loss for the given input.
Parameters: - dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
- dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: A tuple of 3 loss values, namely the
total loss
,loss due to real data
andloss due to fake data
.
-
set_k
(k=0.0)[source]¶ Change the default value of k
Parameters: k (float, optional) – New value to be set.
-
train_ops
(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]¶ Defines the standard
train_ops
used by boundary equilibrium loss.The
standard optimization algorithm
for thediscriminator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value_1 = discriminator(fake)\)
- \(value_2 = discriminator(real)\)
- \(loss = loss\_function(value_1, value_2)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for discriminator
- Update the value of :math: k.
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
Energy Based Loss¶
EnergyBasedGeneratorLoss¶
-
class
torchgan.losses.
EnergyBasedGeneratorLoss
(reduction='mean', override_train_ops=None)[source]¶ Energy Based GAN generator loss from “Energy Based Generative Adversarial Network by Zhao et. al.” paper.
The loss can be described as:
\[L(G) = D(G(z))\]where
- \(G\) : Generator
- \(D\) : Discriminator
- \(z\) : A sample from the noise prior
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
forward
(dgz)[source]¶ Computes the loss for the given input.
Parameters: dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions. Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
-
train_ops
(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]¶ This function sets the
embeddings
attribute of theAutoEncodingDiscriminator
toFalse
and calls thetrain_ops
of theGeneratorLoss
. After the call the attribute is again set toTrue
.Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thegenerator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
EnergyBasedDiscriminatorLoss¶
-
class
torchgan.losses.
EnergyBasedDiscriminatorLoss
(reduction='mean', margin=80.0, override_train_ops=None)[source]¶ Energy Based GAN generator loss from “Energy Based Generative Adversarial Network by Zhao et. al.” paper
The loss can be described as:
\[L(D) = D(x) + max(0, m - D(G(z)))\]where
- \(G\) : Generator
- \(D\) : Discriminator
- \(m\) : Margin Hyperparameter
- \(z\) : A sample from the noise prior
Note
The convergence of EBGAN is highly sensitive to hyperparameters. The
margin
hyperparameter as per the paper was taken as follows:Dataset Margin MNIST 10.0 LSUN 80.0 CELEB A 20.0 Imagenet (128 x 128) 40.0 Imagenet (256 x 256) 80.0 Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - margin (float, optional) – The margin hyperparameter.
- override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
forward
(dx, dgz)[source]¶ Computes the loss for the given input.
Parameters: - dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
- dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
-
train_ops
(generator, discriminator, optimizer_discriminator, real_inputs, device, batch_size, labels=None)[source]¶ This function sets the
embeddings
attribute of theAutoEncodingDiscriminator
toFalse
and calls thetrain_ops
of theDiscriminatorLoss
. After the call the attribute is again set toTrue
.Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
EnergyBasedPullingAwayTerm¶
-
class
torchgan.losses.
EnergyBasedPullingAwayTerm
(pt_ratio=0.1, override_train_ops=None)[source]¶ Energy Based Pulling Away Term from “Energy Based Generative Adversarial Network by Zhao et. al.” paper.
The loss can be described as:
\[f_{PT}(S) = \frac{1}{N(N-1)}\sum_i\sum_{j \neq i}\bigg(\frac{S_i^T S_j}{||S_i||\ ||S_j||}\bigg)^2\]where
- \(S\) : The feature output from the encoder for generated images
- \(N\) : Batch Size of the Input
Parameters: - pt_ratio (float, optional) – The weight given to the pulling away term.
- override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
forward
(dgz, d_hid)[source]¶ Computes the loss for the given input.
Parameters: - dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
- d_hid (torch.Tensor) – The embeddings generated by the discriminator.
Returns: scalar.
-
train_ops
(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]¶ This function extracts the hidden embeddings of the discriminator network. The furthur computation is same as the standard train_ops.
Note
For the loss to work properly, the discriminator must be a
AutoEncodingDiscriminator
and it must have aembeddings
attribute which should be set toTrue
. Also the generatorlabel_type
must benone
. As a result of these constraints it advisable not to use custom models with this loss. This will be improved in future.Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thegenerator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
Wasserstein Loss¶
WassersteinGeneratorLoss¶
-
class
torchgan.losses.
WassersteinGeneratorLoss
(reduction='mean', override_train_ops=None)[source]¶ Wasserstein GAN generator loss from “Wasserstein GAN by Arjovsky et. al.” paper
The loss can be described as:
\[L(G) = -f(G(z))\]where
- \(G\) : Generator
- \(f\) : Critic/Discriminator
- \(z\) : A sample from the noise prior
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the mean of the output. Ifsum
the elements of the output will be summed. - override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
forward
(fgz)[source]¶ Computes the loss for the given input.
Parameters: dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions. Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
WassersteinDiscriminatorLoss¶
-
class
torchgan.losses.
WassersteinDiscriminatorLoss
(reduction='mean', clip=None, override_train_ops=None)[source]¶ Wasserstein GAN generator loss from “Wasserstein GAN by Arjovsky et. al.” paper
The loss can be described as:
\[L(D) = f(G(z)) - f(x)\]where
- \(G\) : Generator
- \(f\) : Critic/Discriminator
- \(x\) : A sample from the data distribution
- \(z\) : A sample from the noise prior
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the mean of the output. Ifsum
the elements of the output will be summed. - clip (tuple, optional) – Tuple that specifies the maximum and minimum parameter clamping to be applied, as per the original version of the Wasserstein loss without Gradient Penalty.
- override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
forward
(fx, fgz)[source]¶ Computes the loss for the given input.
Parameters: - fx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
- fgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
-
train_ops
(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]¶ Defines the standard
train_ops
used by wasserstein discriminator loss.The
standard optimization algorithm
for thediscriminator
defined in this train_ops is as follows:- Clamp the discriminator parameters to satisfy \(lipschitz\ condition\)
- \(fake = generator(noise)\)
- \(value_1 = discriminator(fake)\)
- \(value_2 = discriminator(real)\)
- \(loss = loss\_function(value_1, value_2)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for discriminator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
WassersteinGradientPenalty¶
-
class
torchgan.losses.
WassersteinGradientPenalty
(reduction='mean', lambd=10.0, override_train_ops=None)[source]¶ Gradient Penalty for the Improved Wasserstein GAN discriminator from “Improved Training of Wasserstein GANs by Gulrajani et. al.” paper
The gradient penalty is calculated as:
The gradient being taken with respect to x
where
- \(G\) : Generator
- \(D\) : Disrciminator/Critic
- \(\lambda\) : Scaling hyperparameter
- \(x\) : Interpolation term for the gradient penalty
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the mean of the output. Ifsum
the elements of the output will be summed. - lambd (float,optional) – Hyperparameter lambda for scaling the gradient penalty.
- override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
forward
(interpolate, d_interpolate)[source]¶ Computes the loss for the given input.
Parameters: - interpolate (torch.Tensor) – It must have the dimensions (N, *) where * means any number of additional dimensions.
- d_interpolate (torch.Tensor) – Output of the
discriminator
withinterpolate
as the input. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
-
train_ops
(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]¶ Defines the standard
train_ops
used by the Wasserstein Gradient Penalty.The
standard optimization algorithm
for thediscriminator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(interpolate = \epsilon \times real + (1 - \epsilon) \times fake\)
- \(d\_interpolate = discriminator(interpolate)\)
- \(loss = \lambda loss\_function(interpolate, d\_interpolate)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for discriminator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
Mutual Information Penalty¶
-
class
torchgan.losses.
MutualInformationPenalty
(lambd=1.0, reduction='mean', override_train_ops=None)[source]¶ Mutual Information Penalty as defined in “InfoGAN : Interpretable Representation Learning by Information Maximising Generative Adversarial Nets by Chen et. al.” paper
The loss is the variational lower bound of the mutual information between the latent codes and the generator distribution and is defined as
\[L(G,Q) = log(Q|x)\]where
- \(x\) is drawn from the generator distribution G(z,c)
- \(c\) drawn from the latent code prior \(P(c)\)
Parameters: - lambd (float, optional) – The scaling factor for the loss.
- reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the mean of the output. Ifsum
the elements of the output will be summed. - override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
forward
(c_dis, c_cont, dist_dis, dist_cont)[source]¶ Computes the loss for the given input.
Parameters: - c_dis (int) – The discrete latent code sampled from the prior.
- c_cont (int) – The continuous latent code sampled from the prior.
- dist_dis (torch.distributions.Distribution) – The auxilliary distribution \(Q(c|x)\) over the discrete latent code output by the discriminator.
- dist_cont (torch.distributions.Distribution) – The auxilliary distribution \(Q(c|x)\) over the continuous latent code output by the discriminator.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
-
train_ops
(generator, discriminator, optimizer_generator, optimizer_discriminator, dis_code, cont_code, device, batch_size)[source]¶ Defines the standard
train_ops
used by most losses. Losses which have a different training procedure can eithersubclass
it (recommended approach) or make use ofoverride_train_ops
argument.The
standard optimization algorithm
for thegenerator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value = discriminator(fake)\)
- \(loss = loss\_function(value)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for generator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thegenerator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
Dragan Loss¶
DraganGradientPenalty¶
-
class
torchgan.losses.
DraganGradientPenalty
(reduction='mean', lambd=10.0, k=1.0, override_train_ops=None)[source]¶ Gradient Penalty for the DRAGAN discriminator from “On Convergence and Stability of GANs by Kodali et. al.” paper
The gradient penalty is calculated as:
\[\lambda \times (||grad(D(x))||_2 - k)^2\]The gradient being taken with respect to x
where
- \(G\) : Generator
- \(D\) : Disrciminator
- \(\lambda\) : Scaling hyperparameter
- \(x\) : Interpolation term for the gradient penalty
- \(k\) : Constant
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - lambd (float,optional) – Hyperparameter \(\lambda\) for scaling the gradient penalty.
- k (float, optional) – Constant.
- override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
forward
(interpolate, d_interpolate)[source]¶ Computes the loss for the given input.
Parameters: - interpolate (torch.Tensor) – It must have the dimensions (N, *) where * means any number of additional dimensions.
- d_interpolate (torch.Tensor) – Output of the
discriminator
withinterpolate
as the input. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
-
train_ops
(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]¶ Defines the standard
train_ops
used by the DRAGAN Gradient Penalty.The
standard optimization algorithm
for thediscriminator
defined in this train_ops is as follows:- \(interpolate = real + \frac{1}{2} \times (1 - \alpha) \times std(real) \times \beta\)
- \(d\_interpolate = discriminator(interpolate)\)
- \(loss = loss\_function(interpolate, d\_interpolate)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for discriminator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
Auxillary Classifier Loss¶
AuxiliaryClassifierGeneratorLoss¶
-
class
torchgan.losses.
AuxiliaryClassifierGeneratorLoss
(reduction='mean', override_train_ops=None)[source]¶ Auxiliary Classifier GAN (ACGAN) loss based on a from “Conditional Image Synthesis With Auxiliary Classifier GANs by Odena et. al. “ paper
Parameters: reduction (str, optional) – - Specifies the reduction to apply to the output.
- If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional): A function is passed to this argument,
- if the default
train_ops
is not to be used.
-
train_ops
(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]¶ Defines the standard
train_ops
used by most losses. Losses which have a different training procedure can eithersubclass
it (recommended approach) or make use ofoverride_train_ops
argument.The
standard optimization algorithm
for thegenerator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value = discriminator(fake)\)
- \(loss = loss\_function(value)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for generator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thegenerator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
AuxiliaryClassifierDiscriminatorLoss¶
-
class
torchgan.losses.
AuxiliaryClassifierDiscriminatorLoss
(reduction='mean', override_train_ops=None)[source]¶ Auxiliary Classifier GAN (ACGAN) loss based on a from “Conditional Image Synthesis With Auxiliary Classifier GANs by Odena et. al. “ paper
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – A function is passed to this argument,
if the default
train_ops
is not to be used.
-
train_ops
(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]¶ Defines the standard
train_ops
used by the Auxiliary Classifier discriminator loss.The
standard optimization algorithm
for thediscriminator
defined in this train_ops is as follows (label_g and label_d both could be either real labels or generated labels):- \(fake = generator(noise, label_g)\)
- \(value_1 = classifier(fake, label_g)\)
- \(value_2 = classifier(real, label_d)\)
- \(loss = loss\_function(value_1, label_g) + loss\_function(value_2, label_d)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for discriminator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized. For ACGAN, it must require labels for training
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
- reduction (str, optional) – Specifies the reduction to apply to the output.
If
Feature Matching Loss¶
FeatureMatchingGeneratorLoss¶
-
class
torchgan.losses.
FeatureMatchingGeneratorLoss
(reduction='mean', override_train_ops=None)[source]¶ Feature Matching Generator loss from “Improved Training of GANs by Salimans et. al.” paper
The loss can be described as:
\[L(G) = ||f(x)-f(G(z))||_2\]where
- \(G\) : Generator
- \(f\) : An intermediate activation from the discriminator
- \(z\) : A sample from the noise prior
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place of the default
train_ops
-
forward
(fx, fgz)[source]¶ Computes the loss for the given input.
Parameters: - dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
- dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns: scalar if reduction is applied else Tensor with dimensions (N, *).
-
train_ops
(generator, discriminator, optimizer_generator, real_inputs, device, labels=None)[source]¶ Defines the standard
train_ops
used for feature matching.The
standard optimization algorithm
for thegenerator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value_1 = discriminator(fake)\) where \(value_1\) is an activation of an intermediate
- discriminator layer
- \(value_2 = discriminator(real)\) where \(value_2\) is an activation of the same intermediate
- discriminator layer
- \(loss = loss\_function(value_1, value_2)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for generator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thegenerator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
Historical Averaging¶
HistoricalAverageGeneratorLoss¶
-
class
torchgan.losses.
HistoricalAverageGeneratorLoss
(reduction='elementwise_mean', override_train_ops=None, lambd=1.0)[source]¶ Historical Average Generator Loss from “Improved Techniques for Training GANs by Salimans et. al.” paper
The loss can be described as
\[|| \vtheta - \frac{1}{t} \sum_{i=1}^t \vtheta[i] ||^2\]where
- \(G\) : Generator
math: vtheta[i] : Generator Parameters at Past Timestep :math: i
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place of the default
train_ops
- lambd (float, optional) – Hyperparameter lambda for scaling the Historical Average Penalty
-
train_ops
(generator, optimizer_generator)[source]¶ Defines the standard
train_ops
used by most losses. Losses which have a different training procedure can eithersubclass
it (recommended approach) or make use ofoverride_train_ops
argument.The
standard optimization algorithm
for thegenerator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value = discriminator(fake)\)
- \(loss = loss\_function(value)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for generator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thegenerator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.
HistoricalAverageDiscriminatorLoss¶
-
class
torchgan.losses.
HistoricalAverageDiscriminatorLoss
(reduction='elementwise_mean', override_train_ops=None, lambd=1.0)[source]¶ Historical Average Discriminator Loss from “Improved Techniques for Training GANs by Salimans et. al.” paper
The loss can be described as
\[|| \vtheta - \frac{1}{t} \sum_{i=1}^t \vtheta[i] ||^2\]where
- \(G\) : Discriminator
math: vtheta[i] : Discriminator Parameters at Past Timestep :math: i
Parameters: - reduction (str, optional) – Specifies the reduction to apply to the output.
If
none
no reduction will be applied. Ifmean
the outputs are averaged over batch size. Ifsum
the elements of the output are summed. - override_train_ops (function, optional) – Function to be used in place of the default
train_ops
- lambd (float, optional) – Hyperparameter lambda for scaling the Historical Average Penalty
-
train_ops
(discriminator, optimizer_discriminator)[source]¶ Defines the standard
train_ops
used by most losses. Losses which have a different training procedure can eithersubclass
it (recommended approach) or make use ofoverride_train_ops
argument.The
standard optimization algorithm
for thediscriminator
defined in this train_ops is as follows:- \(fake = generator(noise)\)
- \(value_1 = discriminator(fake)\)
- \(value_2 = discriminator(real)\)
- \(loss = loss\_function(value_1, value_2)\)
- Backpropagate by computing \(\nabla loss\)
- Run a step of the optimizer for discriminator
Parameters: - generator (torchgan.models.Generator) – The model to be optimized.
- discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
- optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the
parameters
of thediscriminator
. - real_inputs (torch.Tensor) – The real data to be fed to the
discriminator
. - device (torch.device) – Device on which the
generator
anddiscriminator
is present. - batch_size (int) – Batch Size of the data infered from the
DataLoader
by theTrainer
. - labels (torch.Tensor, optional) – Labels for the data.
Returns: Scalar value of the loss.