Gradient calculation examples in PyTorch¶

Examples of gradient calculation in PyTorch:

• input is scalar; output is scalar
• input is vector; output is scalar
• input is scalar; output is vector
• input is vector; output is vector

import torch
from torch.autograd import Variable

input is scalar; output is scalar

x = Variable(torch.rand(1), requires_grad=True); x
Variable containing:
 0.2045
[torch.FloatTensor of size 1]
y = 3*x**2
y.backward()
x.grad
Variable containing:
 1.2272
[torch.FloatTensor of size 1]
x*6
Variable containing:
 1.2272
[torch.FloatTensor of size 1]

Equivalently:

x = Variable(torch.rand(1), requires_grad=True); x
Variable containing:
 0.1551
[torch.FloatTensor of size 1]
y = 3*x**2
y.backward(gradient=torch.ones(1))
x.grad
Variable containing:
 0.9308
[torch.FloatTensor of size 1]
x*6
Variable containing:
 0.9308
[torch.FloatTensor of size 1]

Set a non-ones gradient:

x = Variable(torch.rand(1), requires_grad=True); x
Variable containing:
 0.8009
[torch.FloatTensor of size 1]
y = 3*x**2
y.backward(gradient=100*torch.ones(1))
x.grad
Variable containing:
 480.5225
[torch.FloatTensor of size 1]
100*x*6
Variable containing:
 480.5225
[torch.FloatTensor of size 1]

input is vector; output is scalar

x = Variable(torch.rand(2, 1), requires_grad = True); x
Variable containing:
 0.4827
 0.7438
[torch.FloatTensor of size 2x1]
y = x[0]**3 + 2*x[1]**2

y.backward()
x.grad
Variable containing:
 0.6991
 2.9752
[torch.FloatTensor of size 2x1]
3*x[0]**2, 4*x[1]
(Variable containing:
  0.6991
 [torch.FloatTensor of size 1], Variable containing:
  2.9752
 [torch.FloatTensor of size 1])

Equivalently:

x = Variable(torch.rand(2, 1), requires_grad = True); x
Variable containing:
 0.1466
 0.7782
[torch.FloatTensor of size 2x1]
y = x[0]**3 + 2*x[1]**2

y.backward(gradient=torch.ones(1))
x.grad
Variable containing:
 0.0645
 3.1126
[torch.FloatTensor of size 2x1]
3*x[0]**2, 4*x[1]
(Variable containing:
 1.00000e-02 *
   6.4461
 [torch.FloatTensor of size 1], Variable containing:
  3.1126
 [torch.FloatTensor of size 1])

Set a non-ones gradient:

x = Variable(torch.rand(2, 1), requires_grad = True); x
Variable containing:
 0.6475
 0.9496
[torch.FloatTensor of size 2x1]
y = x[0]**3 + 2*x[1]**2

y.backward(gradient=100*torch.ones(1))
x.grad
Variable containing:
 125.7611
 379.8543
[torch.FloatTensor of size 2x1]
3*x[0]**2, 4*x[1]
(Variable containing:
  1.2576
 [torch.FloatTensor of size 1], Variable containing:
  3.7985
 [torch.FloatTensor of size 1])

input is scalar; output is vector

x = Variable(torch.rand(1), requires_grad=True); x
Variable containing:
 0.2806
[torch.FloatTensor of size 1]
y = Variable(torch.zeros(2, 1), requires_grad=False)
y[0] = x**3
y[1] = 2*x**2
y
Variable containing:
 0.0221
 0.1575
[torch.FloatTensor of size 2x1]
y.backward(gradient=torch.ones(y.size()))
x.grad
Variable containing:
 1.3587
[torch.FloatTensor of size 1]

why?

3*x**2 + 4*x
Variable containing:
 1.3587
[torch.FloatTensor of size 1]

input is vector; output is vector

x = Variable(torch.rand(2, 1), requires_grad = True); x
Variable containing:
 0.4342
 0.3613
[torch.FloatTensor of size 2x1]
y = Variable(torch.zeros(3, 1), requires_grad=False)
y[0] = x[0]**2
y[1] = x[1]**3
y[2] = x[1]**4
y.backward(gradient=torch.ones(y.size()))
x.grad
Variable containing:
 0.8684
 0.5802
[torch.FloatTensor of size 2x1]
2*x[0], 3*x[1]**2, 4*x[1]**3
(Variable containing:
  0.8684
 [torch.FloatTensor of size 1], Variable containing:
  0.3916
 [torch.FloatTensor of size 1], Variable containing:
  0.1886
 [torch.FloatTensor of size 1])
2*x[0], 3*x[1]**2 + 4*x[1]**3
(Variable containing:
  0.8684
 [torch.FloatTensor of size 1], Variable containing:
  0.5802
 [torch.FloatTensor of size 1])

Code is here: https://github.com/yang-zhang/deep-learning/blob/master/pytorch_grad.ipynb