"### Automatic Differentiation with ``torch.autograd``\n",
"## Automatic Differentiation with ``torch.autograd``\n",
"\n",
"When training a model, e.g. a neural network, the most frequently used algorithm is\n",
"**back propagation**. In this algorithm, parameters (model weights) are\n",
...
...
%% Cell type:markdown id: tags:
# PyTorch Tutorial: Tensors and Autograd
This Tutorial will give you an overview about PyTorch tensors and PyTorch's build in autograd function to perform backpropagation, which is the key mechanism for machine learning.
This tutorial is adapted from https://pytorch.org/tutorials/beginner/basics/intro.html
%% Cell type:code id: tags:
``` python
%matplotlibinline
```
%% Cell type:markdown id: tags:
## Tensors
Tensors are a specialized data structure that are very similar to arrays and matrices.
In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters.
Tensors are similar to [NumPy’s](https://numpy.org/) ndarrays, except that tensors can run on GPUs or other hardware accelerators. In fact, tensors and
NumPy arrays can often share the same underlying memory, eliminating the need to copy data. Tensors
are also optimized for automatic differentiation.
%% Cell type:code id: tags:
``` python
importtorch
importnumpyasnp
```
%% Cell type:markdown id: tags:
### Initializing a Tensor
Tensors can be initialized in various ways. Take a look at the following examples:
**Directly from data**
Tensors can be created directly from data. The data type is automatically inferred.
%% Cell type:code id: tags:
``` python
data=[[1,2],[3,4]]
x_data=torch.tensor(data)
```
%% Cell type:markdown id: tags:
**From a NumPy array**
Tensors can be created from NumPy arrays.
%% Cell type:code id: tags:
``` python
np_array=np.array(data)
x_np=torch.from_numpy(np_array)
```
%% Cell type:markdown id: tags:
**From another tensor:**
The new tensor retains the properties (shape, datatype) of the argument tensor, unless explicitly overridden.
%% Cell type:code id: tags:
``` python
x_ones=torch.ones_like(x_data)# retains the properties of x_data
x_ones
```
%% Cell type:markdown id: tags:
Create different PyTorch tensors and try to use the Numpy operations and manipulations you learnt in the previous Numpy tutorial. Nearly all operations are applicable analogously.
%% Cell type:code id: tags:
``` python
# your code goes here
```
%% Cell type:markdown id: tags:
### Attributes of a Tensor & its Device
Tensor attributes describe their shape, datatype, and the device on which they are stored.
%% Cell type:code id: tags:
``` python
tensor=torch.rand(3,4)
print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")
```
%% Cell type:markdown id: tags:
As you can see, the tensor is currently stored on `cpu`. If you want to switch the device of your tensor zo e.g. a GPU, you can to that with the `.to()` operation. GPU devices are enumerated starting from zero: `cuda:0`, `cuda:1`, ...
In the scope of this Blockseminar you will get access to a GPU, if you do not have access yet, the following lines are there to show you the concept of PyTorch devices and might throw errors.
Create a tensor directly on the GPU:
%% Cell type:code id: tags:
``` python
data=torch.randn((2,4),device='cuda:0')
data.device
```
%% Cell type:markdown id: tags:
Move it back and forth. `.cpu()` is a shortcut for `.to('cpu)`
%% Cell type:code id: tags:
``` python
data=data.to('cpu')
data.device
```
%% Cell type:code id: tags:
``` python
data=data.to('cuda:0')
data.device
```
%% Cell type:code id: tags:
``` python
data=data.cpu()
data.device
```
%% Cell type:markdown id: tags:
### Automatic Differentiation with ``torch.autograd``
## Automatic Differentiation with ``torch.autograd``
When training a model, e.g. a neural network, the most frequently used algorithm is
**back propagation**. In this algorithm, parameters (model weights) are
adjusted according to the **gradient** of the loss function with respect
to the given parameter.
To compute those gradients, PyTorch has a built-in differentiation engine
called ``torch.autograd``. It supports automatic computation of gradient for any
computational graph.
Consider a simple linear function, with input ``x``,
parameters ``w`` and ``b``, and loss function which computes the binary cross entropy (CE) between the a target `y` and the output `z` of our linear function. It can be defined in
PyTorch in the following manner:
%% Cell type:code id: tags:
``` python
x=torch.ones(5)# input tensor
y=torch.zeros(3)# expected output
w=torch.randn(5,3,requires_grad=True)
b=torch.randn(3,requires_grad=True)
```
%% Cell type:markdown id: tags:
Note that we told PyTorch that we want to compute gradients for the parameters `w` and `b` by stating `requires_grad=True`!
Now we can start operation with our tensors while PyTorch will track each operation which is somehow connected to `w` or `b`.
This means that PyTorch creates a computational graph and stores to compute gradients later.
Not let's compose a linear operation and loss computation to create our graph.
This code defines the following **computational graph**:
In this network, ``w`` and ``b`` are **parameters**, which we need to
optimize. Thus, we need to be able to compute the gradients of the loss
function with respect to those variables. In order to do that, we set
the ``requires_grad`` property of those tensors.
%% Cell type:markdown id: tags:
### Computing Gradients
To optimize weights of parameters in the neural network, we need to
compute the derivatives of our loss function with respect to parameters,
namely, we need $\frac{\partial loss}{\partial w}$ and
$\frac{\partial loss}{\partial b}$ under some fixed values of
``x`` and ``y``. To compute those derivatives, we call
``loss.backward()``, and then retrieve the values from ``w.grad`` and
``b.grad``:
%% Cell type:code id: tags:
``` python
loss.backward()
print(w.grad)
print(b.grad)
```
%% Cell type:markdown id: tags:
Congratulations! You just computed gradients using PyTorch's autograd method. Later, we will use these gradients to update our parameters (here `w` and `b`).
Note that this step is a central point to train your model. Especially for complex and big models holds:
- Computing gradients is very costly, therefore one should use GPUs which are way faster than CPUs
- Storing computational graphs needs a lot of RAM (ideally on your GPU), therefore you should have an eye on efficient gradient tracking
%% Cell type:markdown id: tags:
### Disabling Gradient Tracking
By default, all tensors with ``requires_grad=True`` are tracking their
computational history and support gradient computation. However, there
are some cases when we do not need to do that, for example, when we have
trained the model and just want to apply it to some input data, i.e. we
only want to do *forward* computations through the network. We can stop
tracking computations by surrounding our computation code with
``torch.no_grad()`` block:
%% Cell type:code id: tags:
``` python
z=torch.matmul(x,w)+b
print(z.requires_grad)
withtorch.no_grad():
z=torch.matmul(x,w)+b
print(z.requires_grad)
```
%% Cell type:markdown id: tags:
Another way to achieve the same result is to use the ``detach()`` method
on the tensor:
%% Cell type:code id: tags:
``` python
z=torch.matmul(x,w)+b
z_det=z.detach()
print(z_det.requires_grad)
```
%% Cell type:markdown id: tags:
There are reasons you might want to disable gradient tracking:
- To mark some parameters in your neural network as **frozen parameters**.
- To **speed up computations** when you are only doing forward pass, because computations on tensors that do
not track gradients would be more efficient.
%% Cell type:markdown id: tags:
--------------
%% Cell type:markdown id: tags:
## What you should have learned from this notebook:
- What is a PyTorch Tensor & what is thair advantage compared to plain Numpy arrays?
- Why should you make Tensoroperations on GPUs
- What is PyTorch's autograd mechanism and how does it work?
- How to compute gradients and how to disable gradient tracking
