{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# CS568:Deep Learning Spring 2020 "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pytorch Tutorial\n",
"Pytorch is a python framework for deep learning developed and maintained by Facebook \n",
"\n",
"- GPU-accelerated computations\n",
"- automatic differentiation\n",
"- modules for neural networks\n",
"\n",
"This tutorial will teach you the fundamentals of operating on pytorch tensors and networks."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Install Pytorch in virtual environment\n",
"What is virtual environment and why should we use?\n",
"\n",
"Virtual environment is just like a container which contains libraries or packages. Each libray in this environment have its own dependencies, as opposed to being installed globally (in site-packages). \n",
"\n",
"**create virtual environment**\n",
"\n",
"`conda create -n pytorch_env python`\n",
"\n",
"**activate environment**\n",
"\n",
"`activate pytorch_env`\n",
"\n",
"**deactivate environment**\n",
"\n",
"`deactivate pytorch_env`\n",
"\n",
"**delete enviroment**\n",
"\n",
"`conda remove -n pytorch_env --all`\n",
"\n",
"**check environments list**\n",
"\n",
"`conda info --envs`\n",
"\n",
"**Install PyTorch**\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch version: 1.2.0\n"
]
}
],
"source": [
"import torch\n",
"import numpy as np\n",
"print(\"torch version:\", torch.__version__) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Tensor in Pytorch**\n",
"\n",
"A tensor is an n-dimensional data container similar to the NumPy array. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Construct a tensor with specific values"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[1., 2.],\n",
" [3., 4.]])\n",
"tensor([[1., 2., 3.],\n",
" [4., 5., 6.]])\n"
]
}
],
"source": [
"# creata a tensor\n",
"new_tensor = torch.Tensor([[1,2],[3,4]])\n",
"print(new_tensor) \n",
"two_tensor = torch.Tensor([[1, 2, 3], [4, 5, 6]])\n",
"print(two_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0.0611, 0.1386, 0.5363],\n",
" [0.8549, 0.6718, 0.2395]])\n"
]
}
],
"source": [
"#create a 2 x 3 tensor with random values\n",
"rand_tensor = torch.rand(2, 3)\n",
"print(rand_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[ 0.5184, -0.2175, -0.0854],\n",
" [-0.5723, -0.2873, -0.7551]])\n"
]
}
],
"source": [
"# create a 2 x 3 tensor with random values between -1 and 1\n",
"\n",
"uniform_tensor = torch.Tensor(2, 3).uniform_(-1, 1)\n",
"print(uniform_tensor)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Construct a randomly initialized matrix."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0.3588, 0.0499, 0.1543],\n",
" [0.8727, 0.6540, 0.6864]])\n"
]
}
],
"source": [
"# create a 2 x 3 tensor with random values from a uniform distribution [0, 1)\n",
"rand_tensor = torch.rand(2, 3)\n",
"print(rand_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0., 0., 0.],\n",
" [0., 0., 0.]])\n"
]
}
],
"source": [
"# create a 2 x 3 tensor of zeros\n",
"zero_tensor = torch.zeros(2, 3)\n",
"print(zero_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[1., 1., 1.],\n",
" [1., 1., 1.]])\n"
]
}
],
"source": [
"# create a 2 x 3 tensor of ones\n",
"one_tensor = torch.ones(2, 3)\n",
"print(one_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[1., 0., 0., 0., 0.],\n",
" [0., 1., 0., 0., 0.],\n",
" [0., 0., 1., 0., 0.],\n",
" [0., 0., 0., 1., 0.],\n",
" [0., 0., 0., 0., 1.]])\n"
]
}
],
"source": [
"# create an identity tensor\n",
"id_tensor = torch.eye(5)\n",
"print(id_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[1 2 3]\n",
" [4 5 6]]\n",
"tensor([[1, 2, 3],\n",
" [4, 5, 6]])\n"
]
}
],
"source": [
"# create a tensor from numpy\n",
"numpy_tensor = np.array([[1,2,3],[4,5,6]])\n",
"print(numpy_tensor)\n",
"num_tensor = torch.from_numpy(numpy_tensor)\n",
"print(num_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([0.0000, 0.2500, 0.5000, 0.7500, 1.0000])\n"
]
}
],
"source": [
"# create linearly spaced (evenly spaced numbers over a specified interval) tensors\n",
"lin_space_tensors = torch.linspace(0, 1, steps=5)\n",
"print(lin_space_tensors)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([ 10., 100., 1000.])\n"
]
}
],
"source": [
"# create logarithmically spaced (evenly spaced numbers on a log scale) tensors\n",
"log_space_tensors = torch.logspace(1, 3, steps=3)\n",
"print(log_space_tensors)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([1, 3, 5, 7, 9])\n"
]
}
],
"source": [
"# create a 1-d tensor with the values from start to end\n",
"ar_tensor = torch.arange(1, 10, step=2)\n",
"print(ar_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.FloatTensor\n",
"torch.Size([2, 2])\n",
"torch.Size([2, 2])\n",
"2\n"
]
}
],
"source": [
"new_tensor = torch.Tensor([[1, 2], [3, 4]])\n",
"\n",
"# type of a tensor \n",
"print(new_tensor.type())\n",
"\n",
"# shape of a tensor\n",
"print(new_tensor.shape)\n",
"print(new_tensor.size())\n",
"print(new_tensor.dim())"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[1., 2.],\n",
" [3., 4.]])\n",
"torch.Size([2, 2])\n",
"tensor([[1., 2., 3., 4.]]) torch.Size([1, 4])\n"
]
}
],
"source": [
"# reshape a tensor using view command\n",
"print(new_tensor)\n",
"print(new_tensor.shape)\n",
"\n",
"reshape_tensor = new_tensor.view(1,4)\n",
"print(reshape_tensor,reshape_tensor.shape)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0.6855, 0.4461]]) torch.Size([1, 2])\n",
"torch.Size([4, 2])\n",
"tensor([[0.6855, 0.4461],\n",
" [0.6855, 0.4461],\n",
" [0.6855, 0.4461],\n",
" [0.6855, 0.4461]])\n"
]
}
],
"source": [
"# concatenation operation \n",
"a = torch.rand(1,2)\n",
"print(a, a.shape)\n",
"four_a = torch.cat((a,a,a,a),0)\n",
"print(four_a.size())\n",
"print(four_a)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0.8467],\n",
" [0.1415],\n",
" [0.2987],\n",
" [0.1783],\n",
" [0.2050],\n",
" [0.1003],\n",
" [0.7750],\n",
" [0.5080],\n",
" [0.8448],\n",
" [0.1744]])\n",
"2\n",
"torch.Size([5, 1])\n",
"(tensor([[0.8467],\n",
" [0.1415],\n",
" [0.2987],\n",
" [0.1783],\n",
" [0.2050]]), tensor([[0.1003],\n",
" [0.7750],\n",
" [0.5080],\n",
" [0.8448],\n",
" [0.1744]]))\n"
]
}
],
"source": [
"# break a tensor into two parts\n",
"a = torch.rand(10,1)\n",
"print(a)\n",
"b = torch.chunk(a, 2, 0)\n",
"print(len(b))\n",
"print(b[0].size())\n",
"print(b)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"10\n"
]
}
],
"source": [
"# number of elements in a tensor\n",
"num_elements = torch.numel(a)\n",
"print(num_elements)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"x = torch.rand([2,2])\n",
"print(x)\n",
"\n",
"# concatenate\n",
"con_x = torch.cat((x,x),0)\n",
"print(con_x.shape, con_x)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[1.],\n",
" [3.],\n",
" [5.]])\n"
]
}
],
"source": [
"# select some indices values using Index select\n",
"t = torch.Tensor([[1,2],[3,4],[5,6]])\n",
"indices = torch.LongTensor([0])\n",
"ind_select = torch.index_select(t, 1, indices) # select from dim 1\n",
"print(ind_select)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([2., 3., 4., 5., 6.])\n"
]
}
],
"source": [
"# Masked select\n",
"t = torch.Tensor([[1,2],[3,4],[5,6]])\n",
"mask = t.ge(2) #greater than\n",
"masked_select = torch.masked_select(t,mask)\n",
"print(masked_select)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0, 0],\n",
" [2, 1]])\n"
]
}
],
"source": [
"# Get indices of non-zero elements \n",
"t = torch.Tensor([[1,0],[0,0],[0,6]])\n",
"non_zero_val_indices = torch.nonzero(t)\n",
"print(non_zero_val_indices)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([2, 1, 2, 1]) torch.Size([2, 2]) torch.Size([2, 2, 1])\n",
"torch.Size([1, 2, 2, 1])\n"
]
}
],
"source": [
"# Squeeze and unsqueeze\n",
"t = torch.ones(2,1,2,1) # Size 2x1x2x1\n",
"sq_t = torch.squeeze(t) # Size 2x2\n",
"sq_t1 = torch.squeeze(t, 1) # Squeeze dimension 1: Size 2x2x1\n",
"print(t.shape,sq_t.shape,sq_t1.shape)\n",
"\n",
"# Un-squeeze a dimension\n",
"usq_t1 = torch.unsqueeze(sq_t1, 0) \n",
"print(usq_t1.shape)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0.8808, 0.2138, 0.1377],\n",
" [0.6653, 0.9389, 0.1948]])\n",
"tensor([[0.8808, 0.6653],\n",
" [0.2138, 0.9389],\n",
" [0.1377, 0.1948]])\n"
]
}
],
"source": [
"# transpose of a matrix\n",
"mat = torch.rand([2,3])\n",
"print(mat)\n",
"mat_t = torch.t(mat)\n",
"print(mat_t)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([-2., -4., 8.])\n",
"tensor([2., 4., 8.])\n",
"tensor([12., 14., 18.])\n",
"random tensor([[1.7150, 2.9152],\n",
" [2.2219, 2.4114]])\n",
"clamp tensor([[1.7150, 2.0000],\n",
" [2.0000, 2.0000]])\n",
"tensor([1., 2., 4.])\n"
]
}
],
"source": [
"# Basic Math operations\n",
"\n",
"a = torch.Tensor([-2, -4, 8])\n",
"abs_a = torch.abs(a) \n",
"print(a)\n",
"print(abs_a)\n",
"\n",
"# Add x, y and scalar 10 to all elements\n",
"add_a = torch.add(abs_a, 10)\n",
"print(add_a)\n",
"\n",
"# Clamp the value of a Tensor\n",
"rand_t = torch.rand(2,2)*3\n",
"clam_t = torch.clamp(rand_t, min=0.5, max=2)\n",
"print(\"random \", rand_t)\n",
"print(\"clamp \", clam_t)\n",
"\n",
"# Element-wise divide\n",
"ele_div = torch.div(abs_a, 2)\n",
"print(ele_div)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor(8.)\n",
"tensor(2.)\n",
"tensor([3., 7.])\n",
"tensor([5., 5.])\n"
]
}
],
"source": [
"# To sum all elements of a tensor\n",
"t = torch.Tensor([[2,2],[2,2]])\n",
"sum_t = torch.sum(t) # gives back a scalar\n",
"print(sum_t)\n",
"mean_t = torch.mean(t)\n",
"print(mean_t)\n",
"\n",
"# To sum over all columns\n",
"t = torch.Tensor([[1,4],[2,3]])\n",
"sum1_t = torch.sum(t, dim=0) \n",
"print(sum1_t)\n",
"\n",
"# To sum over all rows\n",
"sum2_t = torch.sum(t, dim=1) \n",
"print(sum2_t)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[ True, True],\n",
" [False, False]])\n"
]
}
],
"source": [
"### Comparison\n",
"a = torch.Tensor([[1,2],[2,2]])\n",
"b = torch.Tensor([[1,2],[3,3]])\n",
" \n",
"\n",
"# Element-wise comparison\n",
"comp = torch.eq(a, b)\n",
"print(comp)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# dot product of two tensors\n",
"a = torch.Tensor([1,3])\n",
"b = torch.Tensor([2,4])\n",
"dot_product = torch.dot(a,b)\n",
"print(dot_product)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[-1.0316, -1.1872, 0.7581, 0.4848],\n",
" [-0.0574, 1.5447, -0.8743, -1.4751]])\n",
"tensor([-0.2977, 0.6603, 2.6992, -0.8614])\n",
"tensor([ 1.1517, -0.0523])\n"
]
}
],
"source": [
"# matrix vector multiplication\n",
"mat = torch.randn(2, 4)\n",
"vec = torch.randn(4)\n",
"res_mv = torch.mv(mat, vec)\n",
"print(mat)\n",
"print(vec)\n",
"print(res_mv)"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[-0.8117, 3.1222, 1.1493, 1.1621],\n",
" [ 0.6321, -0.2488, -3.8010, -2.7912]])\n"
]
}
],
"source": [
"# matrix matrix multiplication\n",
"mat1 = torch.randn(2, 3)\n",
"mat2 = torch.randn(3, 4)\n",
"res_mm = torch.mm(mat1, mat2)\n",
"print(res_mm)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Outer product of 2 vectors\n",
"v1 = torch.arange(1, 4) # size 3\n",
"v2 = torch.arange(1, 3) # size 2\n",
"print(v1,v2)\n",
"r = torch.ger(v1, v2) # 3x2\n",
"print(r)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}