Basics of Linear Algebra for DL

In this section, we will review the basic mathematical concepts and definitions that are essential for comprehensive understanding of deep learning.

Introduction

Let's say we have a list of numbers(x), where x = [10, 20, 30, 40]. If we want to access a certain element of this list, then we have to use the element's location within the list.

Remember that locations (indices) for elements in lists in python starts from 0. Therefore, 10 is the 0^th-element of x and 30 is the 2^nd-element of the list i.e. x[0] = 10 and x[2]=30.

Since our list had only one dimension, we used a single index to refer to the location of a certain element within our list.

Now let's move forwards. Another name for a group of numbers is an Array or a Tensor . These terms are important and you will hear frequently as you go through your journey learning DL. Our previous example had 1-dimension and therefore is a 1d-array. If we have a grid/matrix of numbers (i.e. has rows and columns), therefore our group of numbers will have 2-dimensions and is called a 2d-array.

$\begin{bmatrix} 100 & 110 &120 \\ 200 & 210 & 220\\ 300 & 310 &320 \end{bmatrix}$

To access a certain element within this matrix, we have to give the its location i.e. number of row and number of column for this element. For example: x[1,2] = 110 and x[2,3]=220.

In a similar way, if we have a group made of only one number x = [5] , then our array is a 1d-array.

Therefore, a Tensor is a n-d array of numbers.

In addition to arrays or tensors, other terms are often used interchangeably between computer science and mathematics. Conventionally, it is easier to use the n-d array format to describe any group of numbers. These terms include "Scalar" referring to 0-d tensor/array, "Vector" referring to 1-d tensor/array and "Matrix" referring to 2-d tensor/array.

Hint

Each pixel within a within greyscale DICOM contains a single number. Therefore, pixel data of greyscale DICOM image is a 2-d array.

On the other hand, each pixel in colored images and colored DICOM contains usually 3 numbers (one for each of the color channels used = Red, Green, Blue) i.e. there is an extra dimension (depth). Therefore , pixel data for these images is a 3-d array.

Tensor Properties

There are certain tensor properties that are required to be fully understood before we go further.

Operations

Let's dig deeper into the basic most common mathematical operations related to scalars, vectors, matrices and tensors.

Let's start with understanding some common abbreviations:

A = matrix
a = vector
m = number of columns
n = number of rows

2d-0d Operations

Addition, subtraction, multiplication or division of a 2d-array by a 0d-array results in the desired operation applied to each element of the 2d-array.

For example :

$\begin{bmatrix} 100 & 110 &120 \\ 200 & 210 & 220\\ 300 & 310 &320 \end{bmatrix} * 2 = \begin{bmatrix} 200 & 220 & 240 \\ 400 & 420 & 440\\ 600 & 620 & 640 \end{bmatrix}$

2d-1d Multiplication

Multiplying a 2d-array by a 1d-array results in the following: - multiplication of each row of the 2d-array by the column of the 1d-array. - addition of the results of these multiplications into a single numbers. - output is a new 1d-array with same number of rows as the input 2d-array.

The number columns in the 2d-array must be equal to the number of rows in the 1d-array following the formula:

2d-2d Addition and Subtraction

2d-array-2d-array addition and subtraction results in straight forwards operations on the individual elements of the 2d-arrays.

Both 2d-arrays must have the same dimensions.

For example:

$\begin{bmatrix} 100 & 110 &120 \\ 200 & 210 & 220\\ 300 & 310 &320 \end{bmatrix} + \begin{bmatrix} 1 & 2 &3 \\ 4 & 5 & 6\\ 7 & 8 &9 \end{bmatrix} = \begin{bmatrix} 101 & 112 &123 \\ 204 & 215 & 226\\ 307 & 318 &329 \end{bmatrix}$

2d-2d Multiplication

2d-array-2d-array multiplication can be thought of as repeats of 2d-1d Multiplications done side by side following this formula :

Identity Matrix

An identity matrix is a certain 2d-array that if multiplied by an input 2d-array will result in the same input 2d-array again. A unique characteristic is that this 2d-array has the number 'one' running diagonally and 'zero' elsewhere. The following are identity matrices:

$\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 &0 \\ 0 & 1 & 0\\ 0 & 0 &1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 &0 \\ 0 & 1 & 0 & 0\\ 0 & 0 &1& 0\\ 0 & 0 &0 & 1 \end{bmatrix}$

Inverse

The inverse of a number is that certain value that if multiplied by the original number the resultant will be One. In case of matrices, the inverse of a 2d-array is that 2d-array that when multiplied the result will be the identity matrix.

For example, the inverse of the 2d-array

$\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$

is

$\begin{bmatrix} -2 & 1 \\ .5 & -0.5 \end{bmatrix}$

Transponse

The transponse of a 2d-array is the 2d-array but rotated 45 degrees clockwise.

For example, the transponse of the 2d-array

$\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$

is

$\begin{bmatrix} 3 & 1 \\ 4 & 2 \end{bmatrix}$

Tutorial code for this section can be found on Google Colab here