Deep Learning (Subject) / Deep Learning Basics (Lesson)

This lesson was created by Taggotty.

Learn lesson

back  |  next 1 / 1

• What is Bias? Bias is simply a constant value (vector) that is added to the product of inputs and weigths. - Allows the activation function to be shifted left or right - Allows better fit to the data - Typicalli initialised to be zero - Bias are independent of the output from previous layers
• What are the properties of the sigmoid activation function? σ(x)=1/(1+e^-x) σ'(x)=σ(x)(1 - σ(x)) - Non-linear so can capture more complex patterns - Output values are bounded so don't get too large (still can get large) - can suffer from vaishing gradient
• What does the loss function do? - measures the difference between what we have predicted ȳ, whith the model and what it should have predicted y - usee this information to update the model - in order to minimize the cost we prefer differetiable functions which have simpler optimization algorithms
• What is the Mean Squared Error (MSE)? - usually used for regression L(ȳ, y)= 1/n sum(i, (ȳ - y)2)
• What is Cross Entropy? - Used for classification, the model predicts the probability of observation x having a particular label - Cross Entropy measure the distance between two probabilities L(po, p) = sum(c, pc log(poc))
• What is an Activation function? - A function applied to neurons in a layer during prediction - Properties:     - The function must be continuous and infinite in domain     - Good activation functions are monotonic, never changing direction     - Good activation functions are nonlinear     - Good activation functions (and their derivatives) should be efficiently computable
• What are the properties of the linear activation function? - A straight line - values can get large - does not capture complex patterns
• What are the core properties of the hyperbolic Tangent Activation Function? tanh(x) = ( exp(x)-exp(-x) ) / (exp(x) + exp(-x)) tanh'(x) = 1- tanh(x)2 - Non-linear so can capture more complex patterns - Output values are bounded so cannot get quite large - Negative inputs mapped to negative outputs - Zero inputs mapped to zero output - Can also suffer from vanishing gradient
• What are the core properties of the Rectified Linear Unit activation function? ReLU(x)=max(0,x) ReLU'(x)={ 0, for x < 0 and 1, for x >= 0 - Non-linear so can capture more complex patterns - Positive values can get very large - Simple derivative - Does not allow for negatives - Dying ReLU gradient goes to zero and weight is not updated
• What are the core properties of the Leaky Rectified Linear Unit (LReLU)? LReLU(x)= {ax for x < 0 and x for x>=0 LReLU'(x)= {a for x < 0 and 1 for x >=0 - Non-linear so can capture more complex patterns - Values can get very large - Simple derivative - Attempts to solve for dying ReLU issues
• Advertisement
• What are the core properties of the softmax activation function? S(xi)=exp(x.i) / sum(j, exp(x.j)) S'(xi)=S.i(1-S.j), for i=j and -S.i*S.j, for i !=j - Models probability distribution (All values sum up to 1) - Typically only used in the output layer - Softmax is a vector function so the derivative is calculated via partials
• Specify how to solve a Regression problem? -> Predicting a single numerical value - Final Activation function: Linear - Loss function - Mean Squared Error
• Specify how to predict a binary outcome? - data is or isn't a class - Sigmoid Activation - Binary Cross Entropy
• What is the Binary Cross Entropy? Our model predicts a model distribution of {p, 1-p} we compare this to the true distribution {y, 1-y} BCE =  -(ylog(ȳ)+(1-y)log(1-ȳ))
• Specify how to predict a single label from multiple classes? - Softmax activation - Cross Entropy
• Specify how to predict multiple labels from multiple classes - If there are multiple labels in your data:      - The final layer will have one neuron for each classes     - This neuron will return a value between 0 and 1     - This can be inferred as a probability - Sigmoid - Binary Cross Entropy
• What is Batch Gradient Descent? - Computes the gradient of the cos function with respect to the entire training dataset Pro + theoretical analysis of weights and convergence rates are easy to understand Con - Performs many redundant computations - Slow and potentially intractable
• What is Stochastic Gradient Descent (SDG)? - Performs a parameter update on each training example Pro: + Quicker leaning than Batch Gradient Descent + Easy to update weights if new training samples are acquired Con: - As we frequently update weights, cost function fluctuates heavily
• What is momentum? - Is used to accelerate convergence of gradient descent - It takes the gradient of the previous time steps when updating weights typical value: 0.9 vt=γvt-1-η∇J(θ;x;y) θ=θ-vt
• What is mini-batch gradient descent? - performs an update for every mini-batch of n training examples  Θ(t+1)=Θ(t)-η∇J(Θ(t);x(i:i+n);y(i:i+n)) Pro: + Reduction in variance means more stable convergence + Learning Spead + Good approximation of location of actual minima Con - Total loss not accumulated
• What is Nesterov Accelerated Gradien (NAG)? - Momentum is not always goot, it is often useful to know when to slow down - We calculate the gradient not with respect to the current step but with respect to the future step vt=γvt-η∇J(θ-γvt-1;x;y) θ=θ-vt
• What is Adagrad? - Adaptive Gradient Algorithm - Optimisation Algorithm - Expensive to tune learning (and moment) rates - Adagrad is an adaptive leaning rate method     - We perform larger updates for infrequent parameters and smaller updates for frequent              parameters Gt is the sum of squares of past gradients with respect to all parameters θ gt = ∇J(θ;x,y) ε is a smoothing term to avoid dividing by zero (≈1e-8) θ = θ - (η / sqrt(Gt + ε)) gt
• What is Adadelta? - Optimisation Algorithm - Restricts the window size of accumulated past gradients     - The sum of gradients is recursively defined as a decaying average of all past squared                  gradients - Learning rate is set dynamically as the ratio of the running average of the previous time steps to the current gradient θt+1 = θt + Δθt RMS = Root Mean Squared error Δθ = - (RMS[Δθ]t-1 / RMS[gt]) gt
• What is RMSProp? - Optimisation Algorithm - Root Mean Square Propagation - Uses a moving average of the squared gradient to adjust the learning rate     - It divides the learning rate by an exponentially decying average of squared gradients θt+1 = θt - (η / (Sqrt(1 - γ)gt-1*gt-1 + γ * gt + ε)) gt γ typically set to 0.9 η typically set to 0.001
• What is Adam? - Adaptive Moment Estimation - Claculates adaptive learning rate from estimates of first and second moments of the gradients - Adam is computationally efficient and has very little memory requirment - Adam can be viewed as a combination of Adagrad, which works well on sparse gradients and RMSprop which works well in online and nostationary settings. - The Adam algorithm first updates the exponential moving averages of the gradient (mt) and the squared gradient (vt) which are the estimates of the first and second moment - Hyper parameters β1, β2 ∈ [0, 1) control the exponential decay rates of these moving averages as shown below - mt = β1mt-1 + (1 - β1)gt - mt = β2mt-1 + (1 - β2)gt * gt - Moving averages are initialized as 0 leading to moment estimates that are biased around 0 especially during the initial timesteps - This inisialization bias can be easily counteracted resulting in bias-corrected estimates - mroof_t= mt / (1 - β1t) - vroof_t= mt / (1 - β2t) - θt+1 = θt - (ηmroof_t / sqrt(vroof_t+ε)) - adam is a highly popular approach: works well in practice, achieves fast convergence does not suffer from errors faced by other approaches
• What are weight penalties used for? - Addition of a weight penalty term to the cost funtion -> Large weights make networks unstable - Aim of penalty term is to encourage the model to map the input to the outputs of the training dataset in such a way that the weights of the model are kept small
• What is the L1 Norm? - The sum of the absolute values of the weights - L1 encourages weights to be zero if bossible - Resulting in more sparse weights
• What is the L2 Norm? - The sum of the squared values of the weights - Penalizes larger weights αΩ(θ) = ||θ||22= sqrt(sum(n, |θn|2))
• What is Data Augmentation? - The best way to make a machine learning model generalize better is to train it on more data - One way to get around this problem is to crate fake data and add it to the training set - One must be careful not to apply transformations that would change the correct class
• What does Training with Noise do? - a form of augmentation - adding noise means that the network is less able to memorise training samples     - Input data is changing all of the time - Results in smaller network weights and a more robust network that has lower generalisetion error - Typically to add noise to input data     - white Gaussian noise with mean of 0 and a standard deviation of 1     - Generated as needed using pseudorandom number generator     - We can also add noise activations, to weights, to the gradients and to the outputs
• Advertisement
• What does Dropout do? - Method for training an ensemble of slightly different networks and averaging them - The ensemble of subnetworks is formed by randomly removing nonoutput units during training - Random Outputs at layers are ignored ("dropped out") - Each neuron has p% dropout - for each mini-batch we resample the dropot neurons
• What is Early Stopping? - Neural networks can get worse if you train them too much -> When training a large network there will be a point during training when the model will stop learning the signal and start learning the signal and start learning the statistical noise in the training dataset - training a neural network long enough to learn the mapping, but not so long that it overfits the training data - Trivial to monitor performance on a holdout validation dataset can be monitored during training  - Stop training when generalization error increases
• What does Regularisation do generally? - Processes which adds stability to a learning algorithm by making it less sensitive to the training data and processes - When using regularizetion, we try to reduce the gerealisation error, we try to reduce the generalization error, this may lead to incrase the training error, which is okay, because what we care about is how well the model generalizes - With regularisation, we try to bring back the very complex model that suffers from overfitting to a good model by increasing bias and reducing variance

back  |  next 1 / 1