2.3) Which of the following statements is/are true? For sigmoid activation function, as input to the function becomes larger and positive, the activation value approaches 1. Similarly, as it becomes very negative, the activation value approaches 0. For training MLPs, we do not use activation function as a step function, as in the flat regions there is no slope and hence no gradient to learn. For tanh activation function, as the value increases along the positive horizontal axis, the activation value tends to 1 and along the negative horizontal axis, it tends towards -1, with the centre value at 0.5. The ReLU activation function is a piecewise linear function with a flat region for negative input values and a linear region for positive input values.

3.4) Which of the following statements is/are true? Learning using optimizers such as AdaGrad or RMSProp is adaptive as the learning rate is changed based on a pre-defined schedule after every epoch. Since different dimension has different impact, adapting the learning rate per parameter could lead to good convergence solution. Since AdaGrad technique, adapts the learning rate based on the gradients, it could converge faster but it also suffers from an issue with the scaling of the learning rate, which impacts the learning process and could lead to sub-optimal solution. RMSProp technique is similar to the AdaGrad technique but it scales the learning rate using exponentially decaying average of squared gradients. 0 Adam optimizer always converge at better solution than the stochastic gradient descent optimizer.

2.1) Which of the following statement(s) is/are true? Perceptron network with just input and output layer converges for linearly separable cases only. Logistic regression model and perceptron network are the same model. Unlike Perceptron network, logistic regression model can converge for non-linear decision boundary cases. 0 By changing the activation function to logistic and by using gradient descent algorithm, Perceptron network can be made to address non-linear decision boundary cases.

2.7 Which of the following statements is/are true? At the output layer of a binary classification problem, we could either use sigmoid activation function with single output neuron or softmax function with two neurons, to produce similar results. For a classification application, that predicts whether a task is personal or official and which also predicts whether it is high or low priority, we could use two output neuron and apply sigmoid activation function on each output neuron. For a classification application, that predicts whether Team A would win, lose or draw the match, we could use three output neurons with softmax activation function applied at the output layer.

(2) A report in PDF format (8 marks) should have the following sections a. A description of how to handle the missing values in your code and report the results. (2.5 Marks) b. A description of the regression technique you used. (3 Marks) c. A description of the results to report. (2.5 Marks)

2.2) Which of the following statement(s) is/are true? By adding one or more layers to perceptron network with activation functions, non- linear separable cases can be handled. For a non-linearly separable dataset, if activation function applied on the hidden layer(s) of the Multi-layer Perceptron network is a linear function, the model will converge to an optimal solution. For a linearly separable dataset, applying non-linear activation function such as sigmoid or tanh on hidden layers of a MLP network, can converge to a good solution. All of the above

2.8 Which of the following points hold true for gradient descent? It is an iterative algorithm and every step it finds the gradient of the cost function with respect to the parameters to minimize the cost. For a pure convex function or a regular bowl-shaped cost function, it can converge to the optimal solution irrespective of the learning rate Since cost functions can be of any shape, it can only achieve local minima but not global minimum, provided the cost function is not convex. Normalizing the variables and bringing the magnitude of these variables to same scale, ensures faster convergence.

COMP 1002 Assignment 4 More Induction! Please read the following important notes: Your assignment must be completed on your own. Sharing solutions, looking for and/or copying solutions from unauthorized sources, and posting the assignment questions online are all forms of academic misconduct. Committing an act of academic misconduct will result in a grade of 0 and reported to the department head. • Only one problem set will be graded and it will be selected at random. Students who attempt to solve the problem but do not get a perfect score are allowed to submit a Reflection to obtain bonus marks on their Assignment. If you do not attempt to solve the selected problem set, then you cannot submit a reflection. • Please submit a file for each problem set and clearly state the problem set in the file's name. Acceptable file formats are PDF, JPG, or PNG. Do not include your name/id in your submission. • If your solutions are handwritten, please ensure your solution is legible. Double check both your image quality and handwriting. If we cannot read your solution, then you will get a mark of 0. • While writing your solutions, you will be graded on both your approach and correctness. A given problem may be solvable in a number of different ways - we will not grade you on efficiency. That being said, if a problem says to use a specific technique (i.e. Natural Deduction) then you must solve it using this technique./nWith this understanding please complete the following questions: 1. Theory A Kirby string is defined recursively as follows: i) (^o^) is a Kirby string. ii) (ToT) is a Kirby string. iii) if A is a Kirby string, then <A> is a Kirby string. That is concatenated with A concatenated with >. iv) if A, B, C are all Kirby strings, then A...B...C is a Kirby string. That is A concatenated with ... concatenated with B concatenated with ... concatenated with C. v) Nothing else is a Kirby string. a) Give a Kirby string that requires the use of rule iv, but such that K₁, K₂, K3 are all different Kirby strings. Explain how to derive your Kirby string using the rules given. b) Disprove the following statement: Every Kirby string has the o character in the centre of the Kirby string. c) Prove by structural induction that every Kirby string has an odd number of characters.

3.3) Which of the following statements is/are true? When applying momentum optimization, setting higher momentum co-efficient will always lead to faster convergence. Unlike in gradient descent, in momentum optimization the gradients at a step is dependent on the previous step. Unlike in stochastic gradient descent, in momentum optimization the path to convergence is faster but with high variance.

2.6 Which of the following statements is/are true? To perform regression using MLPs, non-linear activation functions might be required in the hidden layer but generally, no non-linear activation function is required for the output layer. ☐ If you would want to predict an age of a person, we can use ReLU activation function in the output layer.→→ For a regression problem, in which the output value is always within a range of values, we could use sigmoid or tanh function and scale the values to ensure it is in the bounded range.