Part 3: Convolutional Neural Networks and Image Classification (40

points)

This part of the assignment is designed to assess your knowledge and coding skill with

TensorFlow as well as hands-on experience with training Convolutional Neural Network (CNN).

The dataset we use for this part is the STL10 dataset² which consists of 5,000 training images

of airplane, bird, car, cat, deer, dog, horse, monkey, ship, truck; each of which has 500 images.

You are provided with a base code to download the dataset and train a default model.

Question 3.1 Observe the Learning Curve (4 points)

Run the provided cells to train the default model and observe the learning curve. Then, report

your observation (i.e., did the model learn well? If not, what is the problem? What would you do

to improve it?).

Note: for questions 3.2 to 3.9, you must write your own code to implement the model in a way

that makes it easy for you to experiment with different architectures and parameters. The goal is

to be able to pass the parameters to initialize a new instance of YourModel to build different

network architectures with different parameters. You can investigate the code of the class

Base Image Classifier before writing your own model.

Question 3.2 Define Your New CNN (4 points)

Write the code of the YourModel class. Note that this class will be inherited from the

BaseImageClassifier class. You'll only need to re-write the code for the build_cnn

method in the YourModel class.

Question 3.3 Experiment with Skip Connection (6 points)

Once writing your own model, you need to compare two cases: (i) using the skip connection,

and (ii) not using the skip connection. You should set the instance variable use_skip to either

True or False. For your runs, report which case is better and if you confront overfitting in

training.

Question 3.4 Tune Hyperparameters with Grid Search (4 points)

Now, let us tune the num_blocks € {2, 3, 4}, use_skip € {True, False}, and learning_rate €

{0.001, 0.0001). Write your code for this tuning and report the result of the best model on the

testing set. Note that you need to show your code for tuning and evaluating on the test set to

earn the full marks. During tuning, you can set the instance variable verbose of your model to

False for not showing the training details of each epoch.

Question 3.5 Apply Data Augmentation (4 points)

We now try to apply data augmentation to improve the performance. Extend the code of the

class YourMedel so that if the attribute in avan

en is set to True we apply the data/nearn the full marks. During tuning, you can set the instance variable verbose of your model to

False for not showing the training details of each epoch.

Question 3.5 Apply Data Augmentation (4 points)

We now try to apply data augmentation to improve the performance. Extend the code of the

class YourModel so that if the attribute is_augmentation is set to True, we apply the data

augmentation. Also, you need to incorporate early stopping to your training process.

Specifically, you need to early stop the training if the valid accuracy cannot increase in three

consecutive epochs.

Hint that you can rewrite the code of the fit method to apply the data augmentation. In addition,

2 https://cs.stanford.edu/-acoates/stl10/

you can copy the code of build_cnn method above to reuse here.

Question 3.6 Observe Model Performance with Data Augmentation (4 points)

Leverage your best model with the data augmentation and try to observe the difference in

performance between using data augmentation and not using it. Write a report of your

observation.

Question 3.7 Explore Data Mix-up Technique (4 points)

In this question you will explore Data Mix-up Technique for improving generalization ability. Data

mix-up is another super-simple technique used to boost the generalization ability of deep

learning models. You need to incorporate data mix-up technique to the above deep learning

model and experiment its performance. There are some papers and documents for data mix-up

which you can refer to:/nQuestion 3.6 Observe Model Performance with Data Augmentation (4 points)

Leverage your best model with the data augmentation and try to observe the difference in

performance between using data augmentation and not using it. Write a report of your

observation.

Question 3.7 Explore Data Mix-up Technique (4 points)

In this question you will explore Data Mix-up Technique for improving generalization ability. Data

mix-up is another super-simple technique used to boost the generalization ability of deep

learning models. You need to incorporate data mix-up technique to the above deep learning

model and experiment its performance. There are some papers and documents for data mix-up

which you can refer to:

• Main paper for data mix-up: https://openreview.net/pdf?id=r1Ddp1-Rb

• An article on data mix-up: https://www.inference.vc/mixup-data-dependent-data-

augmentation/

You need to extend your model developed above, train a model using data mix-up, and write

your observations and comments about the result.

Question 3.8 Attack Your Best Model (5 points)

Attack your best obtained model with PGD, MIM, and FGSM attacks with = 0.0313, k = 20, n =

0.002 on the testing set. Write the code for the attacks and report the robust accuracies. Also

choose a random set of 20 clean images in the testing set and visualize the original and

attacked images.

Question 3.9 Train a Robust Model (5 points)

Train a robust model using adversarial training with PGD € = 0.0313, k = 10, n = 0.002. Write

the code for the adversarial training and report the robust accuracies. After finishing the training,

you need to store your best robust model in the folder ./models and load the model to

evaluate the robust accuracies for PGD, MIM, and FGSM attacks with € = 0.0313, k = 20, n =

0.002 on the testing set.

Note: for questions 3.8 and 3.9, you should not use third-party libraries such as CleverHans.

Question 3.10 (bonus question) (5 points)

Sharpness-Aware Minimization (SAM) is recent technique to improve generalization ability of

DNNS. Read the SAM paper³ and try applying this technique to your best model and report the

results. For the purpose of implementing SAM, we can freely add any more cells and extensions

to the A2_S2_2023.py file.