/n 1 Introduction 1.1 Summary Computational MACHINE LEARNING Machine Learning Project In this assignment you will design and create an end-to-end machine learning system for a real-world problem. This assignment is designed for you to apply and practice skills of critical analysis and evaluation to circumstances similar to those found in real-world problems. In this assignment you will: • Design and Create an end-to-end machine learning system • Apply multiple algorithms to a real-world machine learning problem • Analyse and Evaluate the output of the algorithms • Research into extending techniques that are taught in class Provide an ultimate judgement of the final trained model(s) that you would use in a real-world setting This assignment has the following deliverables: 1. A report (of no more than 5 pages, plus up to 2 pages for appendices) critically analysing your approach and ultimate judgement. 2. An independent evaluation of your model and ultimate judgement (to be included in the report). 3. Your Python scripts, Jupyter notebooks, and software used to build your learning system and produce the models and results. 2 Task Using machine learning in real-world settings involves a more than just running a data set through a particular algorithm. In this assignment, you will design, analyse and evaluate a complete machine learning system. The key aspect of this assignment is the design, analysis, and evaluation of your methodology, investi- gation, and results. This assignment focuses on both the accuracy of your model, and your understanding of your approach and model. For this assignment you have a choice of your project. You may select this project from the list in Section 3, or you may negotiate a project with the course co-ordinator. Regardless of the problem you choose, you must conduct the following tasks: 1. You need to come up with an approach, where each element of the system is justi ed using data analysis, performance analysis and/or knowledge from relevant literature. 2. Investigate various Machine Learning solutions to the problem 3. Make an ultimate judgement 4. Evaluate your ultimate judgement against independent testing data 5. Produce a report of your design, investigation, evaluation and findings 2.1 Investigation Your investigation will require you to design, use, analyse and evaluate an end-to-end machine learning system. You should consider a variety of techniques that have been discussed in class, and techniques you have researched. Your end-to-end system may consist of elements such as: • • • A well justified evaluation framework. Pre-processing the data set to make it suitable for providing to various machine learning algorithms. Carefully selected and justified baseline model(s). Hyper-parameter setting and tuning to refine the model. Evaluating the trained models, analysing and interpreting the results. Each project features many of these above aspects. Each project also has unique aspects which cover a sample of issues from across machine learning. Additionally, each project has unique mandatory requirement(s), detailed for each project. The details of each project are listed in Section 3. The details in this spec are the minimum requirements. A thorough investigation must consider more that the minimum to receive high grades. 2.2 Ultimate Judgement You must make an ultimate judgement of the "best" model that you would use and recommend for your particular project. It is up to you to determine the criteria by which you evaluate your model and determine what is means to be "the best model". 2.3 Independent Evaluation of your Ultimate Judgement You need to conduct an independent evaluation of your ultimate judgement, using data collected completly outside of the scope of your original training and evaluation. This evaluation simulates how your ultimate judgement would perform if it were deployed in a real-world setting, where you are unable to re-train and adjust the model. 2.4 Approach, Critical Analysis & Report You must compile a report analysing the approach you have taken in your investigation. Your report: • Must be no longer that 5 pages of text • May contain an additional 2 pages for appendices • • Use a single-column layout with no less than size 11pt font The appendices may only contain citations, figures, diagrams, or data tables that provide evidence to support the statements in your report. Any over length content, or content outside of these requirements will not be marked. For example, if you report is too long, ONLY the first 5 pages pages of text will be read and marked. In this report you should analyse elements such as: • Machine learning algorithms that you considered • Why you selected these approaches • • Evaluations of the performance of trained model(s) Your ultimate judgement with supporting analysis and evidence This will allow us to understand your rationale. We encourage you to explore this problem and not just focus on maximising a single performance metric. By the end of your report, we should be convinced that of your ultimate judgement and that you have considered all reasonable aspects in investigating your chosen problem. The key aspect of this assignment isn't your code or model, but the thought process behind your work. Remember that good analysis provides factual statements, evidence and justifications for conclusions that you draw. A statements such as: "I did <xyz> because I elt that it was good" is not analysis. This is an unjustified opinion. Instead, you should aim for statements such as: "I did <xyz> because it is more efficient. It is more efficient because ..." 3 Projects Project Classify Images of Road Traffic Signs This project is to train a model to classify images of European road traffic signs. You will be using a modified version of the Belgium Traffic Sign Classification Benchmark. These are images of road traffic signs taken from real-world vehicles. Note, this dataset, along with it's sister German TSC dataset, appear in many different forms on various research and ML online resources. The data set for you to use in this assignment has been specifically prepared for you, and is provided on Canvas. The dataset consists of 28x28 gray-scale images and you are expected to use the dataset to perform two tasks: • Classify images according to sign-shape, such as diamond, hex, rectangle, round, triangle. • Classify images according to sign-type, such as stop, speed, warning, parking, etc. The correct classification of the images is given by the image sub-directories. Images are first sub-divided by their shape, and then by their sign type. You should also note that some sign types have different individual signs. For example, the speed-sign type has examples of signs of speeds from 10 - 70 mph. You are not required to further sub-divide these signs, but consider all of the different signs as being of the sample type. Your tasks is to investigate classifying the signs using both categories. REQUIREMENTS • • • • . You must investigate at least one supervised machine learning algorithms or each of the two categories (Tasks). That is, you must build at least one model capable of classified the shape of the sign, and at least one model capable of classifying the type of the sign. You are not required to use separate type(s) of machine learning algorithms, however, a thorough insti- gations should consider different types of algorithms. You are required to ully train your own algorithms. You may not use pre-trained systems. You may NOT augment this data set with additional data. Your final report must conduct an analysis and comparison between classifying the two categories. INDEPENDENT EVALUATION • • Your independent evaluation should consist of classifying images of traffic signs that you have collected. You will need to either take your own digital photographs of traffic signs, and/or source suitable signs from internet resources. You will need to process these images so they may be used with your trained algorithms. As part of your evaluation, you should discuss challenges you face in combining this independent data and your models. 4.2 Marking Rubric A detailed rubric is attached on canvas. In summary: • • Approach 60%; • - Ultimate Judgment & Analysis (Independent Evaluation) 20%; Report Presentation 20%. 4.3 Submission Instructions You must submit all the relevant material as listed below via Canvas. 1. A report (of no more than 5 pages, plus up to 2 pages for appendices) critically analysing your approach and ultimate judgement 2. An independent evaluation of your model and ultimate judgement (included in the report). 3. Your Python scripts, Jupyter notebooks, and software used to build your learning system and produce the models and results. The submission portal on canvas consists of two sub-pages. First page for report submission, the second page for code submission. More information is provided on Canvas. Include only source code in a zip file containing your name. We strongly recommend you to attach a README file with instructions on how to run your application. Make sure that your assignment can run only with the code included in your zip file!