In March of 2023, Goldman Sachs published a report, indicating that ~25% of the tasks in US and Europe can be automated using Al. However, as you can see in

this visualization, not all industries will be affected equally. According to the report, certain jobs, like office tasks, legal, architecture and social sciences have a potential for 30%+ automation, while positions like construction, installation and building maintenance are going to be largely unaffected. You can also see supporting evidence in the Facebook Research paper, which highlights Moravec's paradox, the thesis that the hardest problems in Al involve sensorimotor skills, not abstract thought or reasoning, which coincide with Goldman Sachs predictions. While both of these papers are very impressive, they also heavily influenced by the recent advances in Large Language Models (LLMs). For this final project I have prepared a collection of ~200K news articles (about 900 MB) articles on our favorite topics: Data Science, Machine Learning and Artificial Intelligence, and I want you to identify what industries and job lines are going to be most impacted by the Al over the next several years, based on the information you can disseminate from this text corpus. The objective of your final project is to identify what types of tasks and jobs are most likely to see the biggest impact from Al by extracting meaningful insights from unstructured text. Your goal is to provide actionable recommendations on what can be done with Al to automate the jobs and / or improve employee productivity. Please pay attention to the introduction of novel technologies and algorithms, such as Al for image generation and Conversational Al, as they represent the entire paradigm shift in adoption of Al technologies and data science in general. You can access the data by using one of the following methods: • Download data by following this think from your browser: https://storage.googleapis.com/msca-bdp-data- open/news final project/news final project. parquet • Use Pandas from anywhere (your laptop, Colab or any cloud) df_news_final_project = pd.read_parquet('https://storage.googleapis.com/msca-bdp- data-open/news_final_project/news_final_project.parquet, engine='pyarrow') o NLP GCP 11.1 Final Project Starter.ipynb Note: this is live data, so the layout and record counts in your dataframe will vary from the counts in the attached notebook To complete your assignment, I suggest considering the following steps: Clean-up the noise, by eliminating newlines, tabs, remnants of web crawls, and other irrelevant text • Discard irrelevant articles ● • Detect major topics ● Identify top candidates for Al integration - these can be related to any industry and yield positive or negative results (sentiment analysis). Suggest why certain types of jobs are more likely to be impacted by Al • Plot a timeline to illustrate how the sentiment is changing over time Identify new technologies and Al solutions that might be affecting the employment landscape o Plot a timeline to illustrate the introduction of some of these technologies • Demonstrate what companies, academic institutions and government entities can do to accelerate the development of these transformative capabilities • Leverage appropriate NLP techniques to identify organizations, people and locations, then apply targeted sentiment ● o What types of companies (based on the lines of business) are planning to invest in these technologies today or near future (success stories)? ▪ Create appropriate visualization to summarize your recommendations (i.e. word cloud chart or bubble chart) o What types of applications cannot currently be transformed by Al, based on today's state of technology (failures)? ■ Create appropriate visualization to summarize your recommendations (i.e. word cloud chart or bubble chart) Additional guidance: • Clean-up or sample data if you need to shorten processing times or reduce memory usage • Default sentiment will likely be wrong from any software package and will require some tweaking o Keyword / dictionary approach o Data annotation and development of custom classifier Building custom model on open-source data (i.e. Yelp) O ○ Fine-tuning Transformer Pipeline • You are encouraged to explore a combination several techniques to identify key topics: o Topic modeling (i.e. LDA using gensim or ktrain) or using BERTopic o Classification (hand-label several topics on a sample and then train classifier) Clustering (cluster topics around pre-selected keywords or word vectors) o Zero-shot (NLI) modeling O o Please ensure your PowerPoint presentation (in PPTX or PDF format) is submitted to the course module as-is (not zipped). Otherwise we are unable to use Canvas SpeedGrader. o The presentation should look professional - not a collection of screenshots from your analytical software • Roughly 8-12 pages is reasonable for this kind of project but there are no strict restrictions. o On your slides you will want to provide: ▪ Executive Summary ▪ Methodology and source data overview ▪ Actionable recommendations ▪ Apply text summarization algorithms where possible to synthesize your insights • Please submit your actual program codes (Jupyter notebooks) along with your PowerPoint o The slides should be self-sufficient and after reading them, there should not be any need to read the notebook (we are still asking you to provide the notebooks as a proof or work though). • The slides should clearly answer all the questions and the answers should be supported with the plots/tables/numbers produced in the notebook based on the actual data. • The slides should contain the RIGHT amount of supporting material for each question, putting too much supporting material is as bad as putting too little: too much - you would not be able to keep the audience attention and your presentation would be a mess, too little - your statements would not look convincing. o Everything should be clear, logical, well organized, as simple as possible. Use proper English grammar and run spell check. o All the plots should be of production quality and easily readable. Fuzzy plots, untitled plots, unreadable labels, overlapping labels are unacceptable. If you formatting somehow gets corrupted when you put your slides into Canvas (sometimes it happens), it is your responsibility to fix it. For example, try saving it in some other format like PDF, HTML. o Any statements you make should be supported by data. Only recommendations or goals of the project sections can contain elements not directly supported by the data ○ Please submit your actual program codes (i.e. Python Notebook) along with your PowerPoint – as a separate attachment ▪ Your presentation should be targeted toward business audience and must not contain any code snippets • You are welcome to use any software packages of your choice to complete the assignment Grading Rubric: Rubric Executive Summary with meaningful insights Visualize actionable recommendations Article clean-up and filtering Topic detection Sentiment analysis (explicitly customized) Sentiment over time analysis and visualization Entity (organizations and people) identification Targeted (entity) sentiment identification Total Points Points 20 10 10 20 10 10 10 10 100