CSE 635: NLP and Text Mining Spring 201 Group Project Description and Requirements Overview The goal of this semester-long project is to provide hands-on experience designing, implementing, evaluating, and demonstrating

a complete web mining/ text mining/ social media mining solution based on a combination of natural language processing (NLP), information retrieval (IR) and machine learning (ML) techniques. You are provided a choice of four topics that broadly fall into the AI for Social Impact area. Each project (except one) will have a standard dataset and ground truth, enabling quantitative evaluation. Many of these are from past or ongoing challenges and have been attempted by other teams. We encourage you to use any available online tools or platforms to develop your solution. You should strive to produce results in the top 10% of any previously published results on the same dataset. While there is a quantitative evaluation component in the last stage, you must develop a live demo system. This may involve developing a user interface so you can demonstrate the system. This project will satisfy the MS project requirements specified by the CSE department. While the problem definition and evaluation dataset have been fixed, there is ample room for creativity on your part in further enhancement of the solution and implementation. Be creative, and most importantly, pace yourself properly during the semester. Your project is divided into three phases, which are described in more detail later on in this document: Phase 1: Submission of project proposal and in-person presentation of your proposal. This includes a comprehensive literature review on your selected topic, a necessary step before you begin designing your system! Phase 2: Interim report describing the evaluation of the baseline system. Phase 3: Final submission of technical paper, code, and in-class presentation of your end-to-end system. Project Option 1 - LLMs in the Health Sciences Task Overview: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) Big Picture Researchers have initiated a challenge centered around Clinical Trial Reports (CTRs) related to breast cancer treatments to enhance how artificial intelligence interprets and utilizes medical reports. These reports are critical for medical professionals to determine the safety and efficacy of new treatments. Still, they are voluminous and complex, making it challenging for individuals to review each one thoroughly. The challenge involves the AI analyzing summaries of these reports, focusing on key aspects such as eligibility criteria, treatment specifics, trial outcomes, and observed adverse effects. Researchers crafted statements about these summaries, requiring the AI to assess whether these statements are true or false or if there's insufficient information to decide. To further test the Al's capabilities, these statements were intentionally altered by modifying numbers, changing words, or restructuring sentences. The ultimate goal is to refine the Al's consistency in understanding and ability to logically deduce information, thereby supporting medical professionals in making informed decisions about patient care. This underscores the potential of AI to significantly contribute to medical science, particularly in the realm of personalized medicine, by streamlining the interpretation and application of extensive clinical trial data. This task is based on a collection of breast cancer CTRs (extracted from https://clinicaltrials.gov/ct2/home), statements, explanations, and labels annotated by domain expert annotators. Task: Textual Entailment For the task, we have CTRs into 4 sections: • Eligibility criteria - A set of conditions for patients to be allowed to take part in the clinical trial Intervention - Information concerning the treatment type, dosage, frequency, and duration being studied. Results - Number of participants in the trial, outcome measures, units, and the results. ● Adverse events - These are signs and symptoms observed in patients during the clinical trial. The annotated statements are sentences with an average length of 19.5 tokens, that make some type of claim about the information contained in one of the sections in the CTR premise. The statements may make claims about a single CTR or compare 2 CTRs. The task is to determine the inference relation (entailment vs contradiction) between CTR - statement pairs. The training set we provide is identical to the training set used in our previous task, however, we have performed a variety of interventions on the test set and development set statements, either preserving or inversing the entailment relations. We will not disclose the technical details adopted to perform the interventions to guarantee fair competition and in the interest of encouraging approaches that are robust and not simply designed to tackle these interventions. The technical details will be made publicly available after the evaluation phase, and in our task description paper. Intervention targets • Numerical - LLMs still struggle to consistently apply numerical and quantitative reasoning. As NLI4CT requires this type of inference, we will specifically target the models' numerical and quantitative reasoning abilities. • Vocabulary and syntax - Acronyms and aliases are significantly more prevalent in clinical texts than general domain texts, and disrupt the performance of clinical NLI models. Additionally, models may experience shortcut learning, relying on syntactic patterns for inference. We target these concepts and patterns with an intervention. • Semantics - LLMs struggle with complex reasoning tasks when applied to longer premise-hypothesis pairs. We also intervene on the statements to exploit this. - Notes The specific type of intervention performed on a statement will not be available at test or training time. Dataset We will provide you with the dataset for training (which will be available in Github). Evaluation The evaluation of performance on this task will involve several steps. First, we will assess performance on the original NLI4CT statements without any interventions. This assessment will be based on Macro F1-score. Then, we will measure the Faithfulness and Consistency. (More details will be provided in the Github Repository) Bonus Points Completing all Objectives, writing a research paper, and submitting to at least a workshop/conference. Project Option 2 - LLMs in the Social Sciences Task: Multilingual Detection Of Persuasion Techniques In Memes Big Picture Imagine you're in a world where images paired with catchy text, known as memes, are not just for laughs but can also sway people's opinions and spread misinformation. These memes can be powerful on social media, reaching and influencing countless users with simple yet impactful messages. Some memes use sneaky tactics to persuade or mislead, such as making things seem simpler than they are, calling people names to discredit them, or using emotional appeals to bypass rational thinking. Technical Description We refer to propaganda whenever information is purposefully shaped to foster a predetermined agenda. Propaganda uses psychological and rhetorical techniques to reach its purpose. Such techniques include the use of logical fallacies and appealing to the emotions of the audience. Logical fallacies are usually hard to spot since the argumentation, at first sight, might seem correct and objective. However, a careful analysis shows that the conclusion cannot be drawn from the premise without the misuse of logical rules. Another set of techniques makes use of emotional language to induce the audience to agree with the speaker only on the basis of the emotional bond that is being created, provoking the suspension of any rational analysis of the argumentation. Memes consist of an image superimposed with text. The role of the image in a deceptive meme is either to reinforce/complement a technique in the text or to convey itself one or more persuasion techniques. Tasks Subtask 1 - Given only the "textual content” of a meme, identify which of the 20 persuasion techniques, organized in a hierarchy, it uses. If the ancestor node of a technique is selected, only a partial reward is given. This is a hierarchical multilabel classification problem. Subtask 2 - Given a meme, identify which of the 22 persuasion techniques, organized in a hierarchy, are used both in the textual and in the visual content of the meme (multimodal task). If the ancestor node of a technique is selected, only a partial reward will be given. This is a hierarchical multilabel classification problem. You can find info on the hierarchy below. Ad Hominem Persuasion Ethos Pathos Justification Logos Reasoning Distraction Simplification Name Calling Bandwagon Appeal to Emotion (Visual) Slogans Repetition Straw Man Causal Oversimplification Appeal to Doubt Exaggeration Authority Intentional Vagueness Red Herring Black & White Fallacy Smears Glittering Generalities Loaded Language Whataboutism Thought Terminating Cliché Reductio ad Hitlerum Flag Waving Appeal to Fear Transfer Color Legend Black = first level Red = second Level Blue third level Green fourth level White = techniques Hierarchy of the techniques for Subtask 2a (in Subtask 1 "Transfer" and "Appeal to Strong emotion" are not present). The hierarchy is also inspired by this document Dataset We will provide the dataset, more information will be on Github Evaluation Subtask 1 Subtask 1 is a hierarchical multilabel classification problem. Taking the figure above with the hierarchy as an example, any node of the DAG can be a predicted label. The gold label is always a leaf node of the DAG. If the prediction is the correct label, We use hierarchical-F1 as the official evaluation measure. Subtask 2 Subtask 2 is a hierarchical multilabel classification problem. We use hierarchical-F1 as the official evaluation measure. Bonus Points Completing all Objectives, writing a research paper, and submitting to at least a workshop/conference.