r programming Homework Help

Project 1 : Movie Recommendation Engine 11/9/2020 Getting Started Importing the Data For this recommender the first thing we need to do is make sure that we have all the data we need. There are two datasets in this project: movies and ratings. Movies contains a movie id, the title of the movie, and the genre in | delineated lists. Ratings contains the user id making the rating, the movie id they are rating, and the timestamp of the rating. This dataset is a version of movieLens converted to a dataFrame. MovieLens is set as the type of realRatingMatrix. We can convert between the two types as I will show later in the project. library (readr) library (recommenderlab) data.movies <- read_csv("movies . csv") data. ratings <- read_csv("ratings. csv") data ("MovieLense") Importing the Libraries We're going to need a few different libraries for this project that are all very crucial to the success of this project. The first and foremost library that we need is recommenderlab, this gives us the foundation we will need to build our recommendation system, and we use it to get the dataset MovieLens. The next is ggplot2, this will give us powerful data visualization tools. To transform the data between the dataFrame and realRatingMatrix, we will need both data. table and reshape2. To make transformations to the genre column we will need to import the dplyr and tidyr libraries. Each of these libraries vastly simplify the operations we are completing in this project, and help the code be more succint and maintainable. library(ggplot2) library (data. table) library (reshape2) library(dplyr) library(tidyr) 1 Quick Overview of the Data Before we jump into processing the data, let's familiarize ourselves with the data we just imported. Summary of Movie Data There are three fields in movies: movieId, title, and genres. movieId is a numerical value denoted the identifying value of the movie record. title is eponymously the title of the movie in the record. genres is going to be tricky, it's a string of all the movie delineated by |. This column is not formatted for optimal processing because the computer can not parse the formating of the genre in a useful manner yet, we will have to fix this in the data pre-processing. summary (data.movies) ## movieId title 1 Length : 10329 genres ## Min. : Length : 10329 ## 1st Qu. : 3240 Class : character Mode : character ## Median : 7088 Class : character Mode : character ## Mean : 31924 ## 3rd Qu. : 59900 ## Max. : 149532 head(data.movies, n = 2) ## # A tibble: 2 x 3 ## movieId title genres ## <dbl> <chr> <chr> ## 1 1 Toy Story (1995) Adventure | Animation | Children | Comedy | Fantasy ## 2 2 Jumanji (1995) Adventure | Children | Fantasy Summary of the Ratings Data We have four columns of data in this set: userId, movieId, rating, and timestamp. The userId is a unique identifier of the user. The movieId is an identifier linked to the previous dataset with the identifier value of the movie they are reviewing. rating is the rating the user gave, the scale is out of 5 with 5 beings the best rating the user can give. The timestamp is a UTC timestamp using the UNIX time stamping format. This format counting every millisec that has passed since 01/01/1970. summary (data.ratings) ## userId movieId rating timestamp ## Min. : 1.0 Min. : 1 Min. :0.500 Min. :8.286e+08 ## 1st Qu. : 192.0 1st Qu. : 1073 1st Qu. : 3.000 1st Qu. : 9.711e+08 ## Median :383.0 Median : 2497 Median :3.500 Median :1.115e+09 ## Mean :364.9 Mean : 13381 Mean :3.517 Mean :1.130e+09 ## 3rd Qu. : 557.0 3rd Qu. : 5991 3rd Qu. : 4.000 3rd Qu. : 1.275e+09 ## Max. :668.0 Max. : 149532 Max. : 5.000 Max. : 1.452e+09 head(data.ratings, n = 2) ## # A tibble: 2 x 4 ## userId movieId rating timestamp ## <dbl> <dbl> <dbl> <dbl> ## 1 1 16 4 1217897793 ## 2 1 24 1.5 1217895807 2 Pre-Processing the Data Genre We identified in our data exploration process that the genre column was somewhat unusable for any computation or statistical process, which is what we are trying to accomplish here. So we need to process the genres into something usable. In one of our labs we had to do this, but the data there was categorical. This data is categorical with one major caveat the data has multiple categories in a single column. This is a big problem. The only way we can store this data in a meaningful way is to use "one-hot" variables to store each category as a column of binary values with one signifying membership in that genre. The first step to accomplish this is to separate the genres into rows by the delimiting I. Then we're gonna put a one at the end of all data. Then we are going to spread the genres and values, so there are 1s and NAs. We then replace all NAs that are in numeric columns with 0s. We then clean some extraneous columns. movies.genre <- data.movies %>% separate_rows(genres, sep = "[|]") %>% mutate(Value = 1) %>% spread(genres, Value) %>% mutate_if (is.numeric, ~replace(., is.na(.), 0)) movies.genre <- movies. genre [-c(3, 15)] movies. genre A tibble: 10,329 x 20 ## movieId title Action Adventure Animation Children Comedy Crime Documentary ## <dbl> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> ## 1 1 Toy 0 1 1 1 1 0 0 ## 2 2 Juma~ 0 1 0 1 0 0 0 ## 3 3 Grum~ 0 0 0 0 1 0 0 ## 4 4 Wait~ 0 0 0 0 1 0 0 ## 5 5 Fath~ 0 0 0 0 1 0 0 ## 6 6 Heat~ 1 0 0 0 0 1 0 ## 7 7 Sabr~ 0 0 0 0 1 0 0 ## 8 8 Tom ~ 0 1 0 1 0 0 0 ## 9 9 Sudd~ 1 0 0 0 0 0 0 ## 10 10 Gold~ 1 1 0 0 0 0 0 ## # with 10,319 more rows, and 11 more variables: Drama <dbl>, Fantasy <dbl>, ## # `Film-Noir <dbl>, Horror <dbl>, Musical <dbl>, Mystery <dbl>, ## # Romance <dbl>, "Sci-Fi' <dbl>, Thriller <dbl>, War <dbl>, Western <dbl> Real Rating Matrix As stated in the intoduction, the type of realRatingMatrix is important to the project, so much so that the original dataset is in this format. This format is not great for data exploration so I imported a version of the data that is in the dataFrame format and one that is in the realRatingMatrix type. This is a small guide as to how we would convert between the dataFrame and the realRatingMatrix. The realRatingMatrix is a sparse matrix that contains ratings as the primary value of focus. It's important we strip out the userId column otherwise we produce a myriad of errors that actually took an hour to solve. matrix.rating <- dcast (data.ratings, userId ~ movieId, value. var = "rating") matrix.rating <- as.matrix(matrix.rating[, -1]) matrix.rating <- as (matrix. rating, "realRatingMatrix") matrix.rating ## 668 x 10325 rating matrix of class 'realRatingMatrix' with 105339 ratings. 3 Data Exploration Ratings Overview Let's dive deeper into the data. The first thing I want to look at is the overall distribution of the ratings in this dataset. Because I am only looking at one-dimensional data, I will use a histogram to accomplish this task. The ratings are out of five, so naturally we will use 5 bins in the histogram. This will give us a brief overview of the skew of the ratings. ggplot (data = data.ratings) + geom_histogram(aes(x = rating, fill = .. count .. ), bins = 5) + scale_fill_distiller (palette = "Reds") + labs (x = "Rating", y = "Count", title = "Ratings Histogram Ratings Histogram 50000 40000 - count 50000 30000 - 40000 Count 30000 20000 20000 - 10000 10000 - 0 - I I 0 1 2 3 4 5 Rating Looking at the data, the ratings skew towards the 5, meaning people are more likely to rate a movie well vs. critically. 4 Data Exploration Top 10 Most Reviewed Movies I wanted to take a peek at what movies had the most reviews. I wanted to do this to accomplish two goals: find the highest count of reviews on the top-n movies, and to see what movies were popular amoung the MovieLens reviewers. movie. views <- data.ratings %>% group_by (movieId) %>% tally () movie. views <- merge (movie. views, data. movies [, c("movieId", "title")]) movie. views <- movie. views [order (movie. views$n, decreasing = TRUE) , ] movie. views <- movie. views %>% select (-movieId) head (movie. views, n = 10) ## n title ## 261 325 Pulp Fiction (1994) ## 317 311 Forrest Gump (1994) ## 280 308 Shawshank Redemption, The (1994) ## 427 294 Jurassic Park (1993) ## 526 290 Silence of the Lambs, The (1991) 273 Star Wars: Episode IV - A New Hope (1977) ## 231 ## 2057 261 ## 523 253 Terminator 2: Judgment Day (1991) Matrix, The (1999) ## 99 248 Braveheart (1995) ## 472 248 Schindler's List (1993) Number of Movies in Each Genre genre_list <- colnames (movies.genre) [-(1:2)] for (genre in genre_list) { } 5 Model Creation I haven't been able to properly introduce you to recommenderlab yet! Let's change that. So recommenderlab is a package that allows developers to quickly and easily develop and test recommender systems in R. You're probably thinking, "that's great but what does that actually look like?" With recommenderlab we get 11 different recommendation algorithms, 3 different means of evaluation, and several different evaluation measures. Algorithm Selection Of those 11 different algorithms, I selected Item-Based Collaborative Filtering as the most viable option for our model. This algorithm takes movies that are rated well by users of similar taste, and groups those users together recommending movies to other users in the group that have similar taste. Sounds familiar right? That's because this algorithm is a watered-down version of the real algorithm they use on many different streaming services! I think that's pretty cool, I hope you do too. models.rec <- recommenderRegistry$get_entries(dataType = "realRatingMatrix") models. rec$UBCF_realRatingMatrix$parameters ## $method ## [1] "cosine" ## ## $nn ## [1] 25 ## ## $sample ## [1] FALSE ## ## $weighted ## [1] TRUE ## ## $normalize ## [1] "center" ## ## $min_matching_items ## [1] 0 ## ## $min_predictive_items ## [1] 0 Training and Testing Data In order to train the model and then make sure the model is fitted correctly to the data, we need to seperate the data we have in to training data, that will be used to fit the model, and testing data, that will be used to test the fit of the data within the model. We will be using an 80/20 split on training and testing data. We are using the realRatingMatrix from the original dataset to avoid overcomplicating the realRatingSection with the merging of the two dataFrames into a matrix. data.movie. lens <- MovieLense isTrain <- sample(x = c(TRUE, FALSE), size = nrow(data.movie. lens), replace = TRUE, prob = c(0.8, 0.2)) model.data. train <- data.movie. lens [isTrain, ] model.data. test <- data.movie. lens [! isTrain, ] 6 Model Creation Cont. Building the recommendation model model. rec <- Recommender (data = model.data. train, method = "IBCF") model.rec ## Recommender of type 'IBCF' for 'realRatingMatrix' ## learned using 747 users. Model Results Applying the Model to Some Testing Data Now that we have a working model, let's use the testing data to try to predict what movies those users would like to see. We will pick the top 5 movies for them and look at the first 4 user predictions. model.predict <- predict (object = model.rec, newdata = model.data.test, n = 5) matrix. predict <- sapply (model. predict@items, function (x) { colnames (data. movie. lens) [x] matrix. predict [, 1:4] ## 18 ## [1,] "Desperado (1995) " ## [2,] "Haunted World of Edward D. Wood Jr., The (1995)" ## [3,] "Aliens (1986) " ## [4,] "Shall We Dance? (1996)" ## [5,] "Restoration (1995) " ## 21 ## [1,] "Blue Angel, The (Blaue Engel, Der) (1930)" ## [2,] "Paths of Glory (1957) " ## [3,] "Fluke (1995) " ## [4,] "Afterglow (1997) " ## [5,] "Pest, The (1997) " ## 27 ## [1,] "Deceiver (1997) " ## [2,] "Losing Chase (1996) " ## [3,] "Wedding Gift, The (1994)" ## [4,] "Maya Lin: A Strong Clear Vision (1994)" ## [5,] "Powder (1995) " ## 31 ## [1,] "Antonia's Line (1995) " ## [2,] "Haunted World of Edward D. Wood Jr., The (1995)" ## [3,] "Maya Lin: A Strong Clear Vision (1994)" ## [4,] "Kansas City (1996)" ## [5,] "Spice World (1997) " We can see that the model has produced the top five movies for those users, which atleast means our model is up and running. Quick note for anyone running the Rmd file, this knits but will oddly not work in the R console. Don't ask me why, I don't know. 7 Model Results Cont. Similarity Matrix The way IBCF works is as follows: 1. Create a record that user u1 rated the following items: i1, i2, . . . , in. 2. Create a record for all users in the dataset like this 3. Find the similarities between the users We will show this raw similarity matrix. It actually has already been generated as a part of the IBCF recommendation system. model. info <- getModel (model.rec) image (model. info$sim) 500 Row 1000 1500 500 1000 1500 Column Dimensions: 1664 x 1664 This may be faint but the darker the pixel to higher the similarity value between the two indexes (users). Some things we can glean from this data is the later movies in the matrix are somehow similar to nearly every movie in the dataset which may be indicative of faulty ratings or model overfit. 8 Model Results Cont. Number of Times a Movie is Recommended Some movies are more popular than others. This means the algorithm will recommend certain movies more than others. We can see that our data is long-tailed in that certain movies are wildly popular but a vast majority of movies are only recommended to under 5 users. This means that the algorithm is finding more unique movies to each different users taste, which is indicitive of a good model fit. model. predict. num_items <- factor(table (matrix. predict)) qplot (model.predict.num_items) 200 150 - 100 - 50 - 0 - ! I 1 2 3 4 5 6 7 8 9 10 11 12 13 14 17 21 model.predict.num_items 9 ! I I I 1 I 1 1 1 I - 1 1 Model Results Cont. Top 10 Recommended Movies In the last bit of analysis of the model, I'd like to take a peek at the top 10 most recommended movies by the algorithm. We can do this easily by using the built in sort and head functions. model. predict.num_items. sorted <- sort (model.predict.num_items, decreasing = TRUE) head (model.predict.num_items. sorted, n = 10) ## Horseman on the Roof, The (Hussard sur le toit, Le) (1995) ## 21 ## When the Cats Away (Chacun cherche son chat) (1996) ## 17 ## Flower of My Secret, The (Flor de mi secreto, La) (1995) 14 ## Haunted World of Edward D. Wood Jr., The (1995) ## 14 ## Maya Lin: A Strong Clear Vision (1994) ## 14 ## Nadja (1994) ## 13 ## unknown 13 ## Operation Dumbo Drop (1995) ## 12 ## Blood Beach (1981) ## 11 ## ## ## Blue Angel, The (Blaue Engel, Der) (1930) ## 10 ## Levels: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 17 21 10/nDAT 301 Project 1 For Project 1 you need to choose a topic that requires either a data manipulation or a simple statistical or machine learning technique that should be equivalent in load to about 1.5 - 2 labs or 1.5 - 2 homework assignments. · Don't forget to submit all the files you used, including source code (.R), Rmarkdown that produces pdf and slides, data files, possible images, etc. · Do NOT zip the files into a single folder. Submit all the files unzipped. Steps and Requirements · Choose a data set, define a problem (or problems) you want to solve · Perform some exploratory data analysis and if appropriate, apply some statistical procedures to answer your questions. In your report (see below), you will need to provide the code, explain what you have done, and what you have found or what your conclusion is. . Use Rmarkdown to create a pdf report and html ioslides. You can interweave the (textual) discussion with code chunks and their output. Explain which data set you used (what the resource is and what the variables are). Formulate the problem(s) and explain how you tried to answer your questions and what the conclusion is. Examples of projects made by students in earlier semesters are provided in Canvas course shell. . Students must work on the project individually (no group work). Also, you need to make a short presentation (around 5-10 minutes), record the video and submit it on canvas along with the other necessary files that you need to submit. · You MUST submit all the files you used (including source code (.R), Rmarkdown that produces slides and pdf, data files, possible images, etc.) · Do NOT zip the files into a single folder. Submit all the files unzipped. Where to Look for Datasets You can look for a data set wherever you want. The file opendatasites. csv with the list of links where you can search for datasets is provided in Canvas. Geographical notions in this file to which the links are related include regions, states, countries, cities. In addition, here are two popular sites with a lot of datasets: · https://www.kaggle.com/datasets · https://archive.ics.uci.edu/ml/datasets.php Grading Policy You will be assessed on the quality of the written report as well as the presentation in the following way: 1. Background and problem definition (5pts) 2. Data wrangling, munging and cleaning (20pts) 3. Exploratory Data Analysis (30pts) 4. Data Visualization (15pts) 5. Final paper/slides and presentation (30pts) IMPORTANT: IF YOU CHOOSE A TOPIC OTHER THAN SUGGESTED ABOVE, YOU MUST BE ORIGINAL! COPYING PROJECTS FOUND ONLINE IS PLAGIARISM AND ACADEMIC DISHONESTY (CHEATING!) WHICH VIOLATES ACADEMIC INTEGRITY, AND WILL RESULT IN FINAL GRADE OF E! For information on academic integrity, including the policy and appeal procedures, please visit http://provost. asu.edu/academicintegrity Reminder · Don't forget to submit all the files you used, including source code (.R), Rmarkdown that produces pdf and slides, data files, possible images, etc. · Do NOT zip the files into a single folder. Submit all the files unzipped.

2. (30 Points) Vital capacity is a measure of the amount of air that someone can exhale after taking a deep breath, Data was collected on brass players and a control group. Assume the population distributions were normal. Table 1: Table for Question 2 Brass Player Control Group 4.7 4.2 4.6 4.7 4.3 5.1 4.5 4.7 5.5 5.0 4.9 - 5.3 - (a) (10 Points) Conduct a t test to determine whether the population mean for brass is larger than that for control. (b) (5 Points) Provide the 95% confidence interval for the difference in the two population means. (c) (15 Points) A researcher claims that in theory the "spread/variance" in the two popu- lations is the same. Repeat step 1 (and 2) utilizing this assumption with the argument "var.equal" within the "t.test" function.

MT 328 - Non-Linear dynamical systems: Routes to Chaos Problem Sheet 3 3.1 -Sketch the phase portrait for the linear system below and justify its trajectories and the stability and type of its critical point. You need to deduce the equations describing the system to justify your answers. dy1 =- 2y - 3y dt 1 2 dy 2 =3y -2y 1 2 dt 3.2 -Use the plane TrA/DetA to identify the Type and Stability of the critical points for the linear dK systems -= A X, in each of the cases of matrix A, as given below; not to be done using the direct calculation of eigenvalues and eigenvectors of the matrices. (a)A=2 1);(b)A= ( 2 1 );(c)A=(1 -4 );(d)A= 0 2 ; (e)A=( -1 8 ). 1 2) 1 -3 ) ( 2 -1 ( -3 -1 ) (1-2 7) Justify your answers based on the values of Det A and TrA that you obtain directly from the matrices A and hence describe the eigenvalues for each system. (Help: Remember that when Det A<0, you don't need any other calculation as the CP is identified in the lower part of the plane TrA/DetA .... ) 3.3 - For each of the linear dynamical systems dk = A X, with matrix A given below, find the differential equations describing the systems and describe the trajectories, stability, and type of their expected phase portraits and critical points. Sketch their phase portraits. It is crucial that you justify your answers: just drawing a phase portrait without justifying its features attracts no marks. (a) A=\ ( 9 4 ) -9 -3 ) (b) A=| 0 2 ) (c) A=[ -26 -8) 1 1 ( -12 4 ) Below you can find relevant output from Mathematica to use in your solution. A = { {9, 4}, {-9, -3} } A = { {-2, 1}, {0, 2} } A = { {-12, 4}, {-26, -8} } Eigensystem[A] { {-10+10 i, -10-10 i }, { {1-5i, 13}, {1+5i, 13} } } Eigensystem[A] { {3, 3}, { {-2, 3}, {0, 0} } } Eigensystem[A] { {-2, 2}, { { 1, 0}, {1, 4} } }

4. A university is about to buy a new 3D printer. There are two alternatives available. a. The HP printer which is more expensive to buy but also faster. b. The Canon printer which is cheaper but slower Processing times and times between job arrivals are exponential > M/M/1 model. . 1 = 10 jobs per day . MHP = 20 jobs per day . He = 15 jobs per day The buying and waiting costs: . buying price: CHP = $15000, Cc = $10000 . The waiting cost per job and time unit is estimated to be $500 per job and day. Which printer should the university choose in order to minimize the expected costs?

Overview - Confidence Intervals for the Mean Height of American Adults The National Health and Nutritional Examination Survey (NHANES) is a population survey implemented by the Centers for Disease Control and Prevention (CDC). Although NHANES encompasses tens of thousands of individuals and 1,191 variables, we will look at height measurements of 18,391 individuals older than 19 years of age. The purpose of this assignment is to estimate 95% confidence intervals for the average height of adults in the NHANES study. Although the NHANES data is a sample of all adults in America, we will pretend that the individuals in the NHANES study comprise the population of interest, and then we will randomly select 10 individuals from the NHANES data to construct 95% confidence intervals for the mean height of the NHANES population. Since we have the benefit of ascertaining the mean height of the NHANES population, we can observe whether our confidence intervals actually contain the population mean. Learning Outcomes 1. Calculate and interpret measures of the location of data: quartiles, percentiles, and z-scores (CLO: A). 2. Calculate and interpret measures of the center of data: mean, median, and mode (CLO: A). 3. Calculate and interpret measures of the variability of data: variance, standard deviation, and range (CLO: A). 4. Construct confidence intervals for a population mean (CLO: A, B, C). 5. Interpret confidence intervals for a population mean (CLO: A, B, C). 6. Find the probabilities of normal random variables using R functions (CLO: B, D). 7. Find quantiles and percentiles of normal random variables using R functions (CLO: B, D). 8. Find the probabilities of T random variables using R functions (CLO: B, D). 9. Find quantiles and percentiles of T random variables using R functions (CLO: B, D). Directions 1. Find the mean and standard deviation of the height measurements for all the individuals in the NHANES population. 2. Randomly select 10 individuals from the NHANES data and report the sample values in a table. 3. Report the five-number summary, mean, and standard deviation of your sample within a table. 4. Calculate a 95% Confidence interval for the average height of adults in the NHANES study using the population standard deviation. 5. Calculate a 95% Confidence interval for the average height of adults in the NHANES study using the sample standard deviation. Load the NHANES Data into RStudio · Download this .csv of height data into some folder on your computer. Start RStudio · Select the working directory. The working directory is where RStudio saves and loads files. To select the working directory, follow these steps: 1. 1. Select the "Files" tab next to the console in RStudio. 2. On the "Files" tab, navigate to the folder where you saved the .csv file. 3. Select the "More" menu item. 4. Select "Set as working directory" . Load the .csv file into a data frame by typing the following command into the console. population.dataframe = read.csv("NHANES_project_data.csv") · Assign the height measurements to a vector. population.height = population.dataframe$height Find the Mean and Standard Deviation of the Height Measurements of the NHANES Population Fill in the Blank : Each correct answer earns 2 points (4 points total) Using the Rich Content Editor, report the mean and standard deviation of the height measurements of the NHANES population. Remember that the heights are measured in centimeters. · population mean = < your answer here> · population standard deviation = < your answer here> Randomly Select 10 Height Measurements from the NHANES Population (Fill in the Blank) Fill in the blank : A complete table earns 2 points total. Use the Rich Content Editor to construct a table that lists the height measurements in your sample. You can use the following table as a template. A Sample of Height Measurements X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 <Value><Value><Value> <Value> <Value> <Value> <Value> <Value> <Value><Value> · To assign the number of individuals in the NHANES data to a variable, use the following R code: population.size = length(population.height) · To randomly select 10 height measurements from the population, use the following R code: sample.height = population.height[sample(population.size, 10)] Report the Five-Number Summary, Mean, and Standard Deviation of the Sample (Fill in the Blank) Fill in the Blank : Each correct answer earns 1 point (7 points total) Use the Rich Content Editor to construct a table to hold the five-number summary, sample mean, and sample standard deviation of the height measurements in your sample. Five-Number Summary of the Sample Heights Minimum First Quartile Median Third Quartile Maximum Mean Standard Dev. <your answer here> <your answer here> <your answer here> <your answer here> <your answer here> <your answer here> <your answer here> Calculate a 95% Confidence Interval for the Population Mean Using the Population Standard Deviation (Fill in the Blank) Fill in the Blank : Each correct answer earns 2 points (4 points total) State the lower bound and the upper bound of a 95% confidence interval for the average adult height in centimeters. Please use the population standard deviation to calculate the upper and lower bounds, and show your calculations. Does the interval contain the population mean? Lower Boundary : < your work here> Upper Boundary : < your work here> Calculate a 95% Confidence Interval for the Population Mean Using the Sample Standard Deviation (Fill in the Blank) Fill in the Blank : Each correct answer earns 2 points (4 points total) State the lower bound and the upper bound of a 95% confidence interval for the average adult height in centimeters. Please use the sample standard deviation to calculate the upper and lower bounds, and show your calculations. Does the interval contain the population mean? Lower Boundary : < your work here> Upper Boundary : < your work here>

Question 10: Create a histogram to visualize the 60 simulated values from the Exponential distribution. A. Include the histogram here. Include informative axis labels and a title. Hint: You learned how to create a histogram in Data Analysis 1 using ggplot and geom_histogram./n60 simulated values from the Exponential distribution. A. Include the histogram here. Include informative axis labels and a title. Hint: You learned how to create a histogram in Data Analysis 1 using ggplot and geom_histogram. B. Describe the shape, center, and spread of the distribution of the simulated rainfall duration values.

Question 11: Determine and report the mean of the 60 simulated rainfall duration values. How does this value compare to the theoretical expected value of the Exponential random variable? It turns out that statisticians know a lot about the Exponential distribution. Once you choose to model a random variable with an Exponential distribution, you can answer all sorts of questions about that variable related to probability. Take, for example, the question, "What is the probability that a randomly chosen rainfall event in Corvallis in/nExponential distribution. Once you choose to model a random variable with an Exponential distribution, you can answer all sorts of questions about that variable related to probability. Take, for example, the question, "What is the probability that a randomly chosen rainfall event in Corvallis in January is longer than 4 hours?" If we assume that rainfall duration is Exponentially distributed, we can find this theoretical probability by using the pexp() function in R. 1-pexp (q = 4, rate = 1/3) Note that the function pexp() gives the area under the Exponential curve below a given value, q, with a given rate (X) parameter. Since we're interested in the probability that rainfall duration is greater than 4 hours, we have to take one minus that probability. Assuming an Exponential distribution has allowed us to calculate a theoretical probability. If we want to calculate the probability empirically from our simulated data we simply need to determine how/nto calculate the robabinity empirically fontour simulated data, we simply need to determine how many observations fall above 4, then divide this number by the total sample size: df %>% filter (sim_exp > 4) %>% summarise (proportion = n()/nrow(df))

Question 12: Answer the following question, by calculating both the (A) theoretical probability and (B) the empirical probability based on your simulated data. What is the probability that a randomly selected rainfall event in Corvallis in January is shorter than 1.5 hours?" A. Report the theoretical probability. Include the code used to answer this question. B. Report the empirical probability based on your simulated data. Include the code used to answer this question.

Question 5: What is the average dollar amount of the 1000 simulated spins? How does this compare to your answer in Question 2? In Question 1, you determined the theoretical probability mass function for the random variable X. Now let's construct the simulated probability mass function for the 1000 simulated wheel spins. The following R code will construct the simulated PMF based on your 1000 simulated wheel spins. Be sure that you've stored your simulated spins in a vector called spins_thousand. The following code will not work if you don't have this vector properly named. table (as.factor(spins_thousand))/1000 In the code above, the as. factor function assigns each of the outcomes in the spins_thousand vector to a category. The vector of categorical values is then used as the input in the table/neach of the outcomes in the spins_thousand vector to a category. The vector of categorical values is then used as the input in the table function which determines the frequency of each category. The values in this table are then finally divided by 1000 so that we are presented with the/ndivided by 1000 so that we are presented with the proportion of times each dollar amount was spun. This is the PMF.

Question 3: Suppose a contestant spins the wheel four times. How likely is it they will spin and land on $200 all four times? Show your work.

Question 2: A contestant spins the wheel. What is the expected dollar amount they will win? Show your work. Hint: use the PMF from the previous problem to calculate the expected value of X.

Question 1: The table below displays the values the random variable X can take on, along with the number times each value appears on the wheel (labeled Count). Fill in the last row of the table with the correct probability mass function values. That is, for each value X can take on, determine the probability the wheel is spun and lands on the given value, p(x). Round values to three decimal places. x $500 Count 2 p(x) = P(X= x)| $150 $200 $250 $300 $350 2 4 2 2 1

Lending Club Lending Club is an online company that provides platform for peer-to-peer lending that matches borrowers with investors. The company services people who need small personal loans (approximately $1,000 - $4,000). Borrowers receive the full amount of the issued loan minus the origination fee, which is paid to the company. Investors purchase notes backed by the personal loans and pay Lending Club a service fee. The company shares data about all loans issued through its platform during certain time periods. Dataset The dataset (LendingClub40k.csv) contains information of 40,000 loan records and 27 variables as shown in the table below: id loan amnt funded amnt term Attribute int rate installment grade emp_length emp title home_ownership annual inc verification_stat us MIS 740 Spring 2024 issue d Description A unique Lending Club assigned ID for the loan listing. The listed amount of the loan applied for by the borrower. If at some point in time, the credit department reduces the loan amount, then it will be reflected in this value. The total amount committed to that loan at that point in time. The number of payments on the loan. Values are in months and can be either 36 or 60. Interest Rate on the loan The monthly payment owed by the borrower if the loan originates. Lending Club assigned loan grade Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years. The job title supplied by the Borrower when applying for the loan. The home ownership status provided by the borrower during registration or obtained from the credit report (RENT, OWN, MORTGAGE, OTHER). The self-reported annual income provided by the borrower during registration. Indicates if income was verified by Lending Club-not verified, or if the income source was verified The month which the loan was funded loan status pymnt_plan desc purpose zip_code addrState dti delinq2Yrs fico_range_high fico_range_low inq_last_6mths pub_rec revol bal revol util ● Current status of the loan ● Indicates if a payment plan has been put in place for the loan Loan description provided by the borrower A category provided by the borrower for the loan request. The first 3 numbers of the zip code provided by the borrower in the loan application. The state provided by the borrower in the loan application A ratio calculated using the borrower's total monthly debt payments on the total debt obligations, excluding mortgage and the requested Lending Club loan, divided by the borrower's self-reported monthly income. Part 1: Data preparation (Use R or RapidMiner) Determine the role of each variable (dependent variable, independent variable, excluded from analysis, etc.). You should keep as many variables in the dataset as possible. Provide the descriptive statistics of the variables in the dataset. The number of 30+ days past-due incidences of delinquency in the borrower's credit file for the past 2 years The upper boundary range the borrower's FICO at loan origination belongs to. The lower boundary range the borrower's FICO at loan origination belongs to. The number of inquiries in past 6 months (excluding auto and mortgage inquiries) Number of derogatory public records Total credit revolving balance Revolving line utilization rate, or the amount of credit the borrower is using relative to all available revolving credit. Report the detailed process of your data cleaning and preparation, e.g., how you deal with the missing values, outliers, data anomalies, imputation, discretization, why you perform certain data cleanings, and why you use certain parameters. Perform statistical tests for outliers. Based on the variables selected from the previous step, perform missing value imputation on at least three variables. Perform missing value imputation using machine learning algorithm. Report the results and provide your interpretation/conclusion related to anomalies in the dataset. Provide screenshots of the R script and/or RapidMiner process.

1. (20 Points) The time required in minute for 26 students to finish a 50 minute history exam is recorded as the following: 46.23713, 52.58143, 42.89946, 50.66941, 28.42386, 50.77394, 42.65797, 50.42930, 59.45894, 64.33043, 44.48510, 48.43943, 45.10024, 48.38066, 49.75874, 43.07601, 48.27409, 55.65475, 50.56731, 48.17550, 52.46376, 53.82756, 56.37297, 59.33533, 51.23410, 47.14756 (a) (5 Points) Draw a histogram of the data and describe the shape of it. (b) (5 Points) Draw a qqplot to check whether the normality assumption of the data is valid [Hint: Use "qqnorm" command] (c) (10 Points) Perform a hypothesis test to check whether we can say that students can finish the exam in time./n

PROBLEM 3 3. Considerthefollowingmodelfor{(xi,yi)}: Р Yi = Bo+Bjoj (xi) + ɛi for i j=1 = 1, n for some basis functions 1(x), ..., Op(x) where the parameters Bo, B₁, · Р Bo+Bjoj (xi) ≥ 0 for i j=1 = 1, ... n " To estimate ẞo, ẞ1, ..., ßp, we used constrained least squares: Minimize subject to the constraints n Р Σ Yi βο - Σ β;φ; (α;) Σβ,Φ,(;) i=1 Bo+Bjoj (xi) ≥ 0 for i = j=1 = 1, 2 ... n Bp are such that The parameter estimates can be computed using an interior point algorithm whereby we define (Bo(r), · · ·‚ßp(r)) to minimize - Bo - Р 2 n Р ½Σ ( − − £¾(x)* −Ë (+Ź 8,6,(z) j=1 ―r In (xi) i=1 j=1 for r > 0 and let r↓0. (Note that we are implicitly assuming that the unconstrained LS estimates of the parameters violate the constraints.) (a) Derive an IRLS algorithm for computing (Bo(r),···, ßp(r)) for r > 0. 3 (b) The file problem3-data.txt on Quercus contains 1000 observations (xi, yi) where 0 ≤ X1,X1000 ≤ 1. Take p 10 and define 1(x), …, 10(x) to be B-spline functions; in R, = these can be defined as follows: > library(splines) > xx <- bs(x, df=10) The object xx will be matrix with 10 columns with the number of rows equal to the length of x. Compute the constrained LS estimates for these data. (The unconstrained LS estimates violate some of the non-negativity constraints.) 4

STA 412 HW 04 (ANOVA) Question: 1 Total Points: 30 30 Bonus Points: Score: 0 0 1. A large laboratory has four types of devices used to determine the pH of soil samples. The laboratory wants to determine whether there are differences in the average readings given by these devices. A bag of soil is collected and randomly divided into 24 soil samples. So all the soil samples have the same pH. Soil samples are randomly assigned to each device and then tested. The response recorded is the the pH reading of the device. These values, along with summary statistics, are given in the following table. pH readings Sample Sample Sample Device 1 2 3 4 5 6 Size Mean Standard deviation A 6.35 6.41 6.78 6.72 6.56 6.38 6 6.533 0.184 B 6.52 6.57 6.57 6.78 6.79 7.06 6 6.715 0.204 C 6.94 6.64 7.04 6.65 7.02 6.85 6 6.857 0.177 D 6.65 7.19 6.6 6.87 6.92 6.88 6 6.852 0.212 (a) (5 points) The goal is to investigate whether there are differences in pH readings for the four devices. Write down the null and alternative hypotheses. (b) (5 points) Input the data into R, and calculate sample size, mean and variance for each samples with function aggregate and also calculate the overall mean and sample size. (c) (10 points) Assume that the assumptions for one-way ANOVA are satisfied. Starting with the summary statistics of each sample, fill in the numbers in the following ANOVA table. Show your calculation details. df SS MS F-statistic Between-sample Within-sample (d) (5 points) Type the sample data into R and use R to create an ANOVA table. Attach the R output. What is the p-value of the F test? Is the null hypothesis rejected? (e) (5 points) What are the assumptions needed for the one-way ANOVA in b and c?

STA 412 HW 06 (Contrasts F-test) Question: 1 Total Points: 30 30 Bonus Points: Score: 0 0 1. (Textbook, Exercise 9.13)Researchers conducted an experiment to compare the effectiveness of four new weight- reducing agents to that of an existing agent. The researchers randomly divided a random sample of 50 subjects into five equal groups, with preparation Al assigned to the first group, A2 to the second group, and so on. They then gave a prestudy physical to each person in the experiment and told them how many pounds overweight they were. A comparison of the mean number of pounds overweight for the groups showed no significant dif- ferences. The researchers then began the study program, and each group took the prescribed preparation for a fixed period of time. The four new agents are labeled A1, A2, A3, and A4, and the standard agent is labeled agent S, as shown in the table below. Subscript Agent Population mean Sample size 1 A1 με 10 2 A2 μ2 10 3 A3 из 10 4 A4 μ4 10 5 S με 10 The weight losses recorded at the end of the study period are given in the file "hw5_data.txt". You can read the data into R using the following code. df <- read.table("hw06_data.txt", header = TRUE, strings AsFactors = TRUE) df$agent # agent is a factor ## [1] A1 A1 A1 A1 A1 A1 A1 A1 A1 A1 A2 A2 A2 A2 A2 A2 A2 A2 A2 A2 A3 A3 A3 A3 A3 A3 A3 A3 A3 A3 A4 A4 ## [33] A4 A4 A4 A4 A4 A4 A4 A4 S S S S S S SS S S ##23 Levels: A1 A2 A3 A4 S (a) (5 points) Verify that the contrasts below are mutually orthogonal. 1 = μ1 + μ2 + μ3 + μ4 - 4μ5 2 = μ1 + μ2 - μ3 - μ4 3 = 1 - 2 + μ3 - μ4 l₁ = μ1 - μ2 - μ3 + μ4 = = μ5 against Ha at least one of (b) (5 points) Consider the hypothesis testing problem Ho: μ₁ μ1,..., μ5 is different. Use R to generate an ANOVA table. (c) (10 points) Use the glht function in R to construct 95% two-sided confidence intervals for each contrast in (a) (without correction for multiple comparisons). (d) (10 points) Use the aov function in R to generate an ANOVA table including the sum of squares, mean square, F test statistic for each contrast in (a). Based on the ANOVA table, what are the contrasts that are statistically significantly different from zero (a 0.05, without correction for multiple comparisons)? =

STA412 Quiz2 Question: 1 2 3 4 Total Points: 30 30 24 16 100 Bonus Points: 0 0 Score: Factor B (temperature) 1. The effective life (in hours) of batteries is compared by material type (Nickel-Metal Hydride, Lithium-Ion) and operating temperature (70°F,125°F). Batteries are randomly selected from each material type and they are then randomly allocated to each temperature level. A summary of the study is given below. Factor A (material) pulation ample ¡ample Sample mean size mean variance Nickel-Metal Hydride 70°F μ11 50 80 222 Nickel-Metal Hydride 125°F [12 50 120 212 Lithium-Ion Lithium-Ion 70°F μ21 50 82 224 125°F μ22 50 117 216 For your convenience, the within-sample variation is s²w = 218.5. Consider the following contrasts: = • The average effect of factor A (material type): (A = ½μ11 + ½μ12 - μ21 - μ22 • The average effect of factor B (operating temperature): (B = 1½ μ11 - ½½μ12 + ½μ21 - μ22 • The contrast for the interaction effect: AB = μ11- μ12- μ21+ μ22 Conduct t-test to test the following hypothesis at α = 0.05. Hint: You can treat (μ11,μ12,μ21,μ22) as (µ1‚µ2‚µ³‚µ) and perform one-way ANOVA. (a) (10 points) Ho: l₁ = 0 against Ha: lд= 0. 2. (b) (10 points) Ho: B = 0 against Ha: l= 0. (c) (10 points) Ho: AB = 0 against H₁: lÃÅ= 0. A pharmaceutical company is interested in the effectiveness of a new preparation designed to relieve arthritis pain. Three variations have been prepared for investigation, which differ according to the proportion of the active ingredients. Patients enrolled in the study are randomly assigned to one of the three variations, and the time (in minutes) until pain relief is recorded on each subject. A summary of the study is given below. Proportion of the Population Sample Sample Sample active ingredients mean size mean variance 15% 40% με 30 23.45 10.75 με 35 14.08 9.62 50% Из 25 15.02 8.65 For your convenience, the within-sample variation is s²w= 9.729. Consider the contrasts below: l1=241- М2- из 12 = М2- из (a) (10 points) Construct a 95% two-sided confidence interval for 11. (b) (10 points) Construct a 95% two-sided confidence interval for l₂. (c) (10 points) Using the Bonferroni correction, construct 95% simultaneous two-sided confidence intervals for 1 and l2. Page 2 3. Aninvestigatorwishestocomparethetimetorelieffromheadachepainunderfourdistinctmedications (DrugA, B, C, and D). Twenty patients who suffer from chronic headache are randomly selected for the investigation, and five subjects are randomly assigned to each medication. The investigator use R to conduct pairwise comparisons among these drugs. The R output is given below. However, the investigator forgets to address the issue of multiple comparisons, as seen from the sentence Univariate p values reported in the R output. ## Simultaneous Tests for General Linear Hypotheses ## ## Multiple Comparisons of Means: User-defined Contrasts ## ## Fit: Im(formula = time ~ drug, data = data) ## ## Linear Hypotheses: ## Estimate Std. Error t value Pr(>|t|) ## A - B == 0 7.000 2.958 2.366 0.030913 ## A-C == 0 13.000 2.958 4.395 0.000452 ## A - D == 0 -2.000 0.676 0.508618 ## B C == 0 6.000 2.958- 2.958 2.028 0.059507 ## B D == 0 -9.000 2.958 -3.043 0.007759 ## C - D == 0 -15.000 2.958 -5.071 0.000113 ## (Univariate p values reported) (a) (6 points) Without any correction for multiple comparisons (i.e. using univariate tests), list all the pairs of drugs that show a significant difference in the time to relief from headache pain. Use an individual type I error rate of α₁ = 0.05. (b) (6 points) Using Bonferroni correction, list all the pairs of drugs that show a significant difference in the time to relief from headache pain. Use an experiment-wise type I error rate of α = 0.05. (c) (6 points) Using Scheffe's method, list all the pairs of drugs that show a significant difference in the time to relief from headache pain. Use an experiment-wise type I error rate of α = 0.05. (d) (6 points) Using Tukey's method, list all the pairs of drugs that show a significant difference in the time to relief from headache pain. Use an experiment-wise type I error rate of α = 0.05. A study is conducted to understand the relationship among the current through a conductor between two points, the voltage across the two points, and the resistance of the conductor. The response variable is the voltage (in volt). The two factors are the resistance and the current. A summary of the study is given below: Page 3 4. Resistance Current Population mean Sample size Low (12) Low (4A) μ11 Low (12) High (6A) μ12 High (202) Low (4A) μ21 22 22 High (202) High (6A) μ22 We consider three contrasts • Average effect of resistance: 1 = 1½ ½μ11 + ½ ½μ12 - μ21 μ22 • Average effect of current: 2 = 1½μ11 - 1/1μ12 + ½μ21 - μ22 • Interaction effect: l3 = μ11- μ12- μ21 + μ22 R is used to analyze the data and generate the ANOVA table below. In the output, combination refers to the combination of resistance and current. Unfortunately, the output doesn't directly provide the F test for the interaction effect between resistance and current. ## Df Sum Sq Mean Sq F value Pr(>F) ## combination 3 90 30 10 0.025 ## avg_effect_resistance 1 48 48 12 0.026 ## avg_effect_current 1 28 28 7 0.057 ## Residuals 4 16 4 (a) (4 points) Check whether l₁,l2,13 are mutually orthogonal. (b) (6 points) Based on the information given above, calculate the sum of squares for ₤3 the interaction effect between resistance and current. (c) (6 points) Conduct the F test to determine whether the interaction effect between resistance and current is significant at α = 0.05. Page 4/n1. The effective life (in hours) of batteries is compared by material type (Nickel-Metal Hydride, Lithium-Ion) and operating temperature (70°F, 125°F). Batteries are randomly selected from each material type and they are then randomly allocated to each temperature level. A summary of the study is given below. Sample Factor A (material) Factor B (temperature) pulation ample ample mean size mean variance Nickel-Metal Hydride 70-F μ11 50 80 222 Nickel-Metal Hydride 125°F [12 50 120 212 Lithium-Ion Lithium-Ion 70-F 125-F μ21 50 82 224 122 50 117 216 For your convenience, the within-sample variation is s²w = 218.5. Consider the following contrasts: • The average effect of factor A (material type): A = ½μ11 +12-21-22 •The average effect of factor B (operating temperature): B = M11 - ½μ12+21-22 • The contrast for the interaction effect: AB=μ11- μ12-H21+ μ22 Conduct t-test to test the following hypothesis at α = 0.05. Hint: You can treat (11,12,μ21,μ22) as (1234) and perform one-way ANOVA. (a) (10 points) Ho: A=0 against Ha: ld= 0.