4. Locally weighted linear regression and bias-variance tradeoff. (30 points)

Consider a dataset with n data points (xi, yi), xi ERP, following the following linear model

= 3* Ta

Yi

X₁ + €₁, i = 1,..., n,

where ~ N(0,02) are i.i.d. Gaussian noise with zero mean and variance o².

(a) (5 points) Show that the ridge regression, which introduces a squared 2 norm penalty on the

parameter in the maximum likelihood estimate of ß can be written as follows

8(X) = arg min {||XB-y||² + X|| B||³}

B

for property defined matrix X and vector y.

(b) (5 points) Find the close-form solution for B(A) and its distribution conditioning on {x;}.

(c) (5 points) Derive the bias as a function of X and some fixed test point x.

(d) (5 points) Derive the variance term as a function of A.

(e) (5 points) Now assuming the data are one-dimensional, the training dataset consists of two samples

*₁ = 0.1 and 2 = 1, and the test sample x = 0.75. The true parameter 35= -1, 3₁ = 1, the

noise variance is given by o2 = 1. Plot the MSE (Bias square plus variance) as a function of the

regularization parameter X.

(f) (5 points) Now change the test sample to be a x = 0.01, and keep everything else to be the

same as in the previous question. Plot the MSE (Bias square plus variance) as a function of the

regularization parameter A, and comment on the difference from the previous result.