Lecture 08 - Bias-Variance Tradeoff

What people said

Recommends cmishra · 2015-05 · Cross Validated Stack Exchange · on “Calculating the variance of a model?”

I'll start with the second question: It's only possible to calculate the true variance if you have a set of every possible randomly-drawn data set $D$ (or an equivalent PDF) and every single $x$ input possible (or an equivalent PDF). This is the result of the mathematical definition of the variance of a learning model/algorithm: $E_x[E_D[g^d(x)-\bar{g}(x)]]$, where $d\in D$. ( slide source [LINK] ) This will obviously never happen. The only time when you can exactly calculate the variance is when you create your own data. Even then, both $D$ and $x$ will likely be infinite (if you have any continuous predictors or outputs), so you'll just be approximating the variance by drawing large sample

Recommends Jesper for President · 2019-12 · Cross Validated Stack Exchange · on “Expectation in Linear Regression with Least Squares”

Since you are interested in stuff like that I can recommend reading this question [LINK] and watching this video [this video] Both might help you to clarify some concepts.

Cites Peter McHale · 2017-06 · Cross Validated Stack Exchange · on “Bias-variance decomposition and independence of $X$ and $\epsilon$”

Here is a derivation of the bias-variance decomposition, in which I make use of the independence of $X$ and $\epsilon$ . True model Suppose that a target variable $Y$ and a feature variable $X$ are related via $Y = f(X) + \epsilon$ , where $X$ and $\epsilon$ are independent random variables and the expected value of $\epsilon$ is zero, $E[\epsilon] = 0$ . We can use this mathematical relationship to generate a data set $\cal D$ . Because data sets are always of finite size, we may think of $\cal D$ as a random variable, the realizations of which take the form $d = \{ (x_1,y_1), \ldots , (x_m,y_m) \}$ , where $x_i$ and $y_i$ are realizations of $X$ and $Y$ . Estimated model Machine learning u

Cites Yandle · 2020-12 · Cross Validated Stack Exchange · on “Can adding features to linear regression increase model bias?”

I think it may be possible to increase bias in a linear regression model by adding bad features, but I can't think of an example that makes sense intuitively on the basis of the bias-variance decomposition. For instance, using the formulation of bias variance decomposition here [this video] , the square bias term is $(\bar g(x) - f(x))^2$ . I can't think of a scenario where this term would increase when adding an extra feature, since I think adding a bad feature would force its associated parameter to be zero for both $\bar g$ and $f$ . Edited