16. Projection Matrices and Least Squares

What people said

Recommends Robert890 · 2014-08 · Mathematics Stack Exchange · on “Find orthogonal projection on basis of V”

You're almost there! ;-) If your orthonormal vectors for the basis are $o_1, o_2, o_3$, put them into a matrix $O$ (by columns, first column is $o_1$, second is $o_2$, and third is $o_3$), and the projection of the vector $v$ onto $V$ is $OO^T v$. Check out the wiki page: [LINK] And also this video from a linear algebra course at MIT: [this video]

Recommends ShellRox · 2018-04 · Mathematics Stack Exchange · on “Proper way to use projection matrix equation”

Introduction: This lecture [this video] explains how can linear algebraic methods be used to solve linear regression problems. From my understanding, If there is no solution to $Ax=b$, we can project $b$ into the space and get $x$ which will be the closest solution. Although $A$ must be orthogonal for the error $e = b - Ax$ (or it should be in the null space of $A^T$, which is equivalent to orthogonality) in the approximation since then the inner product of $A$ and $e$ would give us the equation. If we are working in the spaces of 3 dimensions or higher (or planes of 2 dimensions and higher), then our projection could be represented as: $P = A(A^TA)^{-1}A^T$ Problem : At this point of the lecture

Recommends saman saffari · 2020-12 · Mathematics Stack Exchange · on “MIT Statistics course: multi-dimensional matrix form for Regression”

you are confusing two different pictures that represent regression using matrices and least squares. picture number 1: you have your data points on say, a 2 dimensional plane (x,y) and you are trying to draw a line that has the "least squared" errors and those errors are quite obviously the magnitude of the line coming out of those data points perpendicular to the line you want to draw. I think you are mixing up that picture with what is drawn in here and what I'm going to elaborate on. notice, $X \hat \beta$ is a vector. to fully understand this you need to remember how a system of equations. for each data point you have y=C+Dt (I'm writing that instead of y=mx+b cause I will be using x and

Cites ShellRox · 2018-04 · Mathematics Stack Exchange · on “Solving least squares problem using partial derivatives”

Let's say we want to solve a linear regression problem by choosing the best slope and bias with the least squared errors. As example, let the points be $x=[1, 2, 3]$ and $y=[1,2,2]$ . This quadratic minimization problem can also be represented as: $||Ax-b||^2=||e||^2=e^2_1+e^2_2+e^2_3$ Linear Algebra We could solve this problem by utilizing linear algebraic methods. We could use projections. For projecting on the $0+$ dimensional subspaces. Projection equation $p = Ax = A(A^TA)^{-1}A^Tb$ could be utilized: $A^T(b-Ax)=0$ $A^TAx = A^Tb$ $x = (A^TA)^{-1} A^Tb$ We know the inner product of $A^T$ and $e=b-p=b-Ax$ is $0$ since they are orthogonal (or since $e$ is in the null space of $A^T$ ). Whic