These are difficult mathematical questions. They are arising from
real applications such as fraud
detection, arbitrage and scoring systems. If you have interesting answers
questions, feel free to email us your comments or solution. The best answers
will be published here. Companies and Organizations interested in submitting problems should
Iterative Algorithm for Linear Regression
I am trying to solve the regression Y=AX where Y is the response, X
the input, and A the regression coefficients. I came up with the
following iterative algorithm:
Ak+1 = cYU + Ak (I-cXU),
- c is an arbitrary constant
- U is an arbitrary matrix such that YU has same dimension as A. For
instance U = transposed(X) works.
- A0 is the initial estimate for A. For instance A0 is the
correlation vector between the independent variables and the response.
- What are the conditions for convergence? Do I have convergence if
and only if the largest eigenvalue (in absolute value) of the matrix I-cXU is strictly
less than 1?
- In case of convergence, will it converge to the solution of the
regression problem? For instance, if c=0, the algorithm converges, but not to the
solution. In that case, it converges to A0.
- n: number of independent variables
- m: number of observations
Why using an iterative algorithm instead of the traditional solution?
- A: (1,n) (one row, n columns)
- I: (n,n)
- X: (n,m)
- U: (m,n)
- Y: (1,m)
Note: this question is not related to the ridge regression algorithm described here.
- We are dealing with an ill-conditioned problem; most independent variables are highly correlated.
- Many solutions (as long as the regression coefficients are positive) provide a very good fit, and the global optimum is not that much better than a solution where all regression coefficients are equal to 1.
- The plan is to use an iterative algorithm to start at iteration #1 with an approximate solution that has interesting properties, then move to iteration #2 to improve a bit, then stop.
- From Ray Koopman
No need to apologize for not using "proper" weights. See
Dawes, Robyn M. (1979). The robust beauty of improper linear
models in decision making. American Psychologist, 34, 571-582.