
These are difficult mathematical questions. They are arising from
real applications such as fraud
detection, arbitrage and scoring systems. If you have interesting answers
to any
questions, feel free to email us your comments or solution. The best answers
will be published here. Companies and Organizations interested in submitting problems should
Email us.
Iterative Algorithm for Linear Regression
I am trying to solve the regression Y=AX where Y is the response, X
the input, and A the regression coefficients. I came up with the
following iterative algorithm:
A_{k+1} = cYU + A_{k} (IcXU),
where:
 c is an arbitrary constant
 U is an arbitrary matrix such that YU has same dimension as A. For
instance U = transposed(X) works.
 A_{0} is the initial estimate for A. For instance A_{0} is the
correlation vector between the independent variables and the response.
Questions:
 What are the conditions for convergence? Do I have convergence if
and only if the largest eigenvalue (in absolute value) of the matrix IcXU is strictly
less than 1?
 In case of convergence, will it converge to the solution of the
regression problem? For instance, if c=0, the algorithm converges, but not to the
solution. In that case, it converges to A_{0}.
Parameters:
 n: number of independent variables
 m: number of observations
Matrix dimensions:
 A: (1,n) (one row, n columns)
 I: (n,n)
 X: (n,m)
 U: (m,n)
 Y: (1,m)
Why using an iterative algorithm instead of the traditional solution?
 We are dealing with an illconditioned problem; most independent variables are highly correlated.
 Many solutions (as long as the regression coefficients are positive) provide a very good fit, and the global optimum is not that much better than a solution where all regression coefficients are equal to 1.
 The plan is to use an iterative algorithm to start at iteration #1 with an approximate solution that has interesting properties, then move to iteration #2 to improve a bit, then stop.
Note: this question is not related to the ridge regression algorithm described here.
Contributions:
 From Ray Koopman
No need to apologize for not using "proper" weights. See
Dawes, Robyn M. (1979). The robust beauty of improper linear
models in decision making. American Psychologist, 34, 571582.

