In the least-squares method, we square residuals and solve the corresponding optimization problem analytically. We should wonder what is so special with squared residuals. We might just as well take the absolute values of the residuals and solve
Another noteworthy point is that in so doing we are essentially considering average values of squared or absolute residuals; please note again that minimizing the sum or the average of squared residuals is the same thing, as dividing the function by n does not change the optimal solution. When minimizing an average, there might be a good fit for most observations, but a rather large discrepancy for a very few ones. These could be outliers that can be omitted, but if this is not acceptable, an alternative fitting approach is to minimize the worst-case residual:
Table 10.1 Calculations for Example 10.1.
This is a min–max problem, where the decision variables are a and b; for given values of slope and intercept, we get a different set of n residuals, and we pick the largest one in absolute value. The aim is minimizing the maximum residuals with respect to regression coefficients.4
In principle, there is nothing wrong with these alternative fitting models. In forecasting, we do consider mean absolute deviations, as we will discover. In function approximation, there is a whole theory concerned with the minimization of the maximum approximation error. The real trouble stems from the fact that the two models described above do not lend themselves to a closed-formula treatment, since the absolute value is not a differentiable function. We will see in Section 12.2 how to transform them into linear programming problems, which can be readily solved by commercially available software packages. However, we just get a numerical solution precluding any further interpretation. On the contrary, a closed formula allows us to cast least-squares fitting within a proper statistical framework, which is essential, as pointed out in the next section.
Leave a Reply