In1801,ItalianastronomerGiuseppePiazzidiscoveredthefirstasteroidCeres.After40daysoftracking,PiazzilostthepositionofCeresbecauseCeresmovedtothebackofthesun.ThenscientistsallovertheworldusedPiazzi'sobservationaldatatostartsearchingforCeres,butsearchingforCeresbasedontheresultsofmostpeople'scalculationshasnoresults.TheorbitofCerescalculatedbyGauss,whowasonly24yearsold,wasconfirmedbytheobservationsoftheAustrianastronomerHeinrichAlbers,allowingtheastronomicalcommunitytopredicttheprecisepositionofCeres.ThesamemethodalsoproducedmanyastronomicalresultssuchasHalley'sComet.ThemethodusedbyGaussistheleastsquaresmethod,whichwaspublishedinhisbook"OntheMovementofCelestialBodies"in1809.Infact,theFrenchscientistLegendreindependentlyinventedthe"leastsquaresmethod"in1806,butitwasunknownbecauseitwasunknowntotheworld.
In1829,Gaussprovidedproofthattheoptimizationeffectoftheleastsquaresmethodisstrongerthanothermethods.
Definition
Theleastsquaremethod(alsoknownastheleastsquaremethod)isamathematicaloptimizationtechnique.Itfindsthebestfunctionmatchofthedatabyminimizingthesumofsquaresoftheerror.Theleastsquaresmethodcanbeusedtoeasilyobtainunknowndataandminimizethesumofsquarederrorsbetweentheobtaineddataandtheactualdata.
Theleastsquaremethodcanalsobeusedforcurvefitting,andsomeotheroptimizationproblemscanalsobeexpressedbytheleastsquaremethodbyminimizingenergyormaximizingentropy.
Basicidea
Theleastsquaremethodisthemostcommonlyusedmethodtosolvecurvefittingproblems.Thebasicideais:Let
Amongthem,isasetoflinearlyindependentfunctionsselectedinadvance,istheundeterminedcoefficient,andthefittingcriterionistominimizethesumofsquaresofthedistancebetweenand,Calledtheleastsquarescriterion.
Basicprinciple
Suppose(x,y)isapairofobservations,andsatisfiesthefollowingtheoreticalfunctions:
whereisaparametertobedetermined.
Inordertofindtheoptimalestimatedvalueoftheparameterofthefunction,foragivengroup(usuallysection>)Observationdata,solvetheobjectivefunction
Taketheminimumparameter.Thistypeofproblemtobesolvediscalledaleastsquaresproblem,andthegeometriclanguageofthemethodtosolvethisproblemiscalledleastsquaresfitting.
Forunconstrainedoptimizationproblems,thegeneralformoftheleastsquaresmethodis:
whereCalledtheresidualfunction.Whenisalinearfunctionof,itiscalledalinearleastsquaresproblem,otherwiseitiscalledanonlinearleastsquaresproblem.
Leastsquaresoptimizationproblem
Inunconstrainedoptimizationproblems,therearesomeimportantspecialcases,suchastheobjectivefunctionconsistingofthesumofthesquaresofseveralfunctions.ThistypeoffunctioncangenerallybeWrittenas:
Amongthem,usuallyrequiresm≥n.Weminimizetheproblemofthistypeoffunction:
Itiscalledtheleastsquaresoptimizationproblem.Leastsquaresoptimizationisaspecialkindofoptimizationproblem.
Thecharacteristicsoftheleastsquaresestimator
Accordingtothesampledata,theleastsquaresestimatorcanbeusedtoobtaintheestimatorofthesimplelinearregressionmodelparameters.Buthowclosetheestimatorparameteristotheoveralltrueparameter?Whetherthereareotherbetterestimationformulas?Thisinvolvestheleastsquaresestimationformulaortheminimumvariance(orbest)(Best)oftheestimator,linearity(Linear)Andunbiased(Unbiased),referredtoasBLUcharacteristics.Thisisthemainreasonforthewidespreaduseofordinaryleastsquarestoestimateeconometricmodels.Thefollowingprovesthattheordinaryleastsquaresestimatorhastheabovethreecharacteristics.
1.Linearcharacteristic
Theso-calledlinearcharacteristicmeansthattheestimatoristhelinearfunctionofthesampleobservationvalue,thatis,thelinearcombinationoftheestimatorandtheobservationvalue.
2.Unbiasedness
Unbiasednessmeansthattheexpectedvaluesofparameterestimatorsareequaltotheoveralltrueparameters.
3.Minimumvarianceproperty
Theso-calledminimumvariancepropertyreferstotheminimumvarianceoftheestimatorcomparedwiththeestimatorobtainedbyothermethods,thatis,thebest.Theminimumvarianceisalsocalledeffectiveness.ThispropertyisthefamousGauss-Markov(Gauss-Markov)theorem.Thistheoremstatesthattheordinaryleastsquaresestimatoristhebestcomparedwithanylinearunbiasedestimatorobtainedbyothermethods.