Solving overdetermined system by QR decomposition












2












$begingroup$


I need to solve $Ax=b$ in lots of ways using QR decomposition.



$$A = begin{bmatrix}
1 & 1 \
-1 & 1 \
1 & 2
end{bmatrix}, b = begin{bmatrix}
1 \
0 \
1
end{bmatrix}$$



This is an overdetermined system. That is, it has more equations than needed for a unique solution.



I need to find $min ||Ax-b||$. How should I solve it using QR?



I know that QR can be used to reduce the problem to
$$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



but what do I do after this?






linear-algebra numerical-methods numerical-linear-algebra







share|cite|improve this question









$endgroup$

















    2












    $begingroup$


    I need to solve $Ax=b$ in lots of ways using QR decomposition.



    $$A = begin{bmatrix}
    1 & 1 \
    -1 & 1 \
    1 & 2
    end{bmatrix}, b = begin{bmatrix}
    1 \
    0 \
    1
    end{bmatrix}$$



    This is an overdetermined system. That is, it has more equations than needed for a unique solution.



    I need to find $min ||Ax-b||$. How should I solve it using QR?



    I know that QR can be used to reduce the problem to
    $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



    but what do I do after this?






    linear-algebra numerical-methods numerical-linear-algebra







    share|cite|improve this question









    $endgroup$















      2












      2








      2





      $begingroup$


      I need to solve $Ax=b$ in lots of ways using QR decomposition.



      $$A = begin{bmatrix}
      1 & 1 \
      -1 & 1 \
      1 & 2
      end{bmatrix}, b = begin{bmatrix}
      1 \
      0 \
      1
      end{bmatrix}$$



      This is an overdetermined system. That is, it has more equations than needed for a unique solution.



      I need to find $min ||Ax-b||$. How should I solve it using QR?



      I know that QR can be used to reduce the problem to
      $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



      but what do I do after this?






      linear-algebra numerical-methods numerical-linear-algebra







      share|cite|improve this question









      $endgroup$




      I need to solve $Ax=b$ in lots of ways using QR decomposition.



      $$A = begin{bmatrix}
      1 & 1 \
      -1 & 1 \
      1 & 2
      end{bmatrix}, b = begin{bmatrix}
      1 \
      0 \
      1
      end{bmatrix}$$



      This is an overdetermined system. That is, it has more equations than needed for a unique solution.



      I need to find $min ||Ax-b||$. How should I solve it using QR?



      I know that QR can be used to reduce the problem to
      $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



      but what do I do after this?






      linear-algebra numerical-methods numerical-linear-algebra




      linear-algebra numerical-methods numerical-linear-algebra






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked 3 hours ago









      Guerlando OCsGuerlando OCs

      7821856




      7821856






















          2 Answers
          2






          active

          oldest

          votes


















          5












          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$. Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q$ is orthogonal, so:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by its left inverse you get



          $$Rx=Q^T b.$$



          This can now be solved quickly by computing $Q^T b$ and then solving the triangular system by back-substitution.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$













          • $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            3 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            3 hours ago





















          1












          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$














            Your Answer





            StackExchange.ifUsing("editor", function () {
            return StackExchange.using("mathjaxEditing", function () {
            StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
            StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
            });
            });
            }, "mathjax-editing");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "69"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            noCode: true, onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            2 Answers
            2






            active

            oldest

            votes








            2 Answers
            2






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            5












            $begingroup$

            The most straightforward way I know is to pass through the normal equations:



            $$A^T A x = A^T b$$



            and substitute in the $QR$ decomposition of $A$. Thus you get



            $$R^T Q^T Q R x = R^T Q^T b.$$



            But $Q$ is orthogonal, so:



            $$R^T R x = R^T Q^T b.$$



            If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by its left inverse you get



            $$Rx=Q^T b.$$



            This can now be solved quickly by computing $Q^T b$ and then solving the triangular system by back-substitution.



            For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






            share|cite|improve this answer











            $endgroup$













            • $begingroup$
              Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
              $endgroup$
              – Martin Argerami
              3 hours ago










            • $begingroup$
              @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
              $endgroup$
              – Ian
              3 hours ago


















            5












            $begingroup$

            The most straightforward way I know is to pass through the normal equations:



            $$A^T A x = A^T b$$



            and substitute in the $QR$ decomposition of $A$. Thus you get



            $$R^T Q^T Q R x = R^T Q^T b.$$



            But $Q$ is orthogonal, so:



            $$R^T R x = R^T Q^T b.$$



            If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by its left inverse you get



            $$Rx=Q^T b.$$



            This can now be solved quickly by computing $Q^T b$ and then solving the triangular system by back-substitution.



            For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






            share|cite|improve this answer











            $endgroup$













            • $begingroup$
              Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
              $endgroup$
              – Martin Argerami
              3 hours ago










            • $begingroup$
              @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
              $endgroup$
              – Ian
              3 hours ago
















            5












            5








            5





            $begingroup$

            The most straightforward way I know is to pass through the normal equations:



            $$A^T A x = A^T b$$



            and substitute in the $QR$ decomposition of $A$. Thus you get



            $$R^T Q^T Q R x = R^T Q^T b.$$



            But $Q$ is orthogonal, so:



            $$R^T R x = R^T Q^T b.$$



            If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by its left inverse you get



            $$Rx=Q^T b.$$



            This can now be solved quickly by computing $Q^T b$ and then solving the triangular system by back-substitution.



            For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






            share|cite|improve this answer











            $endgroup$



            The most straightforward way I know is to pass through the normal equations:



            $$A^T A x = A^T b$$



            and substitute in the $QR$ decomposition of $A$. Thus you get



            $$R^T Q^T Q R x = R^T Q^T b.$$



            But $Q$ is orthogonal, so:



            $$R^T R x = R^T Q^T b.$$



            If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by its left inverse you get



            $$Rx=Q^T b.$$



            This can now be solved quickly by computing $Q^T b$ and then solving the triangular system by back-substitution.



            For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.







            share|cite|improve this answer














            share|cite|improve this answer



            share|cite|improve this answer








            edited 2 hours ago

























            answered 3 hours ago









            IanIan

            69.1k25392




            69.1k25392












            • $begingroup$
              Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
              $endgroup$
              – Martin Argerami
              3 hours ago










            • $begingroup$
              @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
              $endgroup$
              – Ian
              3 hours ago




















            • $begingroup$
              Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
              $endgroup$
              – Martin Argerami
              3 hours ago










            • $begingroup$
              @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
              $endgroup$
              – Ian
              3 hours ago


















            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            3 hours ago




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            3 hours ago












            $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            3 hours ago






            $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            3 hours ago













            1












            $begingroup$

            Note that $Rx$ has the form
            $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
            , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
            then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



            Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
            $$ R_1x = PQ^{-1}b$$
            $$ x = (R_1)^{-1}PQ^{-1}b$$






            share|cite|improve this answer









            $endgroup$


















              1












              $begingroup$

              Note that $Rx$ has the form
              $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
              , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
              then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



              Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
              $$ R_1x = PQ^{-1}b$$
              $$ x = (R_1)^{-1}PQ^{-1}b$$






              share|cite|improve this answer









              $endgroup$
















                1












                1








                1





                $begingroup$

                Note that $Rx$ has the form
                $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
                , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
                then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



                Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
                $$ R_1x = PQ^{-1}b$$
                $$ x = (R_1)^{-1}PQ^{-1}b$$






                share|cite|improve this answer









                $endgroup$



                Note that $Rx$ has the form
                $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
                , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
                then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



                Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
                $$ R_1x = PQ^{-1}b$$
                $$ x = (R_1)^{-1}PQ^{-1}b$$







                share|cite|improve this answer












                share|cite|improve this answer



                share|cite|improve this answer










                answered 2 hours ago









                Adam LatosińskiAdam Latosiński

                5388




                5388






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Mathematics Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    Use MathJax to format equations. MathJax reference.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    Popular posts from this blog

                    Why is a white electrical wire connected to 2 black wires?

                    Image
                    1 I just opened my ceiling light to replace it with another mount. I see a white wire connected to 2 black wires, is that right? The middle switch controls this ceiling light. electrical ceiling light share | improve this question asked 4 hours ago konyak konyak 108 4 New contributor konyak is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. ...
                    Read more

                    Waikiki

                    Image
                    Denne artikkelen kan ha godt av ein språkvask , som reinskar opp målføringa og/eller innfører same språkstilen overalt. Denne artikkelen kan ha godt av ei opprydding for å nå ein høgare standard og/eller for å verta i tråd med standardoppsettet. Sjå korleis du redigerer ei side og stilmanualen for hjelp. Waikiki Land   USA Stat Hawaii Stad Honolulu Wikimedia Commons: Waikiki Utsyn over Waikiki Waikīkī er ein bydel i Honolulu, sør på øya Oahu, som er ei av Hawaii-øyane. Waikīkīstranda er ein av dei meste berømte strendene i verda. Waikīkī strekkjer seg langs stillehavskysten og har ei rekkje velrenommerte hotell. Bydelen er Hawaii-turismens høyborg. På Waikīkī Beach står ei statue av den lokale sportsstjerna Duka Kahanamoku. Kjelder | Denne artikkelen bygger på «Waikiki» frå Wikipedia på bokmål, den 4. november 2018. Denne artikkelen manglar kjelder eller referansar . Hjelp Wikipedia med å finna truverdige kjelder...
                    Read more

                    What are all the squawk codes?

                    Image
                    2 $begingroup$ I love aviation and I love learning new things about it. I was wondering if anyone knew all of the squawk codes and in particular what squawk 7500 means. squawk-codes share | improve this question edited 15 mins ago Jamiec ♦ 14.8k 4 60 91 asked 5 hours ago Zac Callaghan Zac Callaghan 11 2 New contributor Zac Callaghan is a new co...
                    Read more