Chapter 20. Numerical Methods for computing OLS Estimates

528



20. NUMERICAL METHODS FOR COMPUTING OLS ESTIMATES



X = QR, the normal equations read

(20.1.1)



ˆ

X Xβ = X y



(20.1.2)



ˆ

R Q QRβ = R Q y



(20.1.3)



ˆ

R Rβ = R Q y



(20.1.4)



ˆ

Rβ = Q y



This last step can be made because R is nonsingular. (20.1.4) is a triangular system of

equations, which can be solved easily. Note that it is not necessary for this procedure

to compute the matrix X X, which is a big advantage, since this computation is

numerically quite unstable.

Problem 261. 2 points You have a QR-decomposition X = QR, where Q Q =

ˆ

I, and R is upper triangular and nonsingular. For an estimate of V [β] you need

−1

(X X) . How can this be computed without computing X X? And why would

you want to avoid computing X X?

Answer. X X = R Q QR = R R, its inverse is therefore R−1 R−1 .



20.1. QR DECOMPOSITION



529



Problem 262. Compute the QR decomposition of





1 1 2

1 5 −2



(20.1.5)

X=

1 1 0 

1 5 −4

Answer.







(20.1.6)



1

1 1

Q= 

2 1

1



−1

1

−1

1







1

1 

−1

−1



1

R=2 0

0



3

2

0



−1

−2

1



How to get it? We need a decomposition

(20.1.7)



x1



x2



x3 = q 1



q2



r11

0

0



q3



r12

r22

0



r13

r23

r33



where q 1 q 1 = q 2 q 2 = q 3 q 3 = 1 and q 1 q 2 = q 1 q 3 = q 2 q 3 = 0. First column: x1 = q 1 r11 and

q 1 must have unit length. This gives q 1 = 1/2 1/2 1/2 1/2 and r11 = 2. Second column:

(20.1.8)



x2 = q 1 r12 + q 2 r22



and q 1 q 2 = 0 and q 2 q 2 = 1. Premultiply (20.1.8) by q 1 to get q 1 x2 = r12 , i.e., r12 = 6.

Thus we know q 2 r22 = x2 − q 1 · 6 = −2



2



−2



2



. Now we have to normalize it, to get



530



20. NUMERICAL METHODS FOR COMPUTING OLS ESTIMATES



q 2 = −1/2 1/2 −1/2 1/2 and r22 = 4. The rest remains a homework problem. But I am

not sure if my numbers are right.







1 3 −1

Problem 263. 2 points Compute trace and determinant of 0 2 −2. Is

0 0 1

this matrix symmetric and, if so, is it nonnegative definite? Are its column vectors

linearly dependent? Compute the matrix product







1 −1 1 

1 3 −1

1 1

1 







(20.1.9)

1 −1 −1 0 2 −2

0 0 1

1 1 −1



20.2. The LINPACK Implementation of the QR Decomposition

This is all we need, but numerically it is possible to construct, without much

additional computing time, all the information which adds the missing orthogonal

columns to Q. In this way Q is square and R is conformable with X. This is

sometimes called the “complete” QR-decomposition. In terms of the decomposition



20.2. THE LINPACK IMPLEMENTATION OF THE QR DECOMPOSITION



531



above, we have now

(20.2.1)



X= Q



S



R

O



For every matrix X one can find an orthogonal matrix Q such that Q X has

zeros below the diagonal, call that matrix R. Alternatively one may say: every

matrix X can be written as the product of two matrices QR, where R is conformable

with X and has zeros below the diagonal, and Q is orthogonal.

To prove this, and also for the numerical procedure, we will build Q as the

product of several orthogonal matrices, each converting one column of X into one

with zeros below the diagonal.

First note that for every vector v, the matrix I − v 2 v vv is orthogonal. Given

X, let x be the first column of √

X. If x = o, then go on to the next column.





x11 + σ x x





x21





Otherwise choose v = 

, where σ = 1 if x11 ≥ 0 and σ = −1

.

.





.

xn1

otherwise. (Mathematically, either σ − +1 or σ = −1 would do; but if one gives σ

the same sign as x11 , then the first element of v gets largest possible absolute value,



532



20. NUMERICAL METHODS FOR COMPUTING OLS ESTIMATES



which improves numerical accuracy.) Then

√

(20.2.2)

v v = (x2 + 2σx11 x x + x x) + x2 + · · · + x2

11

21

n1

√

(20.2.3)

= 2(x x + σx11 x x)

√

(20.2.4)

v x = x x + σx11 x x

therefore 2v x/v v = 1, and



(20.2.5)



 √



−σ x x





0

2





(I −

vv )x = x − v = 

.

.

.

v v





.

0



Premultiplication of X by I − v 2 v vv gets therefore the first column into the desired

shape. By the same principle one can construct a second vector w, which has a zero

in the first place, and which annihilates all elements below the second element in the

second column of X, etc. These successive orthogonal transformations will convert

X into a matrix which has zeros below the diagonal; their product is therefore Q .

The LINPACK implementation of this starts with X and modifies its elements in

place. For each column it generates the corresponding v vector and premultipies the

matrix by I − v 2 v vv . This generates zeros below the diagonal. Instead of writing



20.2. THE LINPACK IMPLEMENTATION OF THE QR DECOMPOSITION



533



the zeros into that matrix, it uses the “free” space to store the vector v. There is

almost enough room; the first nonzero element of v must be stored elsewhere. This

is why the QR decomposition in Splus has two main components: qr is a matrix

like a, and qraux is a vector of length ncols(a).

LINPACK does not use or store exactly the same v as given here, but uses

√

u = v/(σ x x) instead. The normalization does not affect the resulting orthogonal

transformation; its advantage is that the leading element of each vector, that which

is stored in qraux, is at the same time equal u u/2. In other words, qraux doubles

up as the divisor in the construction of the orthogonal matrices.

In Splus type help(qr). At the end of the help file a program is given which

shows how the Q might be constructed from the fragments qr and qraux.



CHAPTER 21



About Computers

21.1. General Strategy

With the fast-paced development of computer hardware and software, anyone

who uses computers profesionally needs a strategy about how to allocate their time

and money for hardware and software.

21.1.1. Operating System. In my view, there are two alternatives today:

either do everything in Microsoft Windows and other commercial software, or use

GNU/Linux, the free unix operating system together with the free software built on

top of it, see www.linux.org, in addition to Microsoft Windows. I will argue here for

the second route. It is true, GNU/Linux has a steeper learning curve than Windows,

but this also means that you have a more powerful tool, and serious efforts are under

535



536



21. ABOUT COMPUTERS



way to make GNU/Linux more and more user friendly. Windows, on the other hand,

has the following disadvantages:

• Microsoft Windows and the other commercial software are expensive.

• The philosophy of Microsoft Windows is to keep the user in the dark about

how the computer is working, i.e., turn the computer user into a passive

consumer. This severely limits the range of things you can do with your

computer. The source code of the programs you are using is usually unavailable, therefore you never know exactly what you are doing and you cannot

modify the program for your own uses. The unavailability of source code

also makes the programs more vulnerable to virus attacks and breakins.

In Linux, the user is the master of the computer and can exploit its full

potential.

• You spend too much time pointing and clicking. In GNU/Linux and other

unix systems, it is possible to set up menus too,m but everything that can

be done through a menu can also be done on the command line or through

a script.

• Windows and the commercial software based on it are very resource-hungry;

they require powerful computers. Computers which are no longer fast and

big enough to run the latest version of Windows are still very capable to

run Linux.



21.1. GENERAL STRATEGY



537



• It is becoming more and more apparent that free software is more stable

and of higher quality than commercial software. Free software is developed

by programmers throughout the world who want good tools for themselves.

• Most Linux distributions have excellent systems which allows the user to

automatically download always the latest versions of the software; this automates the tedious task of software maintenance, i.e., updating and fitting

together the updates.

Some important software is not available on Linux or is much better on Windows.

Certain tasks, like scanning, voice recognition, and www access, which have mass

markets, are better done on Microsoft Windows than on Linux. Therefore you will

probably not be able to eliminate Microsoft Windows completely; however it is possible to configure your PC so that you can run MS-Windows and Linux on it, or

to have a Linux machine be the network server for a network which has Windows

machines on it (this is more stable, faster, and cheaper than Windows NT).

There are several versions of Linux available, and the one which is most independent of commercial interests, and which is one of the most quality-conscious distributions, in my view, is Debian GNU/Linux, http://www.debian.org. The Linux

route is more difficult at the beginning but will pay off in the long run, and I recommend it especially if you are going to work outside the USA. The Salt Lake Linux



538



21. ABOUT COMPUTERS



Users Group http://www.sllug.org/index.html meets on the third Wednesday of

every month, usually on the University of Utah campus.

In order to demonstrate the usefulness of Linux I loaded Debian GNU/Linux on

an old computer with one of the early Pentium processors, which became available at

the Econ Department because it was too slow for Windows 98. It is by the window

in the Econ Computer Lab. When you log onto this computer you are in the Xwindows system. In Linux and other unix systems, the mouse usually has 3 buttons:

left, right, and middle. The mouse which comes with the computer in the computer

lab has 2 bottons: left and right, but if you press both buttons simultaneously you

get the same effect as pressing the middle button on a unix mouse.

If the cursor is in front of the background, then you will get 3 different menus by

pressing the different mouse buttons. The left mouse button gives you the different

programs, if you press both buttons at the same time you can perform operations on

the windows, and the right button gives you a list of all open windows.

Another little tidbit you need to know about unix systems is this: There are no

drives as in Microsoft Dos or Windows, but all files are in one hierarchical directory

tree. Instead of a backslash \ you have a forward slash /. In order to use the floppy

disk, you have to insert the disk in the disk drive and then give the command mount

/floppy. Then the disk is accessible to you as the contents of the directory /floppy.