Matrix Algebra As A Tool
Hadi, A. S. (1996)
Belmont, CA:
Duxbury Press.
ISBN: 0-534-23712-6
Preface
I teach intermediate and advanced applied statistics courses, such as applied regression and multivariate analysis, at both the undergraduate and graduate levels. Although most of my students are not statistics majors, their various fields of study demand many mathematical tools. Matrix algebra in particular is necessary for a thorough understanding of research methods courses in other disciplines.
Although the students in my classes are highly motivated, they often lack the mathematical skills needed to understand the material. To specify matrix algebra as a prerequisite would be an easy solution for me as a professor but an unsatisfactory one as an educator, because it would prevent or discourage many students who should, on the contrary, be encouraged to take such courses. Therefore, rather than specify matrix algebra as a prerequisite, I have tried two alternative solutions. The first was to present a brief review of the necessary matrix algebra tools and refer students to one of the many standard books on the topic. Matrix algebra books, however, are typically reference books written at a higher mathematical level, and are inaccessible for nonmajors. As a result, many students dropped out, and many who stayed were very confused.
Alternatively, I tried devoting a good portion of the course to matrix algebra. The obvious disadvantage of this solution, however, is that it does not leave enough time to cover the statistics topics that are the real point of the course. I finally compromised, devoting a relatively short time to matrix algebra and then referring students to some lecture notes I had specifically written for them. These notes - expanded, revised, and reorganized - evolved into this book.
The main purpose of the book is to present matrix algebra as a tool for students and researchers in various fields who find it necessary for their own work. It is intended to provide a basic understanding in language they can easily understand. The book's informal style has forced me on occasion to be mathematically imprecise, but this seems to me a small price to pay for increased accessibility. Some students will be able to read this book on their own, and much of it will require little instruction; the remaining part, however, may require a moderate amount of instruction. I hope that even students with prior courses in matrix algebra will find this book helpful.
The book can be used as a supplement to many courses, such as applied regression analysis, applied multivariate analysis, econometrics, biometrics, and other research methods courses in various disciplines such as sociology, psychology, economics, accounting, finance, marketing, and so on. The book may also serve as the primary text for a short course in matrix algebra that might be offered concurrently with the above-mentioned courses, for instance, or during the summer semester or other open periods in the academic calendar. The book may also serve as a text in a more rigorous matrix algebra course, although this was not the original intention in writing the book. In this case, the instructor may wish to provide proofs of some results and assign the proofs of other results as exercises.
I assume no previous acquaintance with the subject. Except for a very brief mention of the word derivative in Section 8.3 (which is an application section), calculus is not invoked. Although the book addresses many advanced topics (e.g., Section 6.3 and Chapters 8-11), these topics are covered at an elementary level and are illustrated by numerical examples.
As the title indicates, the emphasis is on practical rather than theoretical considerations, and much of the book is devoted to applications. Indeed, Chapters 8, 10, and 11 are devoted entirely to applications. Matrix notation, vocabulary, and concepts are emphasized. Although some of the results are derived, nearly all are stated without proof. Rather, the emphasis is on the interpretations and implications of the results and, in lieu of proofs, the results are illustrated and verified by examples.
Although there is a great temptation to do so in matrix algebra, I have tried not to emphasize matrix computations. Numerical examples unavoidably involve some calculations, but the examples are primarily designed to illustrate the concepts rather than matrix manipulations. For this reason, methods which are purely computational are relegated to an appendix to the chapter in which they appear. I use elementary methods for calculations, which are simple and easily understood but are not necessarily preferred in terms of computational efficiency or numerical accuracy. In most cases, computations are illustrated using small matrices. Most real applications, however, involve larger matrices, many of which are not practical to compute by hand. Whenever possible, professionally written computer programs should be used for matrix calculations in real-life applications. There are several main frame and personal computer packages aimed specifically at matrix calculations. Even spreadsheet programs now include some basic matrix calculations. I do not refer to any specific software or hardware because a general book like this should not be tied to specific software or hardware. Therefore, instructors may use any software package of their choice.
Geometry and matrix algebra are intimately connected, and matrix concepts can often be easily understood by relating them to their geometric counterparts. I tried to make this connection as early as possible in the book, beginning with Chapter 4.
Exercises are given at the end of each chapter. The exercises are not really optional, as many of them involve concepts supplementary to, but not covered in, the text. Readers should be able to do the computations by hand for all small matrices; otherwise all numerical work should be done by a computer. Also, readers are urged to substitute their own real data for the small matrices in the exercises and perform the calculations using a computer.
Vocabulary and concepts are defined and explained when they are first encountered, and key words are italicized. Some material, marked by an asterisk, is either more advanced, more theoretical, or more difficult to read. These parts can be skimmed or skipped at first reading. Except for Section 8.2 and the parts marked with asterisks, the first nine chapters constitute a minimum background and should not be skipped. The remaining material consists of either applications or advanced topics that can be read when the need arises. Nevertheless, it is recommended that the chapters be read in the order they are presented.
Two other chapters I originally intended to include, on other matrix decompositions (e.g., Cholesky, Q-R, and L-D-U) and on basic matrix differential calculus, were omitted as beyond the nature and scope of this book.
I am grateful to many of my students for the comments they have given me and to William Gould (Stata Corporation) for sharing with me his own handwritten notes on some topics of matrix algebra. Paul Green, Gary McDonald, and Mary Roybal have done a superb job editing the manuscript. I thank the following colleagues at Cornell University for reading and commenting on an early draft of the book: Jody Enk, Robert Hutchens, Matthew Hutcheson, Hyeseon Joo, Shayle Searle, Paul Velleman, and John Walker. I have also benefited from the comments of many colleagues at other universities. I am indebted to Gulhan Alpargu (McGill University), Bruce Barrett (University of Alabama), Samprit Chatterjee (New York University), Daniel Coleman (Carlos III University of Madrid), Mohammed El-Saidi (Ferris State University), Jose Garrido (University of Concordia), J. Brian Gray (University of Alabama), Brian Jersky (Sonoma State University), Robert Ling (Clemson University), Alberto Luceño (Cantabria University, Spain), Hans Nyquist (University of , Sweden), Daniel Peña (Carlos III University of Madrid), Jaime Puig-Pey (Cantabria University, Spain), Karen Shane (New York University), Simo Puntanen (University of Tampere, Finland), Gad Saad (University of Concordia), Gary Simon (New York University), Jeffrey Simonoff (New York University), Mun Son (University of Vermont), Lenn Stefanski (North Carolina State University), George Styan (McGill University), and Mohammed Youssef (Norfolk State University). Parts of the book were written while I was visiting the Department of Applied Mathematics and Computing Sciences of the University of Cantabria. I am grateful to Enrique Castillo, who made such a visit both enjoyable and productive
Ali S. Hadi
Ithaca, New York
June 1995
Table of Contents
Preface v
1. Introduction 1
-
1.1. Why Matrix Algebra? 1
1.2. Some Definitions and Notation 2
-
1.2.1., What Is a Matrix? 2
1.2.2., Scalar Algebra and Matrix Algebra 4
1.2.3. Matrix Equality 5
1.2.4. Matrix Transpose 5
-
1.3.1. Square Matrices 6
1.3.2. Symmetric Matrices 6
1.3.3. Skew-Symmetric Matrices 7
1.3.4. Triangular Matrices 7
1.3.5. Diagonal Matrices 8
1.3.6. Identity Matrices 8
1.3.7. Unit Vectors 8
1.3.8. Matrices of Ones 8
1.3.9. Null Matrices 9
1.3.10. Constant Matrices 9
1.3.11. Partitioned Matrices 9
2. Some Matrix Calculations 13
-
2.1. Matrix Addition and Subtraction 13
2.2. Element-Wise Product 14
2.3. Matrix Inner Product 15
-
2.3.1. Definitions and Examples 15
2.3.2. Special Types of Matrix Products 18
2.3.3. Some Properties of Matrix Products 21
Exercises 26
3. Linear Dependence and Independence 31
-
3.1. Introduction 31
3.2. Linear Dependence of Two Vectors 32
3.3. Linear Dependence of a Set of Vectors 33
3.4. Rank of a Matrix 37
3.5. Elementary Row Operations 38
3.6. Elementary Matrices 39
3.7. Elementary Column Operations 40
3.8. Permutation Matrices 40
Appendix: Reduction to Row-Echelon Form 41
Exercises 46
4. Vector Geometry 49
-
4.1. Introduction 49
4.2. Algebraic and Graphic Views of Vectors 49
4.3. Geometric Views of Vectors 49
-
4.3.1. Length or Magnitude of a Vector 51
4.3.2. Angle Between Two Vectors 52
4.5. Geometry of Vector Algebra 55
-
4.5.1. Geometry of Vector Addition 55
4.5.2. Geometry of Vector Subtraction 56
4.5.3. Geometry of Multiplying a Vector by a Scalar 58
4.5.4. Geometry of Multiplying a Vector by a Matrix 61
Exercises 67
5. Three Matrix Reductions 71
-
5.1. Trace 71
5.2. Determinant 73
5.3. Vector Norms 76
Exercises 79
6. Matrix Inversion 81
-
6.1. Element-Wise Division 81
6.2. The Regular Inverse 82
-
6.2.1. Definitions 82
6.2.2. Existence of Matrix Inverse 83
6.2.3. An Algorithm for Computing Matrix Inverse 84
6.2.4. Properties of Matrix Inverse 84
-
6.3.1. Definitions 88
6.3.2. Existence and Uniqueness of G-Inverse 88
6.3.3. An Algorithm for Computing a G-Inverse 90
6.3.4. Properties of G-Inverse 91
6.3.5. Particular Types of G-Inverse 91
6.3.6. Full-Rank Factorization 92
Exercises 96
7. Linear Transformation 101
-
7.1. Introduction 101
7.2. Definition 101
* 7.3. Orthogonal Rotation 106
* 7.4. Oblique Rotation 108
* 7.5. Orthogonal Projection 108
7.6. Singular and Nonsingular Transformations 114
Exercises 115
8. Some Applications 117
-
8.1. Simultaneous Linear Equations 117
-
8.1.1. Definitions 117
8.1.2. Graphical Solution 118
8.1.3. Numerical Solution 120
8.1.4. The Case of Square Coefficient Matrix 122
8.1.5. Homogeneous Systems 123
-
8.2.1. The Mean Vector 124
8.2.2. The Variance-Covariance Matrix 124
8.2.3. The Correlation Matrix 129
-
8.3.1. Introduction 130
8.3.2. Least Squares Estimates 132
8.3.3. An Illustrative Example 133
8.3.4. The Fitted Values and Residual Vectors 134
-
8.4.1. Adding Columns to a Data Matrix 136
8.4.2. Adding Rows to a Data Matrix 138
Exercises 142
9. Eigenvalues and Eigenvectors 147
-
9.1. Definition and Derivation 147
9.2. Some Properties 151
* 9.3. Singular Value Decomposition 156
Exercises 157
10. Ellipsoids and Distances 161
-
10.1. Ellipsoids and Spheres 161
10.2. Distances 169
-
10.2.1. The Euclidean Distance 170
10.2.2. The Elliptical Distance 172
Exercises 181
11. Additional Applications 185
-
* 11.1. Matrix Norms 185
-
11.1.1. The Frobenius Norm 185
11.1.2. The Matrix P-Norm 186
11.3. Classification of Quadratic Forms 191
* 11.4. Markov Chains 194
* 11.5. Principal Component Analysis 197
* 11.6. Factor Analysis 199
Exercises 201
Bibliography 205
Index 207
Back Cover
Matrix Algebra As a Tool provides students with a practical, applied approach to matrix algebra. It is intended for use in a one-term course in matrix algebra, or it can provide a matrix algebra background for applied courses such as regression analysis, multivariate analysis, econometrics, and other statistics and research methods courses in business, the social sciences, and other disciplines. Matrix Algebra As a Tool gives instructors a useful, flexible resources that avoids detailed discussions of mathematical proofs. Students will appreciate the book's clear, straightforward approach.
ABOUT THE AUTHOR
Ali S. Hadi (Ph.D., New York University) is Vice Provost and Professor of Mathematical, Statistical, and Computational Sciences at the American University in Cairo, Egypt. He is a Stephen H. Weiss Presidential Fellow and Professor Emeritus at Cornell University. He is a fellow of the American Statistical Association and an elected member of the International Statistical Institute. He is co-author of four other books:
- Regression Analysis by Example, Fourth Edition, 2006, New York: John Wiley & Sons.
- Extreme Value and Related Models in Engineering and Science Applications, 2004, New York: John Wiley & Sons.
- Expert Systems and Probabilistic Network Models, 1997, New York: Springer-Verlag.
- Sensitivity Analysis In Linear Regression, 1988, New York: John Wiley & Sons.