## I. INTRODUCTION When there are some nonrandom quantities affecting a response variable, the analysis of data can be done by using a general linear model. There are some notable references about linear models such as [1-4]. In a general linear model, the response variable is composed of two parts in general; one part is the deterministic portion as a linear function of the unknown parameters of the independent or predictor variables and the other is the random portion. When data are collected, the sample general linear model in matrix form is applicable to the data. However, the matrix equation seems not to be useful for catching any idea about the relationship between the matrix of predictors and error vector only with the assumptions about the error vector; i.e., $E(\epsilon) = 0$ and $var(\epsilon) = \sigma^2 I$. This requires the error vector be the line segment from the origin, which is the point $\mathbf{0}$, to the point $\epsilon$. The general linear sample model in matrix form is $$ \boldsymbol{y} = \boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{\epsilon} $$ where $y$ denotes the $n \times 1$ vector of observations, $X$ denotes the $n \times p$ matrix of known values, $\beta$ denotes the $p \times 1$ vector of unknown parameters and $\epsilon$ denotes the $n \times 1$ vector of unobserved random errors. The detailed discussion on random errors can be seen in Searle[1]. The matrix equation (1) shows that the vector of observations is composed of mean vector and error vector. When $E(y)$ is a 0 or a mean vector acting like an origin, the error vector satisfies the conditions of mean 0 and $var(\epsilon) = \sigma^2 I$. However, if $E(y) = X\beta$ then we might be interested in how to minimize the deviation vector, $y = X\beta$. The least square method can be used as one of available methods for minimizing the error vector to estimate the parameter vector $\beta$. However, the decomposition of an random error vector can be a little bit more comfortable and effective than the minimization of error sum of squares by the method of least squares in perspective that the structure of an random error vector is primarily considered. This idea is applied to breaking up an error vector in model (1). Since $X\beta$ is in a specific vector subspace generated by the column space of $X$, and $\epsilon = y - X\beta$ is nonzero, $\epsilon$ is not in the column space of $X$. The related topics about these can be seen in Graybill[5], Johnson[6] and so forth. Thus, when $X\beta$ is used as a base for getting a error vector of $y$ that is required to have one with the shortest distance from the origin among all the error vectors, we need to break up the error vector into a few component error vectors. This is the main idea of this paper. First, we discuss how to decompose the error vector. Secondly, we study the structure of error vector adjusted for the mean vector. Thirdly, we find the structure of $var(y)$ that is related to the error structure. Finally, we discuss a method that is useful to calculate the sum of squares associated with the error components. ## II. DECOMPOSITION OF RANDOM ERROR VECTOR In the matrix equation (1) let the mean vector $X\beta$ be an nonzero vector. Then the equation can be changed into $$ y = X\beta + \epsilon\tag{2}\\= X\beta + \epsilon_m + \epsilon_r $$ where $\epsilon_{m}$ and $\epsilon_{r}$ are denoting two component vectors of the error vector. Since $E(y)$ is $X\beta$, $\epsilon$ can be broken down into two types of error vector; one type of error vector is the one that is in column space of $X$, and the other has the characteristic orthogonal to an every vector in a vector space generated by the columns of $X$, which is an orthogonal complement of the column space of $X$. In matrix equation (2), $y - X\beta$ defines the error vector based on the mean vector $X\beta$ assumed to have mean 0 and $\sigma^2 I$ in n dimensional space. Since the error vector is defined as a vector of deviations from the mean vector of $E(y)$, we can decompose it into two component vectors depending on sources where the error vector is coming up; i.e., $\epsilon_{m}$ or $\epsilon_{r}$. Let's rewrite the matrix equation (2) in terms of error vector from the mean vector. Then, $$ \begin{array}{l} y - X \beta = \epsilon_ {m} + \epsilon_ {r} \tag {3} \\= X X ^ {-} \epsilon + (I - X X ^ {-}) \epsilon \\\end{array} $$ where $\epsilon_{m} = XX^{-}\epsilon$, $\epsilon_{r} = (I - XX^{-})\epsilon$, and $X^{-} = (X'X)^{-1}X'$ denotes the Moore-Penrose generalized inverse where $(X'X)^{-1}$ exists because $X$ is a full column rank matrix. From the equation (3), we can know that there are two types of error vector when a vector space is decomposed into two orthogonal vector subspaces based on the mean vector, $X\beta$, of $y$. The above equation shows that $E(y - X\beta) = 0$ and $var(y - X\beta) = \sigma^2 I$. Here, the error vector $y - X\beta$ is assumed as a linear combination of two different types of error vector each of which coming from two orthogonal vector subspaces. This is completely different concept of an error vector from the one we have thought traditionally. A lot of stuffs can be developed by the newly idea of viewing the error vector as the sum of a mean component error vector and a residual component error vector. Here, two specific terms are used to differentiate the types of error component: a mean component error for $\epsilon_{m}$ and a residual component error for $\epsilon_{r}$. ## III. STRUCTURE OF RANDOM ERROR VECTOR Since the structure of a random error vector in matrix model (1) is changed depending on the structure of the mean vector, we are going to take a look at it with a bit simpler general linear models. Consider a situation where measurements are measured as deviations from the fixed size for products from a routine process. Let $y$ be a random variable taking an observation on a randomly selected product from a population of products. Data collected from a sample of size $n$ from the population can be arrayed in matrix form. The $i$ th observation of $y$ is expressed as $y_{i} = 0 + \epsilon_{i}$ for $i = 1,2,\dots,n$, where $\epsilon_{i}$ 's are assumed to be independent with $E(\epsilon_i) = 0$ and $var(\epsilon_i) = \sigma^2$. Applying the sample general linear model (1) to the data, the model turns out to be $$ y = 0 + \epsilon $$ where $\pmb{y}$ is in $R_{n}$ denoting a Euclidean n-space, and $E(\pmb{y}) = \mathbf{0}$. Let $V_{0}$ be the vector space consisting only of $\mathbf{0}$ and let $V_{1}$ be the orthogonal complement of $V_{0}$. Since $\pmb{y}$ is the sum of two vectors such that one in $V_{0}$ and the other in $V_{1}$, $\mathbf{0}'\pmb{\epsilon} = 0$ which shows the relationship between mean vector and error vector of $\pmb{y}$; i.e., an orthogonal property. To express the equation (4) as the second expression in (2), $\pmb{\epsilon}$ can be divided into two terms, $\pmb{\epsilon}_{0}$ and $\pmb{\epsilon}_{1}$ where $\pmb{\epsilon}_{0}$ denotes the error vector generated by $\mathbf{0}$ in a basis of $V_{0}$, and $\pmb{\epsilon}_{1}$ denotes the error vector generated by a basis set of $V_{1}$. Now, the matrix equation can be expressed as $$ \boldsymbol {y} = \boldsymbol {0} + \epsilon_ {0} + \epsilon_ {1} \tag {5} $$ where $\epsilon_0$ is in $V_{0}$ of dimension 0, while $\epsilon_{1}$ is in $V_{1}$ of dimension n. Adding the information about the dimension of error component vectors, the equation (5) can be transformed into $$ \boldsymbol{y} - \boldsymbol{0} = \boldsymbol{O} \epsilon + (\boldsymbol{I} - \boldsymbol{O}) \epsilon $$ where $O$ represents $n \times n$ zero matrix and $I$ is $n \times n$ identity matrix. The equation shows that $\epsilon$ from the origin of rank 0 can be decomposed into two orthogonal vectors one of which being in the orthogonal complement of a vector space. In other words, this means that $\epsilon$ is actually composed of a linear combination of two component vectors; i.e. $\epsilon_0$ and $\epsilon_1$. It seems such valuable to grasp the structure of a random error vector for finding the covariance structure of an observation vector. Now, we can study the error structure further with a little bit general but still simpler model having just one quantitative variable as an predictor. Consider the simple linear model with only one nonrandom independent variable in addition to an intercept term; i.e., $y_i = \beta_0 + \beta_1 X_{1i} + \epsilon_i$, for $i = 1,2, \dots, n$. We rewrite this in vector and matrix form as $$ \begin{array}{l} \boldsymbol{y} = \mathbf{j} \beta_0 + X_1 \beta_1 + \epsilon \tag{7} \\= X \beta + \epsilon \\\end{array} $$ where $X = (j, X_1)$ is an $n \times 2$ coefficient matrix of $\beta$, $\beta = (\beta_0, \beta_1)'$ is an $n \times 2$ parameter vector, $j$ is an $n \times 1$ vector of ones, and $X_1$ is an $n \times 1$ vector of quantitative values. $\epsilon$ is an error vector assumed to have $E(\epsilon) = 0$ and $var(\epsilon) = \sigma^2 I$. The equation (7) is different from the one in (6) in that $E(y) \neq 0$. This is not a surprising thing in a general linear model other than that the mean vector $X\beta$ belongs to the column space of $X$, $V_m$ of dimension 2 which is thought to be a vector subspace in a Euclidean n-space, $R_n$. Since the mean vector of $y$, $X\beta$, is in $V_m$, $\epsilon$ should be divided into two components: one in $V_m$ and the other in $V_m^\perp$ denoting an orthogonal complement of $V_m$ in $R_n$; $V_m \oplus V_m^\perp = R_n$. The set of two vectors in the matrix of $X$ can be regarded as a basis set of $V_m$, which implies that $X\beta$ is in $V_m$. Hence, the error vector can be divided into two component vectors such that one component in $V_m$ and the other in $V_m^\perp$; that is, $\epsilon = \epsilon_m + \epsilon_r$. When we add this kind of information to the equation, the model will be $$ y - X\beta = \epsilon_m + (I - \epsilon_m)\epsilon $$ where $y - X\beta \in R_{n}$, $\epsilon \in V_{m}$ and $(I - \epsilon_{m})\epsilon$ in $V_{m}^{\perp}$. Both the set of the columns of a matrix $X$ of rank 2 equivalent to a basis for $V_{m}$ and the set of the columns of a matrix $XX^{-}$ can generate the same space, $V_{m}$. Hence, the equation (8) can be changed into $$ y - X \beta - X X ^ {-} \epsilon = (I - X X ^ {-}) \epsilon \tag {9} $$ where $XX^{-} \epsilon$ replaces $\epsilon_{m}$ and denotes the projection of $\epsilon_{m}$ onto a vector space, $V_{m}$, generated by two vectors $j$ and $x$. The matrix equation (9) turns out to be $$ \begin{array}{l} \left(\boldsymbol {I} - \boldsymbol {X} \boldsymbol {X} ^ {-}\right) \boldsymbol {\epsilon} = \boldsymbol {y} - \left(\boldsymbol {X} \boldsymbol {\beta} + \boldsymbol {X} \boldsymbol {X} ^ {-} \boldsymbol {\epsilon}\right) \tag {10} \\= \left(\boldsymbol {I} - \boldsymbol {X} \boldsymbol {X} ^ {-}\right) \boldsymbol {y} \\\end{array} $$ where $X\beta + XX^{-}\epsilon = XX^{-}y$. ## IV. COVARIANCE STRUCTURE OF VECTOR OF OBSERVATIONS From the decomposition of a random error vector of a general linear sample model in matrix form, we can identify that the matrix model (1) can be transformed into $$ \begin{array}{l} y = X \beta + \epsilon \tag {11} \\= X X ^ {-} y + (I - X X ^ {-}) y \\\end{array} $$ where $y$ is composed of two orthogonal vectors: i.e., $(XX^{-}y)'(I - XX^{-})y = 0$. The model equation (11) implies that all types of a general linear model can be represented by a sum of two orthogonal vectors where one vector belongs to an vector subspace and the other is in the orthogonal complement of the vector space generated by the coefficient matrix of $\beta$: i.e., $XX^{-}y \in V_{m}$, and $(I - XX^{-})y \in V_{m}^{\perp}$. Here, the primary concern is actually in structural aspects of an assumed linear model while the least square method focuses only on getting the best approximate solution from a system of inconsistent equations such that $X\beta - y = \epsilon$ by the method of minimizing the error sum of squares. Hence, they are different approaches developed from different view of points. Now, consider the calculation of $var(y)$. The covariance matrix of $y$ is $$ \begin{array}{l} \operatorname{var} (\boldsymbol{y}) = \operatorname{var} \left(\boldsymbol{X} \boldsymbol{X} ^ {-} \boldsymbol{y} + \left(\boldsymbol{I} - \boldsymbol{X} \boldsymbol{X} ^ {-}\right) \boldsymbol{y}\right) \tag{12} \\= \sigma^ {2} \boldsymbol{X} \boldsymbol{X} ^ {-} + \sigma^ {2} (\boldsymbol{I} - \boldsymbol{X} \boldsymbol{X} ^ {-}) \\\mathbf{\sigma} = \sigma^ {2} \mathbf{I} \\\end{array} $$ From the above equation (12), $var(y)$ can be obtained by identifying the linear transformations of $y$; i.e., the covariance matrix of $y$ can be partitioned as the sum of component covariance matrices, which can be done by ascertaining transformation matrices for component vectors of $y$. There are some referable literature related to covariance matrix such as Milliken and Johnson[7], Hill[8], and Searle[9]. Hence, it is essential to figure out the coefficient matrices of component error vectors to find the projections of $y$ onto the vector subspaces generated by the orthogonal coefficient matrices. Discussions on coefficient matrices are seen in Choi[10-12], where they are related to get nonnegative variance estimates. ## V. PROJECTION METHOD As a result of the decomposition of $e$, we see $y$ can be represented by the sum of two orthogonal component vectors such as (11) where one is in a vector space covering the $E(y)$ and the other is in the orthogonal complement of it. This means that $XX^{-}y$ actually defines a projection of $y$ onto the vector space spanned by the $XX^{-}$ where $X$ is coefficient matrix of $\beta$ and given as $X = (j, X_1)$. For the estimation of parameter vector $\beta$ we can use the mean part of the model in matrix form of (11). From the concept of a projection in a vector space the projection of $y$ onto a column space of $X$ is as follows: $$ X \boldsymbol {\beta} = X X ^ {-} y \tag {13} $$ where $E(y) = X\beta$. When $XX^{-}y$ is viewed as the orthogonal projection of $y$ onto a column space of $X$ we can take $\hat{\beta}_p = X^{-}y$ as the value of $\beta$ where $\hat{\beta}_p$ is an notation for differentiating from $\hat{\beta}$ obtained from the normal equations. When the expression in (13) is viewed as the system of equations, the best approximate solution to the system can be obtained as $\hat{\beta} = X^{-}y$ because the system of equations is inconsistent and $X$ is $n\times 2$ matrix of rank 2. Although solutions of $\beta$ can be obtained in different approaches, the results are actually same. In a similar way that $XX^{-}y$ can be used for the estimation of $\beta$, a quadratic form in $y$ can also be used for the estimation of $\sigma^2$. Here, the required quadratic form is given as $$ Q _ {r} = y ^ {\prime} \left(I - X X ^ {-}\right) y \tag {14} $$ where $(I - XX^{-})$ is a symmetric and idempotent matrix of rank $n - 2$. Since $(I - XX^{-})y$ is regarded as a linear transformation of $y$, it has all the information about the residual random error component, $\epsilon_r$. Hence, the quadratic form $Q_{r}$ in $y$ can be used to estimate the variance $\sigma^2$. Taking the expectation of $Q_{r}$ is given as $$ \begin{array}{l} E \left(\boldsymbol {Q} _ {r}\right) = E \left(\boldsymbol {y} ^ {\prime} \left(\boldsymbol {I} - \boldsymbol {X} \boldsymbol {X} ^ {-}\right) \boldsymbol {y}\right) \tag {15} \\= \sigma^ {2} t r (I - X X ^ {-}) + (X \beta) ^ {\prime} (I - X X ^ {-}) X \beta \\= \sigma^ {2} (n - 2) \\\end{array} $$ where $tr(\cdot)$ means trace of a square matrix denoted by $(\cdot)$, which is defined to be the sum of the diagonal elements of the square matrix. Some theorems and properties of trace can be seen in Graybill[2]. As an estimate of $\sigma^2$ from the equation (15), $\hat{\sigma_p^2}$ can be taken as $Q_r / (n - 2)$ which can also be obtained by the least square method when there is no normality assumption for $e$. Even though those two procedures have the same result, it should be noticed they are basically approaching from different view of point; that is, one is from the decomposition of an error vector, and the other is from the minimization of error sum of squares. ## VI. EXAMPLE As for an example of a simple linear model, we consider following data from Krumbein and Graybill[13]. The data are assumed to satisfy the model $y_{i} = \beta_{0} + \beta X_{1i} + \epsilon_{i}$, for $i = 1,2,\dots,10$, where $\epsilon_{i}$ are independent and identically distributed $N(0,\sigma^2)$. Table 1: Krumbein and Graybill's Data[13]. <table><tr><td>xi 550</td><td>200</td><td>280</td><td>340</td><td>410</td><td>475</td><td>160</td><td>380</td><td>510</td><td>510</td></tr><tr><td>yi 200</td><td>50</td><td>60</td><td>140</td><td>130</td><td>180</td><td>20</td><td>120</td><td>190</td><td>160</td></tr></table> For the estimation of two unknown parameters, $\beta_0$ and $\beta$, we can get $\hat{\beta}_p$ by multiplying $(X'X)^{-1}X'$ on both sides of the equation (13), which is given as: $$ (X'X)^{-1}X'X\beta = (X'X)^{-1}X'XX^{-}y $$ $$ \begin{array}{l} \hat {\boldsymbol {\beta}} _ {p} = \left(\boldsymbol {X} ^ {\prime} \boldsymbol {X}\right) ^ {- 1} \boldsymbol {X} ^ {\prime} \boldsymbol {y} \\= \left( \begin{array}{c c} 0. 9 8 2 0 6 0 8 7 8 & - 2. 3 1 2 0 8 6 e - 0 3 \\- 0. 0 0 2 3 1 2 0 8 6 & 6. 0 6 0 5 1 4 e - 0 6 \end{array} \right) \quad \left( \begin{array}{c} 1 2 5 0 \\5 5 0 5 0 0 \end{array} \right) = \left( \begin{array}{c} - 4 5. 2 2 7 3 4 5 0 \\0. 4 4 6 2 0 5 4 \end{array} \right) \\\end{array} $$ Denoting $\hat{\beta_{0p}}$, and $\hat{\beta}_{p}$ as estimates of $\beta_0$ and $\beta$ respectively, $\hat{\beta_{0p}} = -45.2273450$ and $\hat{\beta}_{p} = 0.4462054$. Least squares estimates are given as $\hat{\beta}_0 = -45.227$ and $\hat{\beta} = 0.446$. For the estimation of $\sigma^2$, we can get an estimate as: $$ \hat{\sigma}_p^2 = \mathbf{y}'(\mathbf{I} - \mathbf{X X}^{-})\mathbf{y}/8 = 2398.13/8 = 299.7663 $$ where $(I - XX^{-})$ is the $10\times 10$ matrix of rank 8. As an estimate of $\sigma^2$ denoted by $\hat{\sigma}_p^2$ is given as 299.7663 which is the same as the least squares estimate $\hat{\sigma}^2 = 299.766$; hence, $\hat{y} = -45.2273450 + 0.446205X_1$. ## VII. DISCUSSION The primary concern of the study is on the decomposition of an error vector in matrix form of a general linear model. When $\epsilon$ is $n \times 1$ vector, the usual assumptions for error vector are sometimes given as $E(\epsilon) = 0$ and $Var(\epsilon) = \sigma^2 I$. Under these assumptions, an idea for breaking up the error vector lies on the thought of which the mean vector is related to the error vector because the error vector is defined to be the deviation vector from the mean of the model. When the error vector is decomposed into two orthogonal components, it is shown that a projection can be defined from the decomposition of the error vector. Hence, a partition of the vector of observations can be seen as the sum of vectors which are orthogonal projections each other. The covariance matrix of $y$ is partitioned into two covariance matrices; that is, one for $(XX^{-})y$, and the other for $(I - XX^{-})y$. This implies that the covariance matrix of a vector of observations can always be partitioned into component matrices each of which corresponding to an orthogonal projection of $y$ respectively. From the decomposition of an error vector of a general linear model, we derived two types of estimators; one is linear transformation of $y$ for $X\beta$ to estimate $\beta$ and the other is quadratic form in $y$ for $\sigma^2$. Partitioning of the covariance matrix can be useful to ascertain the covariance matrix of each component projection. It is worth to note that decomposition of an error vector is actually defines a projection of $y$ onto a column space of $X$ and which is quite different approach from the least square method in a point of view for an error vector. ## VIII. CONCLUSIONS Although the least squares method is very useful and accepted as one of well-known methods for estimating the unknown parameters included in a linear model, it seems not to be right for finding out whether there is any orthogonal property exists among errors. Since the least squares method concentrates only on minimizing the sum of squares of deviations of the observations from the expected values, it is not an appropriate method as a tool for getting the information on an orthogonal property between the groups of errors. The orthogonal property is extremely important in statistics especially in the analysis of variance for getting nonnegative estimates for variance components. There are lots of interesting papers[14-20] related to the negative estimates of variance components seemed to be caused by overlooking the orthogonality. So, it is emphasized that the orthogonal property can be found by the decomposition of the random error vector. Hence, the procedure discussed on this paper is distinct from any other methods for estimating the unknown parameters in a general linear model. Author Contributions: Not applicable. Funding: This research received no external funding. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Table 21.1, pp.231, Krumbein and Graybill[13]. The data analyzed in this study are openly available in reference number [13]. Acknowledgments: Not applicable. Conflicts of Interest: The authors declare no conflict of interest.

Generating HTML Viewer...