Kernel-PCA data integration with enhanced interpretability

Background Nowadays, combining the different sources of information to improve the biological knowledge available is a challenge in bioinformatics. One of the most powerful methods for integrating heterogeneous data types are kernel-based methods. Kernel-based data integration approaches consist of two basic steps: firstly the right kernel is chosen for each data set; secondly the kernels from the different data sources are combined to give a complete representation of the available data for a given statistical task. Results We analyze the integration of data from several sources of information using kernel PCA, from the point of view of reducing dimensionality. Moreover, we improve the interpretability of kernel PCA by adding to the plot the representation of the input variables that belong to any dataset. In particular, for each input variable or linear combination of input variables, we can represent the direction of maximum growth locally, which allows us to identify those samples with higher/lower values of the variables analyzed. Conclusions The integration of different datasets and the simultaneous representation of samples and variables together give us a better understanding of biological knowledge.


Background
With the recent rapid advancements in high-throughput technologies, such as next generation sequencing, array comparative hybridization and mass spectrometry, databases are increasing in both the amount and the complexity of the data they contain. One of the main goals of mining this type of data is to visualize the relationships between biological variables that are involved [1]. For instance, visualizing gene expression guides the process of finding genes with similar expression patterns. However, due to the number of genes involved, it is more effective to display the data by means of a low-dimensional plot. Here we focus on the problem of reducing dimensionality and the interpretability of the resulting data representations.
Principal component analysis (PCA) has a very long history and is known to be a very powerful tool in the linear case. PCA is used as a visualization tool for the analysis of microarray data [2] and [3]. However, the sample space that many research problems deal with is considered nonlinear in nature; for example, the sample space of microarray data. One reason for this nonlinearity might be that the interactions of the genes are not completely understood. Many biological pathways are still not fully understood. So, it is quite naive to assume that genes are connected in a linear fashion. Following this line of thought, research into reducing the nonlinear dimensionality for microarray gene expression data has increased. Finding methods that can handle such data is of great importance if we are to glean as much information as possible from them.
Kernel representation offers an alternative to nonlinear functions by projecting the data into a high-dimensional feature space, which increases the computational power of linear learning machines [4] and [5]. Kernel methods enable us to construct different nonlinear versions of any algorithm which can be expressed solely in terms of dot products; this is known as the kernel trick. Kernel machines can be used to implement several learning algorithms but the interpretability of the resultant output representations may be cumbersome, because input variables are only handled implicitly [6].
Nowadays, combining multiple sources of data to improve the biological knowledge available is a challenging task in bioinformatics. Data analysis of different sources of information is not simply a matter of adding the analysis of each separate dataset; instead it consists of the simultaneous analysis of multiple variables in the different datasets [7].
Some of the most powerful methods for integrating heterogeneous data types are kernel-based methods [8] and [9]. We can describe kernel-based data integration approaches as using two basic steps. Firstly, the right kernel is chosen for each data set. Secondly, the kernels from the different data sources are combined to give a complete representation of the available data for a given statistical task. Basic mathematical operations such as multiplication, addition, and exponentiation preserve properties of kernel matrices and hence produce valid kernels. The simplest approach is to use positive linear combinations of the different kernels.
In this work, we analyze the integration of data from several sources of information using kernel PCA, from the point of view of reducing dimensionality and extending previous results [10]. Moreover, we improve kernel PCA interpretability by adding to the plot the representation of the input variables that belong to any dataset. In particular, for each input variable or linear combination of input variables, we can represent the direction of maximum growth locally, which allows us to identify those samples with higher/lower values of the variables analyzed. Therefore the integration of different datasets and the simultaneous representation of samples and variables together give us a better understanding of biological knowledge. This paper starts by briefly reviewing the notion of kernel PCA (Section 2). Section 3 contains our main results: a set of procedures to enhance the interpretability of kernel PCA when multiple datasets are analyzed simultaneously. We then present our results and apply them in parallel to analyze a nutrigenomic study in mouse [11].

Results and discussion
Kernel methods enable us to construct different nonlinear versions of any algorithm which can be expressed solely in terms of dot products, this is the case of kernel PCA. Kernel PCA can be used to reduce dimensionality, thereby improving on linear PCA, but the interpretability of the output representations may be cumbersome because the input variables are only handled implicitly.
In this section, we propose a set of procedures to improve the interpretability of kernel PCA. The procedures are related to the following aspects: • Representation of input variables.
• Data integration and representation of input variables.
• Representation of linear combinations of input variables.
• Revealing the interpretability of input variables.
To illustrate these procedures we use an example from metabolomics and genomics. The datasets come from a nutrigenomic study in mouse [11]. Forty mice were studied and two sets of variables were acquired: expressions of 120 genes measured in liver cells; and concentrations (in percentages) of 21 hepatic fatty acids (FAs) measured by gas chromatography. Biological units (mice) are cross-classified according to two factors: genotype, which can be wild-type (WT) or PPARα-deficient mice (PPAR); and diet, with 5 classes of diet in accordance with the FA composition.
The oils used for the experimental diet preparation were: corn and rapeseed oils (50:50), as the reference diet (ref); hydrogenated coconut oil, as a saturated FA diet (coc); sunflower oil, as an ω6 FA-rich diet (sun); linseed oil, as an ω3 FA-rich diet (lin); and corn, rapeseed and fish oils (42.5:42.5:15), as the fish diet. In the study, it cannot be assumed that variations in one set of variables cause variations in the other; we do not know a priori if changes in gene expression imply changes in FA concentrations or vice versa. Indeed, the nuclear receptor PPARα, which acts as a ligand-induced transcriptional regulator, is known to be activated by various FAs and to regulate the expression of several genes involved in FA metabolism. It should be noted that the main observations discussed in [11], which were extracted separately from the two datasets by both classical multidimensional tools (hierarchical clustering and PCA) and standard test procedures, are also highlighted by kernel PCA graphical representations.

Representation of input variables
In order to achieve interpretability we add supplementary information into kernel PCA representations. We have developed a procedure to represent any given input variable on the subspace spanned by the eigenvectors ofC (see Methods).
We can consider that our observations are realizations of the random vector X = (X 1 , ..., X n ). Then, to represent the prominence of the input variable X k in kernel PCA, we take a set of points of the form: y = a + se k ∈ ℝ n , where e k = (0, ..., 1, ..., 0) ∈ ℝ n , s ∈ ℝ, and the k-th component is equal to 1 and the others are 0. Then, we can compute the projections of the image of these points,φ y , onto the subspace spanned by the eigenvectors ofC. Taking into account equation (8), the induced curve expressed in matrix form is given by the row vector: where Z s is in the form of (7). In addition, we can represent directions of maximum growth of s k (s) with respect the variable X k by projecting the tangent vector at s = 0. In matrix form, we have: With: and, using the chain rule: In particular, let us consider the Gaussian radial basis function kernel: k(x, z) = exp(-c ||x -z|| 2 ), with c > 0 a free parameter. Using the notation above, we have: For the set of points of the form y = a + se k ∈ ℝ n : In addition, if a = x b (a training point) then: To illustrate our procedure we introduce a toy example. We have generated a dataset which has 18 points in 6-dimensional space. Coordinates of the points are selected in order to distinguish 3 groups clearly separated. The group 1 has 6 points such that the sum of X 1 and X 2 coordinates is equal to 15 for each point. Moreover, in this group, there are 3 points such that the sum of X 3 , X 4 and X 5 is 0, and is equal to 6 for each the another 3 points. The group 2 has 6 points such that the sum of X 3 , X 4 and X 5 coordinates is equal to 0 for each point. Besides, in this group, there are 3 points such that the sum of X 1 and X 2 is 0, and is equal to -4 for each the another 3 points. Finally, the group 3 has 6 points such that the sum of X 1 and X 2 coordinates is equal to 0 for each point. Moreover, in this group, there are 3 points such that the sum of X 3 , X 4 and X 5 is 15, and is equal to 24 for each the another 3 points. All coordinates were perturbed with weak gaussian noise in order to introduce a small amount of variability inside each group. At each group the variable X 6 is assigned randomly according to a Gaussian of mean zero and standard deviation 0.5. The configuration of the points is such that we expect that in reduction of dimension only the first dimensions are necessary to reveal the arrangement of the three groups. It can be seen in Figure 1 where the two leading components of kernel PCA are represented. We can see the group 1 (represented by triangles up and circles) on the negative part of the first principal axe, group 2 (represented by plus signs and by cross) in the central part and the group 3 (represented by diamonds and triangles down) on the positive part. Figure 1 shows samples and the variables from X 1 to X 5 at each sample. Variables are represented by vectors that indicate the direction of maximum growth in each variable. In fact, we can see that the vectors point to those groups characterized by higher values in each variable. For instance, the variables X 1 and X 2 point to the group 1, and the variables X 3 , X 4 , and X 5 point to the group 3. Figure 2 shows the variable X 6 at each sample, we can observe that this variable is poorly represented and has no preferred direction towards any group.
A natural extension of the above procedure is the representation of linear combinations of input variables. Details can be found in section 3.2. With the aim to show this property we displayed in Figure 3 the samples and the linear combinations X 1 + X 2 and X 3 + X 4 + X 5 at each sample. Linear combinations are represented by vectors that point to the direction of maximum growth in each of the linear combinations. We can observe that at each sample vectors point to those groups with higher values in each of linear combinations. For example, vectors representing X 1 + X 2 point to group 1, and vectors representing X 3 + X 4 + X 5 point to group 3.

Analyzing the nutrigenomic dataset
We illustrate the representation of variables by analyzing the dataset in [11]. We apply kernel PCA and representation of variables to the genomic data and FA data.
Firstly, we compute kernel PCA by analyzing only gene expression level data. Figure 4 shows the two leading axes of kernel PCA. We can observe that the genotypes are clearly separated (WT samples are represented in black and PPAR samples in red). Diet representation is: ref diet is represented by the letter x; coc diet by circles; sun diet by diamonds; lin diet by plus signs; and fish diet by triangles). Figure 4 shows the AOX (blue vector) and CAR1 (green vector) genes. Vectors indicate the direction of maximum growth of the gene expression at each sample point. Thus, we can observe that AOX increases towards WT and CAR1 towards PPAR. These results are in agreement with those found in [11] and [12]. Figure 5 and Figure 6 show the profiles of the medians of the expression of AOX and CAR1 grouped by genotype. We can observe that these profiles agree with the kernel PCA representation.
Secondly, to compare results, we compute kernel PCA analyzing only FA levels. In Figure 7 we can observe that the sample points are separated by genotype, but we can also observe that the samples with coc diet (a diet with hydrogenated coconut oil as a saturated FA diet) form a cluster. Figure 7 shows C20.2ω.6 (green vector) and C16.0 (blue vector) FAs. It reveals higher levels of C20.2ω.6 towards PPARα-deficient clustered samples (red) and that levels of C16.0 are higher towards the WT cluster of samples (black).
These results are also in agreement with those found in [11] and [12]. Figure 8 and Figure 9 show the profiles of the medians of the concentrations of C16.0 and C20.2ω FAs, grouped by genotype. We can observe that these profiles agree with the kernel PCA representation.

Data integration and representation of input variables
The kernel formalism allows us to combine heterogeneous datasets for data fusion. Basic algebraic operations such as addition, multiplication and exponentiation preserve the key properties of symmetry and positive semidefiniteness, and thus allow a simple but powerful algebra of kernels. If k 1 and k 2 are kernels defined respectively on X 1 × X 1 and X 2 × X 2 , then their direct sum: is a kernel on (X 1 × X 2 ) × (X 1 × X 2 ) . Here, x 1 , x 1 ∈ X 1 and x 2 , x 2 ∈ X 2 .
This construction can be useful if the different parts of the input have different meanings and should therefore be dealt with differently. In that case, we can split the inputs into two parts, X 1 and X 2 , and use two different kernels for these parts. This is the case when we are integrating two separate datasets. In consequence, our procedure can easily be extended to data fusion. Firstly, we reduce the dimension of the entire data (x 1i , x 2i ), i = 1, ..., m, by applying kernel PCA with the kernel K given by k 1 ⊕ k 2 . Secondly, to find the coordinates of a test point: we proceed by analogy with (8), so that (7) becomes: Z = K y 1 , x 1i , y 2 , x 2i m × 1 = k 1 y 1 , x 1i + k 2 y 2 , x 2i m × 1 .
When we integrate two datasets, we can represent any given input variable that belongs to one of the datasets. Let us suppose that we wish to represent the variable X l k that belongs to the dataset l = 1, 2. Then (2) becomes:    Then, formula (1) allows us to display variables that belong to any of the datasets over the kernel PCA representation of samples, simultaneously.

Analyzing the nutrigenomic dataset
Continuing with the same nutrigenomic study, we compute kernel PCA by analyzing both datasets simultaneously; that is, gene expressions and FA concentrations. We observe that the genotypes are clearly separated (WT is represented in black and PPAR in red) and also mice with the coc diet form a cluster of both genotypes; see Figure 10. Also, Figure 10 shows AOX (black vector) and CAR1 (green vector) genes, and C20.2ω.6 (blue vector) and C16.0 (red vector) FAs. It reveals higher expression of CAR1 and higher concentrations of C20.2ω.6 towards the PPAR cluster. In contrast, AOX gene expression and concentrations of C16.0 are higher towards the WT cluster. These results are in agreement with those found in the individual kernel PCAs above.

Representation of linear combinations of input variables
A natural extension of the above procedure is the representation of linear combinations of input variables. This may be useful for representing gene modules or gene networks. Let us suppose that we wish to represent the linear combination: X k 1 + X k 2 + · · · + X k l , where k 1 , k 2 ,...,k l ∈{1, 2, ..., n}, with ki ≠ kj , i,j = 1, ..., l. Then, when K is the Gaussian radial basis function kernel, (2) becomes: Then, formula (1) allows us to represent any linear combination of input variables.

Analyzing the nutrigenomic dataset
To illustrate this procedure we have analyzed the genes GSTpi2, CYP3A11 and CYP2c29. These genes are involved in the functioning of detoxification [12]. We perform kernel PCA analyzing both dataset simultaneous and represent the sum of the expressions of the genes GSTpi2, CYP3A11 and CYP2c29. Figure 11 shows sample points and the vector corresponding to the sum of the three gene expressions is attached to each point. The vector indicates the direction of maximum growth of the sum of the expressions. We observe that the sum of the expressions increases towards the fish diet. This is in agreement with the findings in [12].

Revealing the interpretability of input variables
Our procedure for representing input variables on the two-dimensional subspace expanded by the two main eigenvectors ofC, displays the variables as vectors whose direction is the direction of maximum growth of the variable at a given point; in particular, at the sample points.
So, if we set a direction in this plane, given by a vector w, we can search for input variables whose representation on the kernel PCA plane are correlated with this direction. Let us suppose that we observe clusters of samples in the kernel PCA representation; then an interesting direction can be given by the vector defined by any two cluster centroids.
Once we have selected a vector w, we denote w i as the parallel vector of w attached to the image given by kernel PCA of the sample point x i , i = 1, ..., m. For any variable X k , we now compute its vector representation in kernel PCA using formula (1); we denote this vector as dσ k ds s=0 . Therefore, for each sample point,  x i , i = 1, ..., m, we have two vectors, one corresponding to the direction w i , and other corresponding to the X k representation, dσ k ds s=0 x i . After this, to measure the strength of the correlation between X k and w, we average the cosine of the angles between each pair of vectors, that is: Finally, we order all the variables according to R k and we can select those with higher values and also those with lower values. Thus, in this way, for each sample cluster, we can find the correlated variables with higher and lower values. Knowledge of such variables can improve the biological interpretability of the results.
A natural extension of this procedure is to take as w the vector corresponding to one of the input variables. Then, if we know that a certain input variable is useful for interpreting the kernel PCA representation, we can search for other input variables whose representation on the kernel PCA plane are correlated with this feature. If we are integrating multiple datasets, we can search for correlated variables in each dataset.

Analyzing the nutrigenomic dataset
To illustrate this procedure. We have selected a preferred direction in the kernel PCA plane. Figure 12 shows this direction (green vector). This direction represents variables that are less expressed in samples with the coc diet than in those with other diets. Tables 1 and 2 summarize the genes and FAs that are most correlated with the selected direction.
In Table 1, we can observe that FAs with negative correlation, such as C16.1ω.7, C20.3ω.9 and C18.1ω.7, represent FAs with higher concentrations in samples with the coc diet. In contrast, FAs that are positively correlated, such as C22.4ω.6, C18.2ω.6, C18.3ω.3 and C22.5ω.6, represent FAs with higher concentrations in samples with other types of diet. Furthermore, in Table 2, we can observe that genes with negative correlation at the top of the table, such as S14, ACC2 and LPL, are more highly expressed in samples with the coc diet, whereas genes at the bottom of the table, that are positively correlated, are less expressed in the coc diet samples. These results are in agreement with those found in [12].

Conclusions
With the rapidly increasing amount of genomic, proteomic, and other high-throughput data that is available, the importance of data integration has increased significantly recently. Biologists, medical scientists, and clinicians are also interested in integrating the highthroughput data that has recently become available with previously existing clinical, laboratory and biological information.
Kernel methods, in particular kernel PCA, constitute a powerfully methodology because they allow us to reduce dimensionality and integrate multiple datasets, simultaneously. Moreover, in this paper we have introduced a set of procedures to improve the interpretability of kernel PCA representations. The procedures are related to the following aspects: 1) representation of variables; 2) linear combination of representations of variables; 3)  data integration and representation of variables; and 4) revealing the interpretability of input variables. Our procedure is a kernel-based exploratory tool for data mining that enables us to extract nonlinear features while representing variables.

Methods
Given a sample space X, a real valued positive definite kernel k on X is a map k : . . , m, and kernel is zero is attained if all the coefficients α j are zero. A kernel can be interpreted as a similarity measure of the samples and allow us to identify each x ∈ Xwith a real function given by which is an element of a dot product vector space that will be called feature space [5]. It consists of all functions for any m ∈ N and x 1 , . . . , x m ∈ X , α 1 , . . . , α m ∈ R. It has the reproducing property Implying ϕ(x), ϕ(y) = k(·, x), k(·, y) = k(x, y). After completion we can turn our feature space into a Hilbert space ℋ k [5]. The space ℋ k is the reproducing kernel Hilbert space (RKHS) induced by the kernel function k.
Given any j and any set of observations x 1 , ..., x m , the Gram or kernel matrix of k with respect x 1 , ..., x m is the m × m matrix K with elements then, the points will be centered. LetK be denote the kernel matrix of centered points,K ij = φ (x i ),φ(x j ) , Because we do not have the centered data (3), we cannot computeK explicitly, however we can express it in terms of its noncentered counterpart K [5]. Using the vector 1 m = (1, ..., 1) T , we get the more compact expression  In ℋ k the covariance matrix takes the form We have to find eigenvaluesλ ≥ 0 and nonzero eigen-vectorsṼ ∈ H k \{0} satisfying CṼ =λṼ To find the solutions of (4) we solve the dual eigenvalue problem withα being the expansion coefficients of an eigenvector (in ℋ k ) in terms of the centered points (3) The solutionα k , k = 1, ..., r, are normalized by normalizing the corresponding vectorṼ k in ℋ k , which translates intoλ k α k ,α k = 1.
Consider a test point y. To find its coordinates we compute projections of centered j-images of y onto the eigenvectors of the covariance matrix of the centered points, Introducing the vector Z = K y, x i m×1 . Then, whereṼ is a m × r matrix whose columns are the eigenvectorsṼ 1 , . . . ,Ṽ r .