Network-based scoring system for genome-scale metabolic reconstructions
© Serrano and Sagués; licensee BioMed Central Ltd. 2011
Received: 3 January 2011
Accepted: 19 May 2011
Published: 19 May 2011
Network reconstructions at the cell level are a major development in Systems Biology. However, we are far from fully exploiting its potentialities. Often, the incremental complexity of the pursued systems overrides experimental capabilities, or increasingly sophisticated protocols are underutilized to merely refine confidence levels of already established interactions. For metabolic networks, the currently employed confidence scoring system rates reactions discretely according to nested categories of experimental evidence or model-based likelihood.
Here, we propose a complementary network-based scoring system that exploits the statistical regularities of a metabolic network as a bipartite graph. As an illustration, we apply it to the metabolism of Escherichia coli. The model is adjusted to the observations to derive connection probabilities between individual metabolite-reaction pairs and, after validation, to assess the reliability of each reaction in probabilistic terms. This network-based scoring system uncovers very specific reactions that could be functionally or evolutionary important, identifies prominent experimental targets, and enables further confirmation of modeling results.
We foresee a wide range of potential applications at different sub-cellular or supra-cellular levels of biological interactions given the natural bipartivity of many biological networks.
A crucial milestone to understand and control cellular behavior is the building up of reliable reconstructions of the interactions spanning different functional levels. Such reconstructions find a natural abstraction in the form of complex networks [1–3], where nodes represent cellular components such as genes, proteins, or metabolites, and edges identify the presence of biological interactions between them . These network representations enable to map the large-scale structure of cellular interactions [5, 6], to explore the basic principles of transcriptome and proteome organization , to identify missing genes encoding for specific metabolic functions , or to analyze emergent global phenomena in metabolism like robustness and regulation [8–11].
At present, the information for complex network representations of cellular systems comes primarily from web-based databases, oftentimes manually curated with information from multiple sources, like annotations from the literature or new experiments . It is common to take the reliability of these data for granted and to draw from them resolute inferences about the properties or the behavior of the investigated organisms. Although, in general, observations are getting more and more precise, uncertainties about components or interactions remain [13, 14]: experimental targets are many times biased towards the most rewarding in terms of expected impact, experimental evidence gathered with different methodologies is not always of the same quality, variability is unavoidably present in different organisms of the same species, and perfect environmental control is often difficult to achieve in experiments. In particular, high-throughput techniques produce massive data in comparison with more dedicated experiments at the price of repeatedly reported inaccuracy .
In this scenario, prediction of interactions in probabilistic terms is possible on the basis of the structure of the network alone and could serve to better characterize network-based descriptions of biological systems and as a guide for new experiments. In particular, the likelihood of biochemical reactions in cellular metabolisms can be assessed in terms of a scoring system that assigns a probability of occurrence to every reaction. This claims for a new interpretation, since canonically a chemical reaction either exists or does not. For network analysts, scoring a biochemical process in terms of its likelihood is a computational challenge. Starting from the assessment of individual links, as it has been done for instance for protein-protein interaction networks [16, 17], the problem is conditioned by the need of non-local models that faithfully capture the large-scale statistical regularities of the networks, and by the requirement of sufficient reliability of the input observations. In this respect, the experimental reconstruction of genome-scale biochemical networks [18–21] is a well-established procedure and available data are of sufficient quality and can be exploited to get accurate network-based predictions.
Here, we propose a computational network-based reaction-scoring system and, subsequently, apply it to the metabolism of E. coli. We introduce a link prediction method that exploits the complex hierarchical structure and the statistical regularities of metabolic networks [22, 23] and takes explicitly into account its bipartite nature [24, 25]. Our model is adjusted to the observations in order to derive connection probabilities [26–28] between metabolite-reaction pairs and, after validation, we integrate individual link information to assess the likelihood of each reaction, both in absolute terms and relative to a random null model. Our network-based scoring system ranks reactions with a continuously distributed index which breaks the degeneracy and nested nature of experimental confidence scores . However, the two schemes are differently targeted: confidence scores in the databases inform about the level of experimental evidence of a reaction, while the network-based score measures occurrence likelihood based on statistical inference. Our aim is, thus, to complement the existing experimental approach with new and essential network-based computationally inferred information.
Results and Discussion
Network-based reaction-scoring system
Beyond experimental evidence, it is possible to assess the likelihood of reactions in genome-scale metabolic reconstructions using theoretical models. Given an experimentally derived metabolic reconstruction, a score can be computed for each reaction on the basis of a suitable stochastic network-based model, as we propose in the next section. In the context of bipartite network representations [24, 25], the model exploits the statistical regularities that underlie the structure of a metabolic network in order to ascertaining how well individual links between metabolites and reactions fit the observed topological patterns. In this way, it is possible to predict probabilities of connection for all potential interactions, those already present in the reconstruction and those absent. These probabilities are further integrated for the specific combination of metabolites involved in any particular reaction.
A network-based reaction score can then be defined as the average number of occurrences n ν that the model associates to its particular combination of metabolites ν. Defined in this way, these scores break the degeneracy of reactions with identical experimental evidence.
The Tree Distance Bipartite model
Previous works have shown that the complex organization of metabolic networks displays characteristic features shared by other complex networks: short topological diameter , steady state cycles  or structural robustness , for instance. We implement a large-scale model that takes advantage of some of those organizing principles, in particular the heterogeneity in the number of connections per metabolite (degree) , to infer connection probabilities p mr between metabolites and reactions. Network-based models are usually defined in terms of connection rules between the nodes. These laws are usually stated independently of observed systems to produce simulated networks that summarize their topological structure. Notice that here, in contrast, we compute the set of connection probabilities that has the maximum likelihood to reproduce the observed patterns in empirical metabolic networks, so we are solving the inverse problem.
Our first step is to assume a metric space underlying the structure of the empirical metabolic network. To this end, we fit to it a hierarchical random graph , once represented as a bipartite network with M metabolites and R reactions (see Figure 1). More specifically, we adjust the empirical bipartite network to a dendrogram, or binary tree structure, where metabolites and reactions appear as leafs. This tree represents the underlying metric space, and each of the M + R - 1 internal nodes t in the dendrogram has an associated distance d t , so that each pair metabolite-reaction for which t is the lowest common ancestor is separated by a distance in the tree d mr = d t , independently from whether the link actually exists in the empirical network or not. We find these tree distances by fitting the tree to the empirical network data combining a maximum-likelihood approach with a Monte Carlo sampling method that explores the space of all possible dendrograms (see Methods). Our results are based on intensive numerical simulations that average a large number of samples in the stationary state when changes in the form of the dendrogram do not modify the likelihood function beyond fluctuations.
where k m is the degree of the metabolite and μ = 1/R ensures that network realizations with these connection probabilities have the same number of links as the observed network. From this equation, it is clear that the closer a pair is, the higher will be its connection probability. A Tree Distance Bipartite (TDB) score for every specific reaction can then be computed by applying Eq. (1) and Eq. (2).
TDB scores for E. coli metabolism. Validation and Implementation
As an application of the methodology, we analyzed the iAF1260 version of the K12 MG1655 strain of E. coli  provided in the BiGG database [33, 34]. From the empirical data, we built a bipartite network representation (see Methods and Additional File 1) with 1479 reactions and 976 metabolites (network provided in Additional File 2). As expected, the number of metabolites entering into a reaction, k r , follows a nearly homogeneous distribution with mean < k r > = 4.82 and mode 5. In contrast, the number k m of reactions in which a metabolite participates displays a heterogeneous degree distribution very close to a scale-free with and an average degree < k m > = 7.30 (Additional File 1, Figure S2). The most highly connected substrates have more than a hundred and up to 841 connections (h+, h2o, atp, pi, adp, ppi, nad, nadh).
The TDB probabilities p mr were validated using standard tools in medical decision making and signal-detection theory , and compared with corresponding results from alternative models: the Configuration Model for bipartite networks [24, 36, 37] (CMB), the Hierarchical Random Graph model  generalized to bipartite networks (HRBG), and a local approach based on the computation of common neighbors (CN)  (reactions) between pairs of metabolites (see Methods). We checked two different aspects: the discrimination power of the models, and the behavior of the predictions under noisy conditions. We first measured the receiver operating characteristic (ROC) curve of predicted probabilities.
The second test helps to evaluate robustness against noise. Links were removed from the empirical network in order to see whether our algorithm was able to distinguish those from non-existing links. After removing a subset of links uniformly at random in the original network, a new set of connection probabilities was calculated on the basis of the remaining part of the network. The new probabilities associated to the removed connections were compared one by one to that of non-existing links. This statistic ranges from 0.5 to 1 and indicates how much better our method performs as compared to a by chance baseline accuracy of 0.5. We calculated this accuracy statistic, that is shown in Figure 2B, for different fractions of removed links. For the TDB model, when 1% of the 7127 links in the network are removed the index takes a value of 0.87, meaning that 87% of the times removed links are ranked higher in probability than non-existing links.
In the BiGG database, every reaction (except for exchanges and transports through outer membrane) is annotated with a confidence score assessing its level of experimental evidence. These values are discrete and range from 4 at the top, when there is direct biochemical proof, to 0 at the bottom, if the reaction is included with no experimental evidence but only because it improves modeling results. In between, values of 3 correspond to genomic evidence, level 2 refers to sequence homology evidence, and 1 stands for physiological evidence. This confidence scoring system presents some shortcomings, one being the degeneracy implicit in the use of only five discrete values for lists of hundreds or thousands of reactions, another the fact that different categories are not disjoint but nested, meaning that one backs the other. An additional worth remarking feature concerns the mean degree of the metabolites participating the reactions in each scoring level. Quite unexpectedly, we found a strong bias as this measure monotonously increases when decreasing rank from 4 to 1. More specifically, the average degree of metabolites entering into reactions with high level of confidence score is, on average, smaller than the average degree of metabolites associated to reactions with low level of confidence score.
It is not completely surprising, as the just mentioned test seems to suggest, that absolute indices based on the architecture of the empirical network may go differently as compared to qualitative indicators assigned from updated experimental information. We, thus, aim at complementing the potentialities of both scoring systems by cross-comparing them. In this respect, first we look for reactions with strong experimental evidence (values 4 and 3) that at the same time score low in the TDB system. Those correspond to very specific reactions that could be functionally or evolutionary important. Examples are the five FMNH2-dependent monooxygenase reactions in the Inorganic Ion Transport pathway, the Pyridoxine 5'-phosphate synthase reaction in the Cofactor and Prosthetic Group Biosynthesis pathway, or the Taurine dioxygenase reaction in the Alternate Carbon metabolism. Conversely, a weak experimental evidence, scores 2 or 1 in the database, but a high value of the S TDB score, qualifies the reaction as a good target for further experimental verification in standard conditions. Many examples are found within the transport subsystems, like reactions of transport via ABC system (iron (II) and (III), phosphatidylglycerol, phosphatidate). If the S TDB score is low, the reaction could be difficult to be observed experimentally except for very specific environments. Finally, high S TDB scores for reactions that where required for modeling, score 0 in the database, denote consistency between our model and steady-state flux optimization solutions. It is worth remarking that these reactions involve more than one carrier metabolite and stand out as potential experimental targets in standard conditions. However, a variety of reactions manifest discrepancies between both model-based likelihoods. More specifically, we differentiate two situations. The first does not entail contradiction and refers to very specific reactions with low S TDB scores involving rare metabolites that, on the other hand, seem to be essential for the viability of the bacteria according to flux balance analysis. This points out to potential experimental targets in non-standard conditions. Reactions in the subsystem of Cofactor and Prosthetic Group Biosynthesis are, for instance, in this category. In contrast, the second situation involves reactions like those with the highest number of participating metabolites, which according to confidence scores in the database were included in the reconstruction on the basis of modeling reasons while our methodology predicts very low scores. For them, we believe that low S TDB scores point indeed to insufficient ne detail resolution in the database and we are suspicious that new experiments may show a split of the high degree reactions into a set of coupled lower degree ones.
Relative TDB scores
Along with absolute scores, we also analyzed relative scores defined on the basis of the Configuration Model for bipartite networks (CMB) [24, 36, 37] (see Methods). The latter assumes the actual degree distributions for metabolites and reactions and it is otherwise maximally random in the assembly of connections. The S CMB score of a reaction represents its probability of occurrence according to the configuration model and we use this value to compute the relative score as the ratio ∑ = S TDB /S CMB (provided in Additional File 2). Since differences between both scores are mainly related to the consideration of tree distances between metabolites and reactions in the TDB model, a relative score ∑ = S TDB /S CMB > 1 (< 1) points to the presence of tree distance correlations (anticorrelations) in the bipartite network, which are absent in the random case. In other words, a ratio higher (smaller) than one for a certain reaction indicates that its metabolites have a tendency to aggregate (avoid each other) as compared to the random case.
The ranking of relative scores is shown in the bottom graph of Figure 3, where several clusters can be differentiated. The first three clusters appear in slightly tilted plateaus with levels well separated by appreciable jumps. Each of them is formed by a subgroup of reactions that, according to the database, tend to belong to the same subsystem and share characteristic combinations of metabolites. The first group includes the FMNH2-dependent monooxygenase reactions, mentioned above as highly specific, with flavin mononucleotide and sulfite as reactants. The two reactions in the second plateau belong to the Glycerophospholipid Metabolism subsystem and are the only two in the database associated to acyl phosphatidylglycerol. The third cluster gathers together the eleven reactions containing 2-Demethylmenaquinone 8. It is remarkable that, in general, the reactions in these clusters have attached a high DB confidence score. Exceptions appear in the third cluster, where one reaction has an experimental evidence score less than 3 while four reactions are included for modeling reasons, so that they become interesting targets for further experimental verification. For the rest, most of the scores have values above but close to one and there are also over two hundred reactions with ratios below one. At the very tail, one finds a set of reactions that share the common characteristic of being those with the highest reaction degree and with weak or just modeling evidence. In particular, thiazole phosphate synthesis is the sole reaction involving twelve metabolites in the database and has the lowest relative score ∑ = 0.3, and noticeably also the lowest absolute S TDB score (its confidence score in the database is 2).
Relative scores conform better than absolute scores to the idea of pathways as functional modules, since they accentuate the e ect of tree distances that we expect to be related with the modular organization of the network. This topic will be explored in depth in future work.
The computational network-based TDB scoring system is able to assess, in probabilistic terms, the likelihood of reactions in metabolic reconstructions solely on the basis of the structure of the bipartite interactions between metabolites and reactions. It relies on a link prediction method adjusted to the observations that exploits the statistical regularities of the empirical network to estimate connection probabilities, that afterwards are integrated at the level of reactions. As a result, our TDB scoring system is able to break the degeneracy of currently employed scores that only use a discrete number of integers to label different levels of empirical evidence or model-based likelihood. We stress the main advantages of our procedure. First, our system permits to single out those reactions, well confirmed experimentally, that turn out to be highly specific as a potential signature of functional or evolutionary impact. Conversely, our analysis can help experimentalists to choose, among their potential targets, those reactions that appear, according to our scheme, more probable as compared to metabolic processes more prone to operate under non-standard conditions. In addition, our scoring system may provide information for unranked databases, and can be contrasted with other model-based alternatives such as steady state flux optimization solutions. When compared with a random null model that just accounts for heterogeneity in the number of connections per element, relative scores detect and quantify the tendency of groups of metabolites to aggregate or disaggregate. These distance-based correlations or anticorrelations in the underlying tree metric space raises a question worth exploring in the future in relation to functional modules.
In a broader context, many biological interactions find a natural representation in the form of bipartite networks. The ubiquity of these bipartite structures in cellular networks foretells a wide range of potential applications of the present methodology, from the estimation of gene-reaction associations in metabolic network reconstructions to the assessment of codon-gene association probabilities and of protein complexes.
Fitting the binary tree to the bipartite network
We adjust a binary tree to the observed data. The tree has M + R - 1 internal nodes at its bifurcation points and M + R leaves corresponding to the nodes of the metabolic bipartite network, M metabolites and its R reactions. Each internal node t has an associated tree probability ρ t that we transform into a distance d t = 1 - ρ t . Each pair metabolite-reaction having as lowest common ancestor t is then separated by a distance d mr = d t . Given a dendrogram, the internal probabilities ρ t are only dependent on the structure of the empirical bipartite network and can be calculated as the fraction of observed connections between leaves in each branch of the internal node over the total possible. To find the dendrogram that best fits the real data in terms of likelihood, we assume that all trees are a priori equally probable and explore the space of possibilities using a Markov chain Monte Carlo method  combined with a maximum likelihood approach, following the methodology in .
As for unipartite graphs, the variable E t stands for the number of actual edges in the empirical bigraph, in our case those that connect metabolites and reactions in the bipartite graph with t as their lowest common ancestor in the tree. The variable ε t corresponds to the total possible number of such edges given reactions and metabolites in the different branches of the common ancestor t, discounting internal combinations. In the unipartite case, ε t = L t R t , being L t and R t the number of leaves in the left and right subtrees rooted at t. In our scheme for bipartite networks, ε t = L tm R tr + L tr R tm , counting possible combinations between metabolite leaves in the left subtree and reaction leaves in the right subtrees and vice versa. Then, ρ t = E t /ε t maximize ℒ.
Starting from a random configuration, we move among all possible sets of dendrograms by performing random swaps between one of the branches of a randomly chosen internal node and the alternative branch at its father level. This exploration is appropriate because it is ergodic and fulfils detailed balance. The likelihood of the new dendrogram produced in this way is computed and the dendrogram is accepted or rejected according to the standard Metropolis-Hastings rule : the transition is accepted whenever the likelihood does not decrease and otherwise it is accepted with a probability exp(Δlogℒ) (for computational purposes, it is more convenient to work with the logarithm of the likelihood function).
After a transient period when ℒ reaches its equilibrium value (except typical fluctuations) the system reaches a stationary state where we sample over 103 dendrograms at regular intervals to produce an average measure of ρ t and so of ρ mr for each possible metabolite-reaction pair. This model, the Hierarchical Random Bipartite Graph (HRBG), is a generalization for bipartite networks of the model introduced in . As explained in the main text, in our TDB model we correct the tree distances d mr (d mr = 1 - ρ mr ) for heterogeneity in the degrees of metabolites. We renormalize the distances according to Eq. (3) to produce the connection probabilities p mr that we use in the construction of the probabilistic reaction scores.
Other alternative methods
The Configuration Model for bipartite networks
that gives the CMB probabilistic score for a metabolic reaction when its set of associated metabolites is ν.
Common neighbors  (CN) is a local similarity measure that counts the number of shared neighbors by a given pair. This measure represents a family of overlap measures quantifying similarity between nodes and a normalized version was specifically introduced for the study of the hierarchical modularity of metabolic networks and to delineate the functional modules based on the network topology . In the case of bipartite metabolic networks, we define common neighbors for a pair of metabolites as the number of reactions in which they are concurrent, o mm' , and we estimate the probability of connection metabolite-reaction as .
E. colibipartite network representation
The iAF1260 version of the K12 MG1655 strain of E. coli  provided in the BiGG database [33, 34] comprises 1039 metabolites and 2381 reactions of different nature happening in three different compartments: cytosol, periplasm, and a third symbolic one representing the extra-organism. For a meaningful conceptualization of this metabolic system as an undirected unweighted bipartite network, self-connections and dangling ends are avoided. To this end, we obviated exchange and diffusion reactions. We also neglected isomerizations and identified isomers as the same metabolite, but only whenever a reaction that carried out the isomerization could be identified in the database. This left a total of 1479 reactions that involve 976 metabolites. Finally, 5 metabolites were removed (mn2, ca2, na1, ag, and cl) because they do not enter in any reaction involving chemical transformation. See Additional File 1 for further details.
We thank Marian Boguñá for helpful discussions. This work was supported by DURSI Project No. 2009SGR1055. F. S. acknowledges financial support by MICINN Project No. FIS2006-03525. M. A. S. was supported by the Ramón y Cajal program of the Spanish Ministry of Science and by MICINN Project No. BFU2010-21847-C02-02.
- Newman MEJ: The structure and function of complex networks. SIAM Review. 2003, 45: 167-256. 10.1137/S003614450342480.View ArticleGoogle Scholar
- Dorogovtsev SN, Goltsev AV, Mendes JFF: Critical phenomena in complex networks. Rev Mod Phys. 2008, 80: 1275-1335. 10.1103/RevModPhys.80.1275.View ArticleGoogle Scholar
- Barrat A, Barthélemy M, Vespignani A: Dynamical Processes on Complex Networks. 2008, Cambridge: Cambridge University PressView ArticleGoogle Scholar
- Aittokallio T, Schwikowski V: Grap-based methods for analysing networks in cell biology. Briefings in Bioinformatics. 2006, 7: 243-255. 10.1093/bib/bbl022.View ArticlePubMedGoogle Scholar
- Barabási AL, Oltvai ZN: Network biology understanding the cell's functional organization. Nature Reviews Genetics. 2004, 5: 101-113. 10.1038/nrg1272.View ArticlePubMedGoogle Scholar
- F Képès E: Biological Networks. 2007, Singapore: World ScientificView ArticleGoogle Scholar
- Kharchenko P, Chen L, Freund Y, Vitkup D, Church GM: Identifying metabolic enzymes with multiple types of association evidence. BMC Bioinformatics. 2006, 7: 177-10.1186/1471-2105-7-177.PubMed CentralView ArticlePubMedGoogle Scholar
- Stelling J, Klamt S, Bettenbrock K, Schuster S, Gilles ED: Metabolic network structure determines key aspects of functionality and regulation. Nature. 2002, 420: 190-193. 10.1038/nature01166.View ArticlePubMedGoogle Scholar
- Guimerà R, Amaral LAN: Functional cartography of complex metabolic networks. Nature. 2005, 433: 895-900. 10.1038/nature03288.PubMed CentralView ArticlePubMedGoogle Scholar
- Motter AE, Gulbahce N, Almaas E, Barabási AL: Predicting synthetic rescues in metabolic networks. Molecular Systems Biology. 2008, 4: 168-PubMed CentralView ArticlePubMedGoogle Scholar
- Smart AG, Amaral LAN, Ottino J: Cascading failure and robustness in metabolic networks. Proc Natl Acad Sci USA. 2008, 105: 13223-13228. 10.1073/pnas.0803571105.PubMed CentralView ArticlePubMedGoogle Scholar
- Palsson BO: Systems Biology Properties of Reconstructed Networks. 2006, Cambridge: Cambridge University PressView ArticleGoogle Scholar
- Amaral LAN: A truer measure of our ignorance. Proc Natl Acad Sci USA. 2008, 105 (19): 6795-6796. 10.1073/pnas.0802459105.PubMed CentralView ArticlePubMedGoogle Scholar
- Kaltenbacha HM, Dimopoulosa S, Stelling J: Systems analysis of cellular networks under uncertainty. FEBS Letters. 2009, 583: 3923-3930. 10.1016/j.febslet.2009.10.074.View ArticleGoogle Scholar
- von Mering C, Krause R, Snel B, Cornell M, Oliver SG, Fields S, Bork P: Comparative assessment of large-scale data sets of protein-protein interactions. Nature. 2002, 417: 399-403.View ArticlePubMedGoogle Scholar
- Saito R, Suzuki H, Hayashizaki Y: Interaction generality, a measurement to assess the reliability of a protein-protein interaction. Nucleic Acids Research. 2002, 30: 1163-1168. 10.1093/nar/30.5.1163.PubMed CentralView ArticlePubMedGoogle Scholar
- Goldberg DS, Roth FP: Assessing experimentally derived interactions in a small world. Proc Natl Acad Sci USA. 2003, 100 (8): 4372-4376. 10.1073/pnas.0735871100.PubMed CentralView ArticlePubMedGoogle Scholar
- Francke C, Siezen RJ, Teusink B: Reconstructing the metabolic network of a bacterium from its genome. TRENDS in Microbiology. 2005, 13: 550-558. 10.1016/j.tim.2005.09.001.View ArticlePubMedGoogle Scholar
- Oberhardt MA, Palsson BO, Papin JA: Applications of genome-scale metabolic reconstructions. Molecular Systems Biology. 2009, 5: 320-PubMed CentralView ArticlePubMedGoogle Scholar
- Yus E, et al: Impact of Genome Reduction on Bacterial Metabolism and Its Regulation. Science. 2009, 326: 1263-1268. 10.1126/science.1177263.View ArticlePubMedGoogle Scholar
- Thiele I, Palsson BO: A protocol for generating a high-quality genome-scale metabolic reconstruction. Nature Protocols. 2010, 5: 93-121.PubMed CentralView ArticlePubMedGoogle Scholar
- Wagner A, Fell D: The small world inside large metabolic networks. Proc R Soc Lond B. 2001, 268: 1803-1810. 10.1098/rspb.2001.1711.View ArticleGoogle Scholar
- Ravasz E, Somera AL, Mongru DA, Oltvai ZN, Barabási AL: Hierarchical Organization of Modularity in Metabolic Networks. Science. 2002, 297: 1551-1555. 10.1126/science.1073374.View ArticlePubMedGoogle Scholar
- Newman MEJ, Strogatz SH, Watts DJ: Random graphs with arbitrary degree distributions and their applications. Phys Rev E. 2001, 64: 026118-View ArticleGoogle Scholar
- Holme P, Liljeros F, Edling CR, Kim BJ: Network bipartivity. Phys Rev E. 2003, 68: 056107-View ArticleGoogle Scholar
- Serrano MA, Krioukov D, Boguñá M: Self-similarity of complex networks and hidden metric spaces. Phys Rev Lett. 2008, 100: 078701-View ArticlePubMedGoogle Scholar
- Clauset A, Moore C, Newman M: Hierarchical structure and the prediction of missing links in networks. Nature. 2008, 453: 98-101. 10.1038/nature06830.View ArticlePubMedGoogle Scholar
- Guimerà R, Sales-Pardo9 M: Missing and spurious interactions and the reconstruction of complex networks. Proc Natl Acad Sci USA. 2009, 106: 22073-22078. 10.1073/pnas.0908366106.PubMed CentralView ArticlePubMedGoogle Scholar
- Ma H, Zeng AP: Reconstruction of metabolic networks from genome data and analysis of their global structure for various organisms. Bioinformatics. 2003, 19: 270-277. 10.1093/bioinformatics/19.2.270.View ArticlePubMedGoogle Scholar
- Wright J, Wagner A: Exhaustive identification of steady state cycles in large stoichiometric networks. BMC Systems Biology. 2008, 2: 61-10.1186/1752-0509-2-61.PubMed CentralView ArticlePubMedGoogle Scholar
- Jeong H, Tombor B, Albert R, Oltvai ZN, Barabási AL: The large-scale organization of metabolic networks. Nature. 2000, 407: 651-654. 10.1038/35036627.View ArticlePubMedGoogle Scholar
- Feist AM, Henry CS, Reed JL, Krummenacker M, Joyce AR, Karp PD, Broadbelt LJ, Hatzimanikatis V, Palsson BO: A genome-scale metabolic reconstruction for Escherichia coli K-12 MG1655 that accounts for 1260 ORFs and thermodynamic information. Molecular Systems Biology. 2007, 3: 121-PubMed CentralView ArticlePubMedGoogle Scholar
- Schellenberger J, Park JO, Conrad TC, Palsson BO: BiGG: a Biochemical Genetic and Genomic knowledgebase of large scale metabolic reconstructions. BMC Bioinformatics. 2010, 11: 213-10.1186/1471-2105-11-213.PubMed CentralView ArticlePubMedGoogle Scholar
- The BiGG database. [http://bigg.ucsd.edu/]
- Fawcett T: An introduction to ROC analysis. Pattern Recognition Letters. 2006, 27: 861-874. 10.1016/j.patrec.2005.10.010.View ArticleGoogle Scholar
- Newman MEJ, Watts DJ, Strogatz SH: Random graph models of social networks. Proc Natl Acad Sci USA. 2002, 99 (suppl 1): 2566-2572.PubMed CentralView ArticlePubMedGoogle Scholar
- Guillaume JL, Latapy M: Bipartite graphs as models of complex networks. Physica A Statistical and Theoretical Physics. 2006, 371: 795-813. 10.1016/j.physa.2006.04.047.View ArticleGoogle Scholar
- Lorrain F, White HC: Structural equivalence of individuals in social networks. J Math Sociol. 1971, 1: 49-80. 10.1080/0022250X.1971.9989788.View ArticleGoogle Scholar
- Hanley JA, McNeil BJ: The meaning and use of the area under a receiver characteristic (ROC) curve. Radiology. 1982, 143: 29-36.View ArticlePubMedGoogle Scholar
- Newman MEJ, Barkema GT: Monte Carlo Methods in Statistical Physics. 1999, Oxford: Oxford University PressGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.