- Software
- Open Access
JuPOETs: a constrained multiobjective optimization approach to estimate biochemical model ensembles in the Julia programming language
- David M. Bassen^{2},
- Michael Vilkhovoy^{1},
- Mason Minot^{1},
- Jonathan T. Butcher^{2} and
- Jeffrey D. Varner^{1}Email authorView ORCID ID profile
https://doi.org/10.1186/s12918-016-0380-2
© The Author(s) 2017
- Received: 8 June 2016
- Accepted: 16 December 2016
- Published: 25 January 2017
Abstract
Background
Ensemble modeling is a promising approach for obtaining robust predictions and coarse grained population behavior in deterministic mathematical models. Ensemble approaches address model uncertainty by using parameter or model families instead of single best-fit parameters or fixed model structures. Parameter ensembles can be selected based upon simulation error, along with other criteria such as diversity or steady-state performance. Simulations using parameter ensembles can estimate confidence intervals on model variables, and robustly constrain model predictions, despite having many poorly constrained parameters.
Results
In this software note, we present a multiobjective based technique to estimate parameter or models ensembles, the Pareto Optimal Ensemble Technique in the Julia programming language (JuPOETs). JuPOETs integrates simulated annealing with Pareto optimality to estimate ensembles on or near the optimal tradeoff surface between competing training objectives. We demonstrate JuPOETs on a suite of multiobjective problems, including test functions with parameter bounds and system constraints as well as for the identification of a proof-of-concept biochemical model with four conflicting training objectives. JuPOETs identified optimal or near optimal solutions approximately six-fold faster than a corresponding implementation in Octave for the suite of test functions. For the proof-of-concept biochemical model, JuPOETs produced an ensemble of parameters that gave both the mean of the training data for conflicting data sets, while simultaneously estimating parameter sets that performed well on each of the individual objective functions.
Conclusions
JuPOETs is a promising approach for the estimation of parameter and model ensembles using multiobjective optimization. JuPOETs can be adapted to solve many problem types, including mixed binary and continuous variable types, bilevel optimization problems and constrained problems without altering the base algorithm. JuPOETs is open source, available under an MIT license, and can be installed using the Julia package manager from the JuPOETs GitHub repository
Keywords
- Ensemble modeling
- Multiobjective optimization
- Julia
Background
Ensemble modeling is a promising approach for obtaining robust predictions and coarse grained population behavior in deterministic mathematical models. It is often not possible to uniquely identify all the parameters in biochemical models, even when given extensive training data [1]. Thus, despite significant advances in standardizing biochemical model identification [2], the problem of estimating model parameters from experimental data remains challenging. Ensemble approaches address parameter uncertainty in systems biology and other fields like weather prediction [3–6] by using parameter families instead of single best-fit parameter sets. Parameter families can be selected based upon simulation error, along with other criteria such as diversity or steady-state performance. Simulations using parameter ensembles can estimate confidence intervals on model variables, and robustly constrain model predictions, despite having many poorly constrained parameters [7, 8]. There are many techniques to generate parameter ensembles. Battogtokh et al., Brown et al., and later Tasseff et al. generated experimentally constrained parameter ensembles using a Metropolis-type random walk [3, 5, 9, 10]. Liao and coworkers developed methods to generate ensembles that all approach the same steady-state, for example one determined by fluxomics measurements [11]. They have used this approach for model reduction [12], strain engineering [13, 14] and to study the robustness of non-native pathways and network failure [15]. Maranas and coworkers have also applied this method to develop a comprehensive kinetic model of bacterial central carbon metabolism, including mutant data [16]. We and others have used ensemble approaches, generated using both sampling and optimization techniques, that have robustly simulated a wide variety of signal transduction processes [9, 10, 17–19], neutrophil trafficking in sepsis [20], patient specific coagulation behavior [21], uncertainty quantification in metabolic kinetic models [22] and to capture cell to cell variation [23]. Further, ensemble approaches have been used in synthetic biology to sample possible biocircuit configurations [24]. Thus, ensemble approaches are widely used to robustly simulate a variety of biochemical systems.
Identification of biochemical models requires significant training data perhaps taken from diverse sources. These real-world data sets often contain intrinsic conflicts resulting from, for example, the use of different cell lines, different measurement technologies, different reagent vendors or lots, uncontrollable experimental artifacts or general cross laboratory variability. Parameter ensembles that optimally balance these inherent conflicts lead to more robust model performance. Multiobjective optimization is an ensemble generation technique that naturally balances conflicts in noisy training data [25]. Multiobjective optimization has been used to identify signal transduction models [18, 23], for the design of synthetic circuits [24], to design the folding behaviors of novel RNAs [26], to design bioprocesses [27], and to understand bacterial adaptation [28]. Thus, it is a widely used approach for a variety of biochemical applications. Previously, we developed the Pareto Optimal Ensemble Technique (POETs) algorithm to address the challenge of competing or conflicting training objectives. POETs, which integrates simulated annealing (SA) and multiobjective optimization through the notion of Pareto rank, estimates parameter ensembles which optimally trade-off between competing (and potentially conflicting) experimental objectives [29]. However, the previous implementation of POETs, in the Octave programming language [30], suffered from poor performance and was not configurable. For example, Octave-POETs does not accommodate user definable objective functions, bounds and problem constraints, cooling schedules, different variable types e.g., a mixture of binary and continuous design variables or custom diversity generation routines. Octave-POETs was also not well integrated into a package or source code management (SCM) system. Thus, upgrades to the approach containing new features, or bug fixes were not centrally managed.
Implementation
In this software note, we present an open-source implementation of the Pareto optimal ensemble technique in the Julia programming language (JuPOETs). JuPOETs takes advantage of the unique features of Julia to address many of the shortcomings of the previous implementation. Julia is a cross-platform, high-performance programming language for technical computing that has performance comparable to C but with syntax similar to MATLAB/Octave and Python [31]. Julia also offers a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive function library. Further, the architecture of JuPOETs takes advantage of the first-class function type in Julia allowing user definable behavior for all key aspects of the algorithm, including objective functions, custom diversity generation logic, linear/non-linear parameter constraints (and parameter bounds constraints) as well as custom cooling schedules. Julia’s ability to naturally call other languages such as Python or C also allows JuPOETs to be used with models implemented in a variety of languages across many platforms. Additionally, Julia offers a built-in package manager which is directly integrated with GitHub, a popular web-based Git repository hosting service offering distributed revision control and source code management. Thus, JuPOETs can be adapted to many problem types, including mixed binary and continuous variable types, bilevel problems and constrained problems without altering the base algorithm, as was required in the previous POETs implementation.
JuPOETs optimization problem formulation
The quantity O _{ j } denotes the j ^{ t h } objective function (\(j=1,2, \ldots, \mathcal {K}\)), typically the sum of squared errors for the j ^{ t h } data set for biochemical modeling applications. The terms \(\mathbf {f} \left (t,\mathbf {x}(t,\mathbf {p}),\dot {\mathbf {x}}(t,\mathbf {p}),\mathbf {u}(t),\mathbf {p}\right)\) denote the system of model equations (e.g., differential equations, differential algebraic equations or linear/non-linear algebraic equations) where p denotes the decision variable vector e.g., unknown model parameters (\(\mathcal {D}\times 1\)). In typical biochemical modeling applications, the model equations f(·) are a system of continuous real-valued non-linear differential equations that comprise a kinetic model, but other types of models e.g., stoichiometric models are also common. The quantity t denotes time, x(t,p) denotes the model state (with an initial state x _{0}), and u(t) denotes an input vector. The decision variables (e.g., kinetic parameters) can be subject to bounds constraints, where \(\mathcal {L}\) and \(\mathcal {U}\) denote the lower and upper bounds, respectively as well as \(\mathcal {C}\) problem specific constraints \(g_{i}\left (t,\mathbf {x}(t,\mathbf {p}),\mathbf {u}(t),\mathbf {p}\right),i=1,\ldots,\mathcal {C}\). The decision variables p are typically real-valued kinetic constants, or metabolic fluxes in the case of stoichiometric models. However, other variables types e.g., binary or categorical decision variables can also be accommodated.
where λ denotes the penalty parameter (λ=100 by default). However, because both the neighbor and objective functions are user defined, different constraint implementations are easily defined.
To use JuPOETs, the user specifies the neighbor, acceptance, cooling and objective functions along with an initial decision variable guess. Default implementations of the neighbor, acceptance and cooling functions can be used directly, or they can be overridden by user defined logic. However, the user must provide an implementation of the objective function and provide an initial decision variable guess. Lastly, if the user is operating JuPOETs in hybrid mode, then a refinement function pointer must also be specified. Hybrid mode temporarily switches the search from a multiobjective to a single objective problem, where the sum of the objective functions can be used to update the best (or initial) parameter guess. The specific hybrid mode search logic is up to the user; by default hybrid mode is off, and the default refinement implementation is simply a pass through function. However, we have shown previously that POETs operated in hybrid mode (where the single objective problem used a pattern search approach) had better performance that POETs alone [29]. Thus, hybrid mode is generally recommended for most applications. In addition, there are several user configurable parameters that can be adjusted to control the performance of JuPOETs: maximum_number_of_iterations controls the number of iterations per temperature (default 20); rank_cutoff controls the upper rank bound on the solution archive (default 5); temperature_min controls the minimum temperature after which JuPOETs returns the error and solution archives (default 0.001); show_trace controls the level of output shown to the user (default true). After the completion of the run, JuPOETs returns the parameter solution archive \(\mathcal {S}\), objective archive \(\mathcal {O}\) and rank archive \(\mathcal {R}\). The parameter solution archive \(\mathcal {S}\) contains is an \(\mathcal {D}\times \mathcal {A}\) array, where \(\mathcal {A}\) denotes the number of solutions in the archive when JuPOETs terminated. On the other hand, the objective archive \(\mathcal {O}\) is an \(\mathcal {K}\times \mathcal {A}\) array containing the performance values for each objective corresponding the columns of \(\mathcal {S}\). Lastly, JuPOETs returns the rank archive \(\mathcal {R}\) which is an \(\mathcal {A}\times {1}\) array of Pareto ranks corresponding to the columns of \(\mathcal {S}\). One technical note, if JuPOETs is run from multiple starting locations, and the archives from each of these runs is combined into a single collective archive, the combined parameter rank archive may become invalid. In these cases, it is required to re-rank the parameter sets using the built-in rank function to produce a collective parameter ranking.
Results and discussion
Multi-objective optimization test problems. We tested the JuPOETs implementation on three two-dimensional test problems, with one-, two- and three-dimensional parameter vectors. Each problem had parameter bounds constraints, however, on the Binh and Korn function had additional non-linear problem constraints. For the Fonesca and Fleming problem, N = 3
Name | Dimension | Function | Domain | Constraints |
---|---|---|---|---|
Schaffer function | 1 | O _{1}(x)=x ^{2} | −10≤x≤10 | |
O _{2}(x)=(x−2)^{2} | ||||
Binh and Korn function | 2 | O _{1}(x,y)=4x ^{2}+4y ^{2} | 0≤x≤5 | g _{1}(x,y)=(x−5)^{2}+y ^{2}≤25 |
O _{2}(x,y)=(x−5)^{2}+(y−5)^{2} | 0≤x≤3 | g _{2}(x,y)=(x−8)^{2}+(y+3)^{2}≤7.7 | ||
Fonseca and Fleming function | 3 | \(O_{1}(x_{i})= 1 - \text {exp} \left (- \sum \limits ^{N}_{i= 1} \left (x_{i} - \frac {1}{\sqrt {N}}\right)^{2} \right) \) | −4≤x _{ i }≤4 | |
\(O_{2}(x_{i})= 1 - \text {exp} \left (- \sum \limits ^{N}_{i= 1} \left (x_{i} + \frac {1}{\sqrt {N}}\right)^{2} \right) \) |
where z _{ sl } denotes the uptake flux of substrate s through mode l. Each unknown kinetic parameter was continuous and real-valued, and subject to bounds constraints: \(\mathcal {L} \leq \mathbf {p} \leq \mathcal {U}\).
Currently, JuPOETs does not consider parameter identifiability when constructing parameter ensembles. Although JuPOETs produces parameter estimates that give model performance similar to the training data, we do not have strict statistical confidence that the true parameter values are contained within the ensemble. However, despite this, ensembles produced by POETs can be predictive [18, 23]. Thus, JuPOETs produces a collection of parameters that are constrained by the performance of the model, and not by specific hypotheses regarding the individual values of the raw model parameters. Of course, knowledge of specific parameter values, or the relationship between parameter combinations, can be used to inform the search through either bounds or problem specific constraints (for example, as demonstrated in the first example problem).
Conclusions
In this software note, we presented JuPOETs, a multiobjective technique to estimate parameter ensembles in the Julia programming language. JuPOETs is open source, and available for download under an MIT license from the JuPOETs GitHub repository at https://github.com/varnerlab/POETs.jl. We demonstrated JuPOETs on a suite of algebraic test problems, and a proof-of-concept ODE based biochemical model. While JuPOETs outperformed (and was significantly more flexible) than the previous Octave implementation, there are several areas that could be explored further. First, JuPOETs should be compared with other multiobjective evolutionary algorithms (MOEAs) to determine its relative performance on test and real world problems. Many evolutionary approaches e.g., the non-dominated sorting genetic algorithm (NSGA) family of algorithms, have been adapted to solve multiobjective problems [36, 37]. However, since there is a lack of open source Julia implementations of these alternative approaches, we did not benchmark the relative performance of JuPOETs in this note. One advantage that JuPOETs may have when compared to a strictly evolutionary approaches, is the inclusion of a local refinement step (hybrid mode), which temporarily reduces the problem to a single objective formulation. Previously, POETs run in hybrid mode led to better convergence on a proof-of-concept signal transduction model compared to the same approach without the hybrid refinement step [29]. Other hybrid multiobjective methods have also been shown to be more efficient than evolutionary approaches alone, for a variety of biochemical optimization problems [24, 38]. Thus, there are several different algorithms that we can use to benchmark, and improve the performance of JuPOETs, after we implement them in Julia. Another strategy to improve the performance of JuPOETs is to reduce the number (or cost) of function evaluations that are required to obtain optimal or near optimal solutions. For example, in many real world parameter estimation problems, the bulk of the execution time is spent evaluating the objective functions. One strategy to improve JuPOETs performance could be to optimize surrogates [39], while another would be parallel execution of the objective functions. Currently, JuPOETs serially evaluates the objective function vector. However, parallel evaluation of the objective functions e.g., using the parallel Julia macro or other techniques, could be implemented without significantly changing the JuPOETs run loop. Taken together, JuPOETs demonstrated improved flexibility, and performance over POETs in parameter identification and ensemble generation for multiple objectives. JuPOETs has the potential for widespread use due to the flexibility of the implementation, and the high level syntax and distribution tools native to the Julia programming language.
Availability and requirements
JuPOETs is open source, available under an MIT software license. The JuPOETs source code is freely available from the JuPOETs GitHub repository at https://github.com/varnerlab/POETs.jl. All samples used in this study are included in the sample/biochemical and sample/test_functions subdirectories of the JuPOETs GitHub repository. JuPOETs can be run on all common.Operating system environments: (Linux, Mac OS, Windows).
Declarations
Acknowledgements
We gratefully acknowledge Ani Chakrabarti, Russell Gould and Kathy Rogers for their input and suggestions regarding new features to include into JuPOETs. We also gratefully acknowledge the suggestions from the anonymous reviewers to improve this manuscript and JuPOETs.
Funding
This study was supported by an award from the National Science Foundation (NSF CBET-0955172) and the National Institutes of Health (NIH HL110328) to J.B, and by a National Science Foundation Graduate Research Fellowship (DGE-1144153) to D.B. Lastly, J.V was supported by an award from the US Army and Systems Biology of Trauma Induced Coagulopathy (W911NF-10-1-0376).
Authors’ contributions
JV developed the software presented in this study. MM and MV developed the proof-of-concept biochemical model. The manuscript was prepared and edited for publication by DB, JB and JV. All authors read and approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.
Consent for publication
Not applicable.
Ethics approval and consent to participate
Not applicable.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
- Gadkar KG, Varner J, Doyle FJ. Model identification of signal transduction networks from data using a state regulator problem. Syst Biol (Stevenage). 2005; 2(1):17–30.View ArticleGoogle Scholar
- Gennemark P, Wedelin D. Benchmarks for identification of ordinary differential equations from time series data. Bioinformatics. 2009; 25(6):780–6. doi:http://dx.doi.org/10.1093/bioinformatics/btp050.
- Battogtokh D, Asch DK, Case ME, Arnold J, Shüttler HB. An ensemble method for identifying regulatory circuits with special reference to the qa gene cluster of Neurospora crassa. Proc Natl Acad Sci USA. 2002; 99(26):16904–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Kuepfer L, Peter M, Sauer U, Stelling J. Ensemble modeling for analysis of cell signaling dynamics. Nat Biotechnol. 2007; 25(9):1001–6. doi:http://dx.doi.org/10.1038/nbt1330.
- Brown KS, Sethna JP. Statistical mechanial approaches to models with many poorly known parameters. Phys Rev E. 2003; 68:021904–19.View ArticleGoogle Scholar
- Palmer TN, Shutts GJ, Hagedorn R, Doblas-Reyes FJ, Jung T, Leutbecher M. Representing model uncertainty in weather and climate prediction. Ann Rev Earth Planetary Sci. 2005; 33:163–93.View ArticleGoogle Scholar
- Gutenkunst RN, Waterfall JJ, Casey FP, Brown KS, Myers CR, Sethna JP. Universally sloppy parameter sensitivities in systems biology models. PLoS Comput Biol. 2007; 3(10):1871–8. doi:http://dx.doi.org/10.1371/journal.pcbi.0030189.
- Song S, Varner J. Modeling and analysis of the molecular basis of pain in sensory neurons. PLoS ONE. 2009; 4:6758–72.View ArticleGoogle Scholar
- Tasseff R, Nayak S, Salim S, Kaushik P, Rizvi N, Varner JD. Analysis of the molecular networks in androgen dependent and independent prostate cancer revealed fragile and robust subsystems. PLoS ONE. 2010; 5(1):8864. doi:http://dx.doi.org/10.1371/journal.pone.0008864.
- Tasseff R, Nayak S, Song SO, Yen A, Varner JD. Modeling and analysis of retinoic acid induced differentiation of uncommitted precursor cells. Integr Biol (Camb). 2011; 3(5):578–91. doi:http://dx.doi.org/10.1039/c0ib00141d.
- Tran LM, Rizk ML, Liao JC. Ensemble modeling of metabolic networks. Biophys J. 2008; 95(12):5606–17. doi:http://dx.doi.org/10.1529/biophysj.108.135442.
- Tan Y, Rivera JGL, Contador CA, Asenjo JA, Liao JC. Reducing the allowable kinetic space by constructing ensemble of dynamic models with the same steady-state flux. Metab Eng. 2011; 13(1):60–75. doi:http://dx.doi.org/10.1016/j.ymben.2010.11.001.
- Contador CA, Rizk ML, Asenjo JA, Liao JC. Ensemble modeling for strain development of l-lysine-producing escherichia coli. Metab Eng. 2009; 11(4–5):221–33. doi:http://dx.doi.org/10.1016/j.ymben.2009.04.002.
- Tan Y, Liao JC. Metabolic ensemble modeling for strain engineers. Biotechnol J. 2012; 7(3):343–53. doi:http://dx.doi.org/10.1002/biot.201100186.
- Lee Y, Lafontaine Rivera JG, Liao JC. Ensemble modeling for robustness analysis in engineering non-native metabolic pathways. Metab Eng. 2014; 25:63–71. doi:http://dx.doi.org/10.1016/j.ymben.2014.06.006.
- Khodayari A, Zomorrodi AR, Liao JC, Maranas CD. A kinetic model of escherichia coli core metabolism satisfying multiple sets of mutant flux data. Metab Eng. 2014; 25:50–62. doi:http://dx.doi.org/10.1016/j.ymben.2014.05.014.
- Luan D, Zai M, Varner JD. Computationally derived points of fragility of a human cascade are consistent with current therapeutic strategies. PLoS Comput Biol. 2007; 3(7):142. doi:http://dx.doi.org/10.1371/journal.pcbi.0030142.
- Song SO, Varner J. Modeling and analysis of the molecular basis of pain in sensory neurons. PLoS ONE. 2009; 4(9):6758. doi:http://dx.doi.org/10.1371/journal.pone.0006758.
- Nayak S, Siddiqui JK, Varner JD. Modelling and analysis of an ensemble of eukaryotic translation initiation models. IET Syst Biol. 2011; 5(1):2. doi:http://dx.doi.org/10.1049/iet-syb.2009.0065.
- Song SO, Song SOK, Hogg J, Peng ZY, Parker R, Kellum JA, Clermont G. Ensemble models of neutrophil trafficking in severe sepsis. PLoS Comput Biol. 2012; 8(3):1002422. doi:http://dx.doi.org/10.1371/journal.pcbi.1002422.
- Luan D, Szlam F, Tanaka KA, Barie PS, Varner JD. Ensembles of uncertain mathematical models can identify network response to therapeutic interventions. Mol Biosyst. 2010; 6(11):2272–86. doi:http://dx.doi.org/10.1039/b920693k.
- Andreozzi S, Miskovic L, Hatzimanikatis V. iSCHRUNK–in silico approach to characterization and reduction of uncertainty in the kinetic models of genome-scale metabolic networks. Metab Eng. 2016; 33:158–68. doi:http://dx.doi.org/10.1016/j.ymben.2015.10.002.
- Lequieu J, Chakrabarti A, Nayak S, Varner JD. Computational modeling and analysis of insulin induced eukaryotic translation initiation. PLoS Comput Biol. 2011; 7(11):1002263. doi:http://dx.doi.org/10.1371/journal.pcbi.1002263.
- Otero-Muras I, Banga JR. Multicriteria global optimization for biocircuit design. BMC Syst Biol. 2014; 8:113. doi:http://dx.doi.org/10.1186/s12918-014-0113-3.
- Handl J, Kell DB, Knowles J. Multiobjective optimization in bioinformatics and computational biology. IEEE/ACM Trans Comput Biol Bioinform. 2007; 4(2):279–92. doi:http://dx.doi.org/10.1109/TCBB.2007.070203.
- Taneda A. Multi-objective optimization for RNA design with multiple target secondary structures. BMC Bioinformatics. 2015; 16(1):280. doi:http://dx.doi.org/10.1186/s12859-015-0706-x.
- Sendin J, Otero-Muras I, Alonso AA, Banga J. Improved Optimization Methods for the Multiobjective Design of Bioprocesses. Ind Eng Chem Res. 2006; 45:8594–603.View ArticleGoogle Scholar
- Angione C, Lió P. Predictive analytics of environmental adaptability in multi-omic network models. Sci Rep. 2015; 5:15147. doi:http://dx.doi.org/10.1038/srep15147.
- Song SO, Chakrabarti A, Varner JD. Ensembles of signal transduction models using pareto optimal ensemble techniques (poets). Biotechnol J. 2010; 5(7):768–80. doi:http://dx.doi.org/10.1002/biot.201000059.
- Eaton JW, Bateman D, Hauberg S. GNU octave version 3.0.1 manual: a high-level interactive language for numerical computations. North Charleston: CreateSpace Independent Publishing Platform; 2009.Google Scholar
- Bezanson J, Edelman A, Karpinski S, Shah VB. Julia: A fresh approach to numerical computing. arXiv CoRR. abs/1411.1607. Ithaca: Cornell University; 2014.Google Scholar
- Kirkpatrick S, Gelatt Jr CD, Vecchi MP. Optimization by simulated annealing. Science. 1983; 220(4598):671–80. doi:http://dx.doi.org/10.1126/science.220.4598.671.
- Fonseca CM, Fleming PJ. Genetic Algorithms for Multiobjective Optimization: Formulation, Discussion and Generalization. arXiv CoRR Publisher. In: Proceedings of the 5th International Conference on Genetic Algorithms. Ithaca: Cornell University: 1993. p. 416–23.Google Scholar
- Kim J, Varner J, Ramkrishna D. A hybrid model of anaerobic e. coli gjt001: Combination of elementary flux modes and cybernetic variables. Biotechnol Prog. 2008; 24(5):993–1006. doi:http://dx.doi.org/10.1002/btpr.73.
- Schuster S, Fell DA, Dandekar T. A general definition of metabolic pathways useful for systematic organization and analysis of complex metabolic networks. Nat Biotechnol. 2000; 18(3):326–2. doi:http://dx.doi.org/10.1038/73786.
- Kalyanmoy D, Pratap A, Agarwal S, Meyarivan T. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE Trans Evol Comp. 2002; 6:182–97.View ArticleGoogle Scholar
- Huband S, Hingston P, Barone L, While L. A review of multiobjective test problems and a scalable test problem toolkit. IEEE Trans Evol Comp. 2006; 10:477–506.View ArticleGoogle Scholar
- Sendin JOH, Otero-Muras I, Alonso AA, Banga JR. Improved optimization methods for multiple objective design of bioprocesses. Ind Eng Chem Res. 2006; 45:8594–603.View ArticleGoogle Scholar
- Booker AJ, Dennis JE, Frank PD, Serafini DB, Torczon V, Trosset MW. A rigorous framework for optimization of expensive functions by surrogates. Struct Optim. 1999; 17:1–13.View ArticleGoogle Scholar