MISCORE: a new scoring function for characterizing DNA regulatory motifs in promoter sequences

Abstract

Background

Computational approaches for finding DNA regulatory motifs in promoter sequences are useful to biologists in terms of reducing the experimental costs and speeding up the discovery process of de novo binding sites. It is important for rule-based or clustering-based motif searching schemes to effectively and efficiently evaluate the similarity between a k-mer (a k-length subsequence) and a motif model, without assuming the independence of nucleotides in motif models or without employing computationally expensive Markov chain models to estimate the background probabilities of k-mers. Also, it is interesting and beneficial to use a priori knowledge in developing advanced searching tools.

Results

This paper presents a new scoring function, termed as MISCORE, for functional motif characterization and evaluation. Our MISCORE is free from: (i) any assumption on model dependency; and (ii) the use of Markov chain model for background modeling. It integrates the compositional complexity of motif instances into the function. Performance evaluations with comparison to the well-known Maximum a Posteriori (MAP) score and Information Content (IC) have shown that MISCORE has promising capabilities to separate and recognize functional DNA motifs and its instances from non-functional ones.

Conclusions

MISCORE is a fast computational tool for candidate motif characterization, evaluation and selection. It enables to embed priori known motif models for computing motif-to-motif similarity, which is more advantageous than IC and MAP score. In addition to these merits mentioned above, MISCORE can automatically filter out some repetitive k-mers from a motif model due to the introduction of the compositional complexity in the function. Consequently, the merits of our proposed MISCORE in terms of both motif signal modeling power and computational efficiency will make it more applicable in the development of computational motif discovery tools.

Background

Gene transcription is controlled by the essential interactions between Transcription Factor Binding Sites (TFBSs, or simply Binding Sites) and Transcription Proteins known as Transcription Factors (TFs) [1]. Understanding these interactions requires a knowledge on all binding sites associated with their TFs and cis-regulatory modules. Hence, discovering unknown motifs (i.e., a collection of binding sites) in co-expressed genes or finding de novo binding sites associated with a known TF is crucial to understand the gene regulatory mechanisms [2–4]. Experimental approaches for finding DNA motifs are laborious and expensive [5, 6]. Additionally, experimental techniques such as ChIP-chip [7], ChIP-seq [8] and micro-array technology are mostly incapable of predicting specific locations of the binding sites.

It was the biological significance of the cost-effective identification of the DNA motifs that computational motif discovery has received considerable attention in the last two decades. In addition to being cost-effective and time-efficient, the nature of computational techniques offers the fastest and usually the easiest means of adopting rapidly emerging new and revised understandings on the biological process to produce more sensible motif discovery results. Despite being enormously attempted, an effective motif discovery performance by the computational approaches still remains challenging [9–11]. This is partly due to the lack of effective characterization on regulatory motifs which helps in distinguishing the functional motifs from the non-functional ones.

Due to the functional significance in gene regulation, motifs are evolutionarily conserved. Hence, motif instances appear to be rather similar to each other despite having variability in their nucleotide compositions [12]. Motif instances are rarely found in the background sequences, which is often termed as the background rareness. Also, functional motifs are often over-represented in the regulatory regions (foreground) compared to the backgrounds [13–16]. Thus, a motif's background-to-foreground appearance ratio should be smaller than the random ones. Over-representation can be similarly interpreted with the rareness characteristic. However, they are typically expressed with different statistical representations. Another useful characteristic of functional motifs is related to the compositional complexity of the nucleotides, which is termed as motif complexity [17].

Information Content (IC) [18] and Maximum a Posteriori (MAP) [19] score are two conventional motif scoring schemes that are widely adopted in evaluating and ranking candidate motifs. They are capable of characterizing the model conservation and the background rareness properties of the functional motifs. However, they suffer from the following shortcomings:

• 1. IC evaluates a motif by quantifying the relative entropy of the motif PFM (Positional Frequency Matrix) under assumption of model independence. This assumption on model independence is fundamentally weak as shown in [20–23].

• 2. MAP, on the other hand, requires a higher order Markov chain model to estimate the background probabilities [24] prior to motif evaluation. Its computational time and cost increases along with the increment of the order of the used Markov chain model. Also, MAP score can not be used to evaluate the similarity between a k-mer and a motif model, which is essential in computational motif discovery exercises.

• 3. Both IC and MAP score ignore the motif-complexity feature in the evaluation of the candidate motifs. Hence, a complexity score-based filtering [17] has to be used in candidate motif evaluation. The complexity threshold is empirically set in the filtering process that needs human intervention and careful attempts.

• 4. Computational motif discovery can be guided by some known motif models as useful a priori knowledge (pk). Motif evaluation in terms of ranking then becomes a motif-to-motif similarity task. Unfortunately, IC and MAP score are not able to embed the pk models in scoring.

Motivated by the above issues, this paper introduces a new motif scoring function, termed as MISCORE (mismatch-based matrix similarity scores), to quantify similarity between a k-mer and a motif PFM using a mismatch computation on the nucleotides. By evaluating each instance k-mer (a candidate binding site) of a motif, MISCORE can quantify the likeliness of the candidate motif to be functional by a combined characterization on the model conservation, the background rareness and the compositional complexity. Our proposed MISCORE share the following three remarkable features: (i) computational efficiency due to its simplicity; (ii) free from any assumption on model dependency; and (iii) an embedability of a priori knowledge in motif scoring. An extension of MISCORE, that adopts a biologically inclined pattern localization approach for an improved recognition of the functional motifs, is also reported in this paper.

Experiments on 33 benchmark DNA datasets have been carried out for evaluating the performance of MISCORE with comparison to IC and MAP score. Firstly, we examine how well these metrics can separate the functional motifs from the random ones. Secondly, we are interested in learning how well they can recognize the functional motifs from a set of putative motif models in terms of candidate ranking. Lastly, we evaluate the effectiveness of MISCORE in recognizing the functional motifs using pk models. The experimental results are found promisingly supportive to MISCORE.

Over-representation is a widely recognized numerical feature for characterizing functional motifs [13–15], that typically differs from the statistical quantification of the background rareness property. Due to their common objective of motif characterization, correlating them through a single framework is fundamentally meaningful and it has not been addressed sufficiently in the literature. MISCORE can be utilized as a similarity metric to perform this correlation as detailed in the latter portion of this paper.

Methods

This section describes MISCORE and its localized version in details. For the sake of completeness, some preliminaries are given, including the notations and the k-mer encoding scheme used throughout this paper, followed by a preliminary introduction on the motif complexity score [17], Information Content [18] and the Maximum a Posteriori score [19].

Preliminaries

Model representation

In this paper, Positional Frequency Matrix (PFM) is employed as the motif model [18]. The PFM-based motif model, denoted by M, is a matrix, i.e., M = [f(b _i , i)]_4×k, where b _i ∈ χ = {A, C, G, T} and i = 1, . . ., k, and each entry f(b _i , i) represents the probability of nucleotide b _i at position i. Similarly, a k-mer K _s = q₁q₂ . . . q _k is encoded as a binary matrix K = [k(b _i , i)]_4×kwith k(q _i , i) = 1 and k(b _i , i) = 0 for b _i ≠ q _i . For example, a k-mer K _s = AGCGTGT can be encoded as,

K=encode\left(K_s\right)=\begin{array}{c}A\hfill \\ C\hfill \\ G\hfill \\ T\hfill \end{array}{\left[\begin{array}{ccccccc}1\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & 1\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill \\ 0\hfill & 1\hfill & 0\hfill & 1\hfill & 0\hfill & 1\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 1\hfill & 0\hfill & 1\hfill \end{array}\right]}_{4\times k}.

For a given binary encoded set of k-mers, S = {K₁, K₂, ..., K _P }, the motif PFM model M _S can be computed by M_S=\frac{1}{P}{\sum }_{i=1}^P{K}_i.

Model complexity

Motif discovery tools often return models with low complexity, that show a repetitive occurrence of nucleotides. Hence, a motif-complexity score was proposed in [17] to filter out models with lower complexities, that is,

c\left(M\right)={\left(\frac{1}{4}\right)}^k{\displaystyle \prod _{\forall b_i\in \chi }}{\left(\frac{k}{{\sum }_{i=1}^{k}f\left(b_i,i\right)}\right)}^{{\Sigma }_{i=1}^{k}f\left(b_i,i\right)},

(1)

where k is the length of k-mers and f(b _i , i) is the observed frequency of the base b _i at position i in the model M. Here, the complexity score lies in [(1/4) ^k , 1], where 1 refers to a fully complex motif PFM.

Maximum a posteriori (MAP) score

MAP score [19] is a powerful quantifier that evaluates the merit of a candidate motif (a set of k-mers, S) by considering its model conservation and the background rareness. The background rareness of S is computed using a higher order Markov chain model [24]. For each K ∈ S, this model can produce an estimation of background probability, namely p(K|B), for a given background model B:

p\left(K|B\right)=p\left(b_1,b_2,\dots ,b_m\right){\displaystyle \prod _{i=m+1}^k}p\left(b_i|b_{i-m},b_{i-m+1},\dots ,b_{i-1}\right),

(2)

where m is the Markov chain order; k is the length of k-mers; p(b₁, b₂, . . ., b _m ) is the estimated probability of subsequence b₁, b₂, . . ., b _m and p(b _i |b_i-m, b_i-m+1, . . ., b_i-1) is the conditional probability of the subsequence b _i under b_i-m, b_i-m+1, . . ., b_i-1occurrence constraint. Then, for the candidate motif S, MAP score can be expressed as,

MAP\left(S\right)=-\frac{\text{ln}\left(\left|S\right|\right)}{k}\left(E\left(S\right)+\frac{1}{\left|S\right|}{\displaystyle \sum _{\forall K\in S}}\text{ln}p\left(K|B\right)\right),

(3)

where |S| is the cardinality of the set S and E(S) is the entropy [25] of the PFM (M), expressed as,

E\left(S\right)=-{\displaystyle \sum _{i=1}^k}{\displaystyle \sum _{\forall b_i\in \chi }}f\left(b_i,i\right){\text{log}}_{2}f\left(b_i,i\right).

(4)

A higher MAP score indicates a better likeliness of the motif S to be functional.

Information content (IC)

IC [18], measuring the average binding energy of the k-mers set S, can be given by,

IC\left(M\right)={\displaystyle \sum _{i=1}^k}{\displaystyle \sum _{\forall b_i\in \chi }}f\left(b_i,i\right)\text{ln}\left(\frac{f\left(b_i,i\right)}{p\left(b_i\right)}\right),

(5)

where f(b _i , i) is frequency of the base b _i at position i in the model M, and p(b _i ) is the pre-computed background frequency of the nucleotide base b _i . A higher IC score of a candidate motif indicates a better potential of being a functional one.

MISCORE for motif characterization

MISCORE is a new scoring function for modeling motif signals that uses a combined characterization on the model conservation, the background rareness and the compositional complexity of functional motifs. It quantifies a similarity between a k-mer K and a putative model M with respect to the background reference model M _ref , that is,

r\left(K,M\right)=\frac{d\left(K,M\right)}{d\left(K,M_{ref}\right)+c\left(K\right)},

(6)

where d(K, M) is defined as a generalized Hamming distance, expressed as,

d\left(K,M\right)=1-\frac{1}{k}{\displaystyle \sum _{i=1}^k}{\displaystyle \sum _{\forall b_i\in \chi }}f\left(b_i,i\right)k\left(b_i,i\right),

(7)

where f(b _i , i) and k(b _i , i) are the observed frequencies of base b _i at position i in M and K, respectively.

Motivated by the well-known Gini index to quantify impurity of data clusters, we define c(K) in Eq (6) to compute the compositional complexity of K as follows:

c\left(K\right)=\frac{4}{3}\left[1-\frac{1}{k^2}{{\displaystyle \sum _{\forall b_i\in \chi }}\left({\displaystyle \sum _{i=1}^k}k\left(b_i,i\right)\right)}^2\right],

(8)

where the complexity is scored according to the distribution of bases (A, C, G, T) in the K. An equal distribution gives the maximum score of 1 and a dominant distribution, i.e., a nucleotide appears at all positions of the K, gives the minimum complexity of 0. In Eq (6), the score range for both d(K, M _ref ) and c(K) is 0[1]. The complexity measure given in Eq (6) helps in automatically eliminating the low-complex motifs from the top rank. In this way, an empirical threshold-based filtering [17] for filtering the low-complex candidate motifs can be avoided.

While no pk model associated with the target motifs is available, then we need to employ some searching tools to generate a model that is qualified to be an approximation M of the target motifs. Then, this putative model is essentially derived from the information embedded in the input sequences by the employed search algorithms. For instance, in the clustering type of motif finding algorithms [17, 26, 27], the putative models can be obtained by grouping k-mers based on a similarity metric.

Binding sites are evolutionarily constrained with limited mutations, hence a K can be a putative motif instance if d(K, M) <d(K, M _ref ) holds, which implies a smaller mismatch to the putative model M than the background reference model M _ref . Note that the M _ref is a PFM that can be constructed by all k-mers from the background sequences. For a large sized background, each column of the M _ref approximates the nucleotides background frequency. Thus, the M _ref can be conveniently composed of the nucleotides pre-computable background frequency in each column. Large sequence-portions that have a minimal chance of having the true binding sites can be taken as the backgrounds, e.g., random chunks of large genomic portions or a large collection of upstream regions from the relevant species. Note that a smaller r(K, M) score characterizes a higher similarity of that K to M in respect to its dissimilarity to M _ref and a better nucleotide complexity in K, which implies a combined characterization on the model conservation, the background rareness and the compositional complexity.

A mathematical expectation of the MISCORE values of a set of k-mers can be viewed as a metric to characterize the candidate motifs. Given a set of k-mers S and its PFM model M _S , a MISCORE-based Motif Score (MMS), denoted as R(S), can be evaluated by,

R\left(S\right)=\frac{1}{\left|S\right|}{\displaystyle \sum _{\forall K\in S}}r\left(K,M_S\right),

(9)

where | * | is the set cardinality and r(*, *) is the MISCORE given in Eq (6). A smaller MMS score indicates a better potential for a candidate motif to be functional.

Remark

Initially, MISCORE was introduced in [28] to quantify a mismatch-based similarity between a K and a model M _S , i.e., d\left(K,M_S\right)=k-{\sum }_{i=1}^k{\sum }_{\forall b_i\in \chi }f\left(b_i,i\right)k\left(b_i,i\right). A corresponding MMS was defined by MMS\left(S\right)=\frac{1}{\left|S\right|}{\sum }_{\forall K\in S}d\left(K,M_S\right), and utilized as a motif scoring function to quantify the conservation property of a motif S. In [29], an improved version of MISCORE, termed as relative-MISCORE, was introduced to characterize a motif's conservation and the rareness properties by introducing a background reference model M _ref in the MISCORE computation. Let r(K, M _S ) denote a relative similarity between a K and a model M _S . Then, it can be computed by r(K, M _S ) = d(K, M _S )/d(K, M _ref ) that results in a relative-MMS: RMMS\left(S\right)=\frac{1}{\left|S\right|}{\sum }_{\forall K\in S}r\left(K,M_S\right). As a new scoring function, it was employed as a fitness function in our GAPK framework for motif discovery. In this paper, we introduce a compositional complexity term in the relative-MISCORE as shown in Eq (6), which improves our previous work by preventing k-mers with repetitive nucleotides from motif models. This new characterization simultaneously addresses the model conservation, background-rareness and the compositional complexity properties of the regulatory motifs, which makes the present MISCORE functionally advantageous than IC, MAP score and the previous MISCORE versions. It should be pointed out that other forms of characterization on regulatory motifs exist, provided that they can model the motif signals effectively and efficiently.

Observation: Experiments on real DNA datasets demonstrated that R scores of the functional motifs are with statistically significant p-values and z-scores, that can be computed using large collections of (i) random and (ii) conserved models, generated from the respective promoter sequences. Results obtained on 12 real DNA datasets are presented in Table 1, showing that R scores of the true models M _t (functional motif) are mostly rare with comparison to the conserved-models M _c , indicated by close to zero p-values. Each M _c is generated by a random selection of a seed K from a random sequence and by collecting the most similar Ks to the seed, only one was picked from each sequence. It shows that, despite being conserved, M _c models are rarely putative to be functional in MMS scoring as anticipated. In regard to this, R(M _t ) scores are found to be the rarest with comparison to the random models M _r , which is indicated clearly by the 0.000 p-values and reasonably high z-scores. Each random model M _r was composed of one randomly selected K from each sequence.

Table 1 Conservation and rareness characterization of functional motifs

Localized-MISCORE

Transcription proteins rarely contact a single nucleotide without interacting with the adjacent bases in the binding process. Hence, the positions with a higher binding energy given by IC (and also a lower binding energy) are usually clustered as local information blocks in the PFM model of functional motifs [30]. Position-specific similarity metrics assign an equal weight to every position in the model and ignore the variability among the local blocks appearing in the motif PFMs. Since, a motif PFM can be regarded as a descriptor of its binding preferences, the underlaying nucleotide blocks are believed to carry useful information that constitutes the overall characterization of the motif. Based on this understanding, we aim to decompose a motif PFM into a set of local blocks and assign a weight to each block according to its potential of being functional.

MISCORE is then extended to a localized-MISCORE, denoted by r _l (K, M _S ), that can be written as,

r_l\left(K,M_S\right)={\displaystyle \sum _{j=1}^{k-w+1}}{g}_j\left(\frac{d\left({\beta }_j\left(K\right),{\beta }_j\left(M_S\right)\right)}{d\left({\beta }_j\left(K\right),{\beta }_j\left(M_{ref}\right)\right)}\right),

(10)

where β _j (K), β _j (M _S ) and β _j (M _ref ) are the j^th local block in the K, the M _S and the background model M _ref , respectively. A w-length local block β _j (.) can be produced by shifting a small matrix window β_[4×w]such that (2 ≤ w <k) in the K, the M _S and the M _ref so that, k - w + 1 number of blocks can be produced.

The weight g _j for the j^th block in M _S (i.e., β _j (M _S )) can be assigned as,

g_j=\frac{G\left({\beta }_j\left(M_S\right)\right)}{{\sum }_{q=1}^{k-w+1}G\left({\beta }_q\left(M_S\right)\right)},

(11)

where G(β _j (M _S )) is a modified Gini purity index (a complement of the Gini impurity index) that can be evaluated by,

G\left({\beta }_j\left(M_S\right)\right)=\frac{1}{w}{\displaystyle \sum _{i=j}^{j+w-1}}{\displaystyle \sum _{\forall b_i\in \chi }}{\left(\frac{f\left(b_i,i\right)}{p\left(b_i\right)}\right)}^2,

(12)

where p(b _i ) is a background frequency of the base b _i . Inspired by IC, G(β _j (M _S )) can characterize the conservation and the rareness properties of a block. Then, a localized-MMS with notation R _l (S), for evaluating the merit of a set of k-mers S as a potential motif, can be given by,

R_l\left(S\right)=\frac{1}{\left|S\right|}{\displaystyle \sum _{\forall K\in S}}{r}_l\left(K,M_S\right),

(13)

where r _l (K, M _S ) is the localized-MISCORE given by Eq (10).

Note that the localized-MMS aims to improve the discrimination power for weak motifs, while it performs closely to the MMS for the strong motifs.

Results and discussion

In this section, we evaluate the separability and the recognizability performances of MISCORE with comparison to IC and MAP score. The latter portion of the recognizability analysis describes how our MISCORE can perform motif-to-motif similarity computation and incorporate pk models in recognizing functional motifs.

Separability

It is interesting to observe the performance of MISCORE, IC and MAP score in terms of separating functional motifs from the random ones. Hence, a separability performance evaluation on these modeling metrics are conducted, where the separability is considered as a metric to measure the discriminative score-gaps (normalized) between a functional motif model and a large collection of random non-functional ones.

Separability metric

Sep(*, *) score compares two metrics to learn which one has stronger discriminative power to distinguish a true motif from the random models. Given two metrics A and B, a true motif S _t and a large collection of random models \left(S_{r_q},forq=1,2,3,\dots ,N\right), Sep(A, B) can be defined by

Sep\left(A,B\right)=E\left\{1-\frac{{\gamma }_A\left[A\left(S_t\right)-A\left(S_{r_q}\right)\right]}{{\gamma }_B\left[B\left(S_t\right)-B\left(S_{r_q}\right)\right]}\right\},

(14)

where E{*} represents the mathematical expectation, γ _A = [A _max - A _min ]^-1, γ _B = [B _max - B _min ]^-1, and \left[A\left(S_t\right)-A\left(S_{r_q}\right)\right] is the score-gap produced by metric A for S _t and S_{r_q}, \left[B\left(S_t\right)-B\left(S_{r_q}\right)\right] reads similarly for the metric B. A _max (A _min ) and B _max (B _min ) are the metric-specific maximum (minimum), i.e., the best (worst) possible scores, that perform a normalization. Sep(A, B) > 0 score interprets that the metric B outperforms the metric A, and Sep(A, B) < 0 score indicates the opposite case, while Sep(A, B) = 0 score indicates an equal separability performance by the two metrics.

For each dataset, firstly a true motif S _t is generated by carefully aligning all known binding sites using CLUSTAL W [31]. Then, N = 5000 random models are generated by collecting random k-mers from the dataset and by carefully avoiding overlap with the true binding sites subject to \left|S_{r_q}\right|=\left|S_t\right|. The metric bounds, i.e., the best and the worst possible scores, for score normalization is required in Eq (14). The best-possible score (upper bound) of a metric can be obtained by ensuring the maximum quantification of the motif characteristics. To find the upper bound of a metric, we assume that there exist a hypothetical set of k-mers S^* that can ensure the best-possible score of a metric. With an assumption of a perfect conservation between the motif instances, i.e., \delta \left(K_a^{*},K_b^{*}\right)=0, \forall K_{a,b}^{*}\in S^{*}, where δ(*, *) is a similarity quantification, the upper bound for the metrics can be deduced using their respective equation. However, the lower bound (i.e., the worst-possible score) of the metrics are difficult to be computed since the conservation characteristic of a given motif can not be completely eliminated in any situation. Having no viable solution to compute this, the lower-bound of these metrics are practically approximated by the worst score produced by the metrics over a large collection of random models.

Separability results

The datasets used in this paper are split into three groups based on their origins. The first data group (denoted as dg₁) contains 8 datasets that are composed of 200bp promoters that contain the known binding sites (functional motifs) associated with the following TFs: ERE, MEF2, SRF, CREB, E2F, MYOD, TBP and CRP. The whole datasets were collected from [32], and each dataset contains a varying number of sequences and a verified motif with known location of the binding sites. The second group (dg₂) contains 20 mixed datasets (real and artificial) with 500bp ~ 2000bp sequences that were collected from [10]. The third group (dg₃) contains 5 datasets that are composed of 500bp promoters with known binding sites associated with the following TFs: CREB, SRF, TBP, MEF2 and MYOD. The 500bp promoters were collected from the Annotated regulatory Binding Sites (ABS, v1.0) database [33]. Details on these 33 datasets are presented in Table 2.

Table 2 Description of the used 33 datasets

First of all, Sep(R, R _l ) scores are computed to evaluate the improvement of the localized version. Several criteria for the local block-length (w) selection have been examined; and the Sep(R, R _l ) scores are presented in Table 3, showing that the localized version is likely to perform favorably with a smaller w, e.g., w = round(k/3), since Sep(R, R _l ) > 0 holds for most of the datasets. As w becomes larger and gets closer to k, the Sep(R, R _l ) scores tend to be zero, which makes sense in logic.

Table 3 Sep(R, R _l ) score comparison for different local block length w in R _l

A separability comparison among R, R _l , IC and MAP score is then conducted on the 33 datasets. The results are presented in Table 4, showing that MISCORE can achieve a comparable separability performance to IC and a remarkably improved performance than MAP score, which is indicated by the average Sep(*, *) scores on the three data groups, that is, [Sep(IC, R), Sep(IC, R _l ), Sep(MAP, R), Sep(MAP, R _l )]= [-0.144, 0.016, 0.273, 0.374]. In our experiments, MAP score is computed using a 3rd-order Markov chain model. A higher order Markov chain model may improve the separability performance for MAP score, however, the computational cost would be much higher in such a case.

Table 4 Sep(*, *) score comparison among R, R _l , IC and MAP score

Recognizability

It is often observed that after evaluating a set of candidate motifs returned by a discovery tool, the top ranked candidates are not necessarily functional. The ineffectiveness of the motif evaluation metric used can be one of the reasons behind this. Therefore, we have conducted a recognizability performance comparison among these metrics.

Recognizability refers to how well a metric can recognize the best candidate motif from a set of putative candidates in terms of ranking, where the best candidate motif is expected to be top ranked. To conduct this evaluation, we need to have a set of putative candidate motifs generated by some motif discovery tools on each dataset. In this study, we employed MEME [34] to generate a set of putative motifs for each dataset. Then, the best candidate motif is identified by the F-measure [35]: F = 2PR/(P + R), where P = TP/(TP + FP), R = TP/(TP + FN), where TP, FP and FN are the number of true positive, false positive and the false negative predictions, respectively. TP refers to the number of the true binding sites overlapped by at least one predicted site. In this study, we considered a true positive count if a true binding site is overlapped by a predicted site with at least 25% of the length of the true site. FP is the number of the predicted sites that do not have more than 25% overlap with any true binding sites; and FN is the number of the true binding sites that are not overlapped by any predicted sites with at least 25% of the length of the binding site.

These candidate motifs for each dataset are then scored by IC, MAP score, R, and R _l respectively, and ranked according to their scores. The assigned rank of the best motif is recorded for each dataset in order to find that which metric can assign a comparatively higher rank to the best motif. In order to evaluate the ranking order, the following criterion is adopted to compute a mean rank (μ) score [36]:

\mu =\frac{Q\left(Q+1\right)}{2{\sum }_{i=1}^{Q}rank\left(M_i\right)},

(15)

where Q is the number of the relevant items whose rank orders are to be considered. In our case, only the best motif's rank is considered, hence Q = 1 and Eq (15) becomes μ = 1/rank(best motif).

An average μ score over 10 runs with each metric on each dataset is recorded using a set of candidate motifs produced by MEME during each run. The results are presented in Table 5, which also includes a data group-wise E{μ} score as result summary showing that both R and R _l offer a considerably better recognizability than MAP score, while IC is likely to perform the best recognizability performance. However, we observed that a 10-run average μ score computed using dg₁ and dg₂ (i.e., 28/33 datasets) indicates that both R and R _l can outperform IC and MAP score.

Table 5 Recognizability scores for the best candidate motifs

Recognizability on degenerated motifs

Weak motif characterization and recognition is challenging to all evaluation metrics. Therefore, in order to observe how the considered metrics perform in recognizing degenerated motifs, we first split the 33 datasets into two categories, i.e., strong and weak motif classes, based on the average positional conservation of the motif PFMs, which is defined as apc\left(S_t\right)=\frac{1}{k}{\sum }_{i=1}^k{\displaystyle \underset{b_i}{\text{max}}}\left\{f\left(b_i,i\right)\right\}, b _i ∈ {A, C, G, T}.

Table 6 reports the average recognizability scores of these metrics on the datasets. The results show that MISCORE can noticeably outperform MAP score and perform comparably to IC in recognizing weak motifs. However, IC outperforms our MISCORE and MAP score in recognizing strong motifs.

Table 6 Strong/weak motif class-wise average recognizability scores

Motif recognition using priori-known models

If there exists priori known (pk) estimation of the target motif profile during the search in the query sequences, then the motif discovery algorithms can greatly benefit by utilizing such a priori knowledge in finding motifs that have similar characteristics to the pk model. Often a priori estimation of a target motif model can be obtained from the public databases e.g., [37–39], or by collecting a set of binding sites from the sequences that are known to be co-regulated by the target TF [29]. These pk models can only be the estimation of the target motifs in the search, since: (i) the known binding sites in the public databases are usually incomplete, which may cause the pk profiles to have an incomplete representation that may not be able to reliably discriminate a true motif from a false one [40], and (ii) due to the sequence dissimilarity between the query sequences and the sequences that are known to be co-regulated by the target TF.

One plausible use of the pk models is their involvement in the process of motif evaluation, where the putative motifs will be recognized by referring to the pk models. The ranking of the candidate motifs then becomes a motif-to-motif similarity quantification between the putative and the pk models.

MAP score is unable to evaluate the motif-to-motif similarity. IC, on the other hand, is not originally meant for motif-to-motif similarity computation. However, it has been extended as the average log likelihood ratio (ALLR) [41] for this task. Several other metrics can perform motif-to-motif similarity quantification, e.g., Pearson correlation coefficient (PCC) [42], Kullback-Leibler divergence (KLD) [43–45], Euclidean distance (ED) [46] and Sandelin-Wasserman (SW) metric [47]. But, these metrics can only compute a motif-to-motif similarity without considering motif characteristics.

Motivated by the above facts, MISCORE framework is examined to perform the motif-to-motif similarity while taking account of the motif characterization. Let a candidate motif S be ranked by using a pk model M _pk . Then, MISCORE becomes

r_{pk}\left(K,M_{pk}\right)=\frac{d\left(K,M_{pk}\right)}{d\left(K,M_{ref}\right)+c\left(K\right)}.

(16)

The MMS score (R) given in Eq (9) then can be written as,

R_{pk}\left(S\right)=\frac{1}{\left|S\right|}{\displaystyle \sum _{\forall K\in S}}{r}_{pk}\left(K,M_{pk}\right).

(17)

Note that R _pk and r _pk , characterizing motif signals with assistance of pk models, can be regarded as the supervised counterparts of R and r, respectively. localized-MISOCRE can be expressed to accommodate the pk models in a similar manner. Similarly, MISCORE can be employed to compute the motif-to-motif similarity in order to group similar candidate motifs in the relevant applications.

For simplicity, we demonstrate that MISCORE with the use of pk models can help in recognizing putative motifs, and performs favorably against other metrics. To do this, we first generated a pk model for each dataset by extracting the non-redundant known binding sites associated with CREB, E2F, MEF2 and SRF transcription factors from JASPAR [37]; ERE, MYOD and TBP from TRANSFAC (public v7.0) [38]; and CRP from RegulonDB [39] databases. After alignment, the pk models are generated for the datasets in dg₁ and dg₃ since they share common transcription factors. For the 20 datasets in dg₂, we applied a multiple sequence alignment tool GLAM [48] to align the binding sites of each dataset. Then, the longest conserved block from the alignment is extracted to form a pk model for each dataset.

The data group-wise average recognizability scores obtained by the metrics over 10 runs are presented in Table 7, showing that MISCORE others a promising performance with comparison to other metrics in terms of recognizing the best candidate motifs using the pk models.

Table 7 Recognizability scores for the best candidate motifs using pk models

Background rareness and over-representation

Another key concept in computational motif discovery is over-representation [13–15, 49]. It looks for motifs that have significant occurrences in the query sequences (input promoters) than the background sequences through some statistical quantification [13, 16]. The functionality of this site multiplicity, i.e., 'the shadow appearances of the binding sites', in the regulatory regions could constitute a mechanism for lateral diffusion of the transcription factors along the sequences, and/or the shadow sites might be the fossils from the process of binding site turnover [16, 50]. Even though the biological reasons behind this site multiplicity are yet to be fully understood [16], it is often considered as a useful motif characteristic and well recognized in the working field.

It is interesting to analyze the correlation between a functional motif's background rareness and over-representation, although both can partially characterize the functional motifs. This section tries to make a sensible link between these two key concepts.

Correlation between background rareness and over-representation

Our aim is to show how MISCORE can be used to characterize a motif's background rareness through its over-representation feature using foreground (i.e., promoters) and background information. We first define a constrained frequency (cf) measure in order to compute an occurrence score of a given motif using MISCORE. Given a set S _all to contain all possible k-mers from a set of sequences (either foreground or background) and a motif S with a PFM model M _S , cf is defined as:

cf\left(M_S,S_{all}\right)=\frac{\left|T\right|}{\left|S_{all}\right|},whereT=\left\{\forall K\in S_{all}:r\left(K,M_S\right)\le \theta \right\},

(18)

where | * | represents the set cardinality, r(K, M _S ) is the MISCORE given in Eq (6) and θ is a cut-off threshold that can be defined as θ = R(S) + std(S)λ, where std represents the standard deviation operator, λ is a threshold regulator and R(*) is the MMS given in Eq (9).

Regulatory regions often contain more frequent occurrences of a functional motif compare to the sequence-backgrounds, due to the mutational constraints in the foreground compared to the backgrounds. Hence, a true motif is expected to produce a larger cf in the promoter regions (foreground) than the backgrounds for a given similarity threshold. Therefore, the MISCORE-based over-representation score ORS _r for a motif S can be given using Eq (18) as,

OR{S}_r\left(M_S\right)=\frac{cf\left(M_S,S_{bg}\right)}{cf\left(M_S,S_{fg}\right)},

(19)

where S _bg and S _fg are the sets of all k-mers produced by window shifting in the background and in the foreground regions, respectively.

The condition ORS _r (M _S ) < 1 indicates that M _S has a higher frequency in the foreground than the background for a given threshold, which implies that there are comparatively less occurrences of that motif in the background (i.e., background rareness) than the foreground. Hence, the background rareness of a motif can be characterized through its over-representation feature, that can be statistically quantified.

Demonstration

We collected the background sequences for CREB, SRF, TBP, MEF2 and MYOD datasets from public databases (e.g., http://www.ncbi.nlm.nih.gov and http://www.ebi.ac.uk) as the respective sequence backgrounds. The respective 200bp and 500bp promoter regions are then taken as the sequence-foregrounds for each TF. The ORS _r (M _t ) scores for different thresholds are computed for each TF and presented in Table 8, showing that the background rareness can be characterized through the over-representation of the functional motifs since ORS _r (M _t ) < 1 holds for all cases. It also shows that, as the promoter region grows in length from 200bp to 500bp, the ORS _r scores tend to increase for the functional motifs, as anticipated.

Table 8 ORS _r (M _t ) scores with several threshold regulators

In order to conduct a statistical evaluation, the ORS _r (M _t ) score of the true motif of each dataset is evaluated using two large sets of (i) conserved (M _c ) and (ii) random models (M _r ). Each random model M _r and conserved model M _c is generated according to the criteria that have been described earlier. It has been observed that the following holds for all cases with a given similarity threshold, that is,

(20)

This implies that ORS _r (M _t ) scores are relatively rare in respect to E{ORS _r (M _c )} with a given similarity threshold. Since the M _c models, despite being conserved, have less chance of being over-represented than a true model M _t . In addition to this, ORS _r (M _t ) scores are found to be the rarest with comparison to the random models M _r . In other words, E{ORS _r (M _r )} >ORS _r (M _t ) implies that, the random models have a comparatively larger background-to-foreground occurrence ratio (see Eq (19)) than the functional motifs. This characterizes the background rareness property of a functional motif through its over-representation property. Figure 1 demonstrates the correlation between the background rareness and the over-representation for 10 datasets.

Figure 1

Correlation between the over-representation and the background rareness. ORS _r scores for the functional models M _t , the random models M_{r_q}, and the conserved models M_{c_q} for q = 1, 2, 3, . . ., 1000 are plotted for each dataset with 200bp and 500bp promoters in the left and in the right column, respectively. Threshold θ = R(M _t ) + std(M _t )λ, λ = 0.0 is used. The figure depicts a rareness interpretable visualization through the statistical over-representation property of the functional motifs by showing that, the ORS _r (M _r ) scores are found distant from the ORS _r (M _t ) scores for all cases which implies that the random models have close to zero chance of being over-represented with comparison to the true models. In addition to this, the ORS _r (M _t ) scores are found to be mostly rare with comparison to the ORS _r (M _c ) scores, i.e., these non-functional conserved models have a rare chance of having better over-representation scores than the true models, for most of the datasets.

Conclusions

This paper contributes a mismatch-based fast computational tool for modeling DNA regulatory motifs. It is free from any assumption on the model dependency, and it escapes from the use of background modeling using Markov chain models. Simultaneously, it embeds the compositional complexity in modeling the motif signals. Our proposed MISCORE can be used as a metric to measure the similarity between k-mers and a motif model, also it can be employed to compute the motif-to-motif similarity.

The experimental results on 33 datasets indicate that MISCORE performs favorably with comparison to the well-known IC and MAP score in terms of the separability and the recognizability. These results also show that MISOCRE is functionally effective in recognizing degenerated motifs, and it can embed the pk models to perform candidate motif ranking.

MISCORE has good potential to be employed as a similarity metric in rule-based or clustering-based motif discovery algorithms, it can also be employed as a numerical feature in machine learning approaches for finding motifs. Furthermore, MISCORE-based Motif Score (MMS) can be employed as a fitness function in evolutionary computation approaches for motif discovery, and for candidate motif ranking in computational motif discovery tools.