Skip to main content

Table 2 Comparison of methods for direct causal effects for simulated data with moderate variability ( σ = 0.1 )

From: Joint estimation of causal effects from observational and intervention gene expression data

Setting

Criterion

MCMC-Mallows

Pinna

IDA (opt)

IDA (pes)

 

AUROC

0.79 (0.041)

0.773 (0.064)

0.651 (0.083)

 

AUPRC

0.633 (0.061)

0.577 (0.085)

0.472 (0.102)

Observation only

Spearman

0.474 (0.094)

0.484 (0.122)

0.246 (0.17)

 

MSE

0.059 (0.006)

0.193 (0.057)

0.138 (0.035)

 

AUROC

0.951 (0.03)

0.842 (0.051)

0.746 (0.06)

0.678 (0.073)

 

AUPRC

0.841 (0.051)

0.688 (0.084)

0.5 (0.081)

0.465 (0.091)

Mixed

Spearman

0.668 (0.055)

0.534 (0.097)

0.409 (0.132)

0.306 (0.181)

 

MSE

0.048 (0.015)

0.107 (0.01)

0.35 (0.131)

0.211 (0.067)

 

AUROC

0.871 (0.064)

0.784 (0.018)

0.749 (0.068)

0.655 (0.094)

 

AUPRC

0.721 (0.089)

0.663 (0.054)

0.532 (0.088)

0.459 (0.115)

Partial KO

Spearman

0.574 (0.106)

0.606 (0.04)

0.437 (0.104)

0.272 (0.185)

 

MSE

0.055 (0.015)

0.088 (0.007)

0.228 (0.068)

0.161 (0.052)

 

AUROC

0.962 (0.017)

0.839 (0.032)

0.746 (0.06)

0.678 (0.073)

 

AUPRC

0.864 (0.034)

0.69 (0.046)

0.5 (0.081)

0.465 (0.091)

Multiple KO

Spearman

0.683 (0.033)

0.614 (0.051)

0.409 (0.132)

0.306 (0.181)

 

MSE

0.038 (0.009)

0.108 (0.008)

0.35 (0.131)

0.211 (0.067)

 

AUROC

0.94 (0.045)

0.561 (0.189)

0.576 (0.156)

0.555 (0.133)

Multiple KO

AUPRC

0.483 (0.085)

0.288 (0.107)

0.279 (0.078)

0.276 (0.073)

(3 hidden genes)

Spearman

0.633 (0.106)

0.048 (0.37)

0.07 (0.311)

0.064 (0.305)

 

MSE

0.069 (0.048)

0.149 (0.032)

0.454 (0.207)

0.296 (0.109)

  1. Several intervention designs were simulated: 1) 20 observational (wild-type) replicates with no interventions, 2) mixed setting with 10 wild-types and one knock-out per gene, 3) partial knock-out design with 15 wild-types and one knock-out for five genes {N1, N4, N6, N7, N9}, 4) multiple knock-out design with 10 wild types, one knock-out per gene and five double knock-outs: {N1, N5}, {N1, N6}, {N4, N7}, {N6, N9}, and {N7, N10}, and 5) a multiple knock-out design as in the previous setting, with three hidden variables. Results were averaged over 100 simulations (standard deviations in parentheses): area under the ROC curve (AUROC), area under the precision-recall curve (AUPRC), Spearman correlation between true and estimated direct causal effects, and mean squared error (MSE) of estimated direct causal effects.