Netbenchmark

In the last decade, several methods have tackled the challenge of reconstructing gene regulatory networks from gene expression data. Several papers have compared and evaluated the different network inference methods relying on a simulated data.

This is new comparison that assesses different methods in a high-heterogeneity data scenario which could reveal the specialization of methods for the different network types and data.

This package characterizes the performance, data requirements and noise robustness of gene regulatory networks inference methods.

This package allows replication this comparison between the different networks inference algorithms with only one line of code.

Toy example for main benchmark:

    library(netbenchmark)
    top20.aupr <- netbenchmark(methods="all",datasources.names = "Toy",
                               local.noise=20,global.noise=10,
                               noiseType=c("normal","lognormal"),
                               datasets.num = 2,experiments = 40,
                               seed=1422976420)

## Warning in netbenchmark(methods = "all", datasources.names = "Toy", local.noise = 20, : The specified number of experiments and 
##                     datasets is bigger than the orginal number of experiments 
##                     in the datasource: toy, 
##                     sampling with replacement will be used

## datasource: toy 
## 
##  dataset: 1 
## 
## aracne.wrap 
## 
## c3net.wrap 
## 
## clr.wrap 
## 
## GeneNet.wrap

## Estimate (local) false discovery rates (partial correlations):

## Genie3.wrap 
## 
## mrnet.wrap 
## 
## mutrank.wrap 
## 
## mrnetb.wrap 
## 
## pcit.wrap 
## 
## zscore.wrap 
## 
##  dataset: 2 
## 
## aracne.wrap 
## 
## c3net.wrap 
## 
## clr.wrap 
## 
## GeneNet.wrap

## Estimate (local) false discovery rates (partial correlations):

## Genie3.wrap 
## 
## mrnet.wrap 
## 
## mutrank.wrap 
## 
## mrnetb.wrap 
## 
## pcit.wrap 
## 
## zscore.wrap

The first element of the returned list is the \(AUPR_{20}\):

   print(top20.aupr[[1]])

##   Origin experiments aracne.wrap c3net.wrap  clr.wrap GeneNet.wrap
## 1    toy          48  0.14858187 0.13085487 0.1714301   0.05583115
## 2    toy          35  0.09763924 0.08670886 0.1172456   0.04225142
##   Genie3.wrap mrnet.wrap mutrank.wrap mrnetb.wrap pcit.wrap zscore.wrap
## 1   0.1747381  0.1701856   0.11900207   0.1760170 0.1753417  0.03307668
## 2   0.1427582  0.1197440   0.07485347   0.1216965 0.1425235  0.01477506
##          rand
## 1 0.017541112
## 2 0.005341518

The package provides an easy way to compare new techniques with
state-of-the-art ones and to make new different benchmarks in the future.

First, define the wrapper functions:

    Spearmancor <- function(data){
        cor(data,method="spearman")
    }

    Pearsoncor <- function(data){
        cor(data,method="pearson")
    }

Note that the wrapper function returns a matrix which is the weighted adjacency matrix of the network inferred by the algorithm and that their columns and rows are named.

Evaluate five times these two simple inference methods with syntren300 datasource:

    res <- netbenchmark(datasources.names="syntren300",
        methods=c("Spearmancor","Pearsoncor"))

## datasource: syntren300 
## 
##  dataset: 1 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 2 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 3 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 4 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 5 
## 
## Spearmancor 
## 
## Pearsoncor

    aupr <- res[[1]][,-(1:2)]

Make a boxplot the \(AUPR_{20}\) results:

    boxplot(aupr, main="Syntren300")

Plot the mean Precision-Recall curves:

    PR <- res[[5]][[1]]
    col <- rainbow(3)
    plot(PR$rec[,1],PR$pre[,1],type="l",lwd=3,col=col[1],xlab="Recall",
        ylab="Precision",main="Syntren300",xlim=c(0,1),ylim=c(0,1))
    lines(PR$rec[,2],PR$pre[,2],type="l",lwd=3,col=col[2])
    lines(PR$rec[,3],PR$pre[,3],type="l",lwd=3,col=col[3])
    legend("topright", inset=.05,title="Method",colnames(PR$rec),fill=col)

We can also compare these two simple inference methods with the fast network inference algorithms using syntren300 datasource:

    comp <- netbenchmark(datasources.names="syntren300",
        methods=c("all.fast","Spearmancor","Pearsoncor"))

## datasource: syntren300 
## 
##  dataset: 1 
## 
## aracne.wrap 
## 
## c3net.wrap 
## 
## clr.wrap 
## 
## GeneNet.wrap

## Estimate (local) false discovery rates (partial correlations):

## mrnetb.wrap 
## 
## mutrank.wrap 
## 
## pcit.wrap 
## 
## zscore.wrap 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 2 
## 
## aracne.wrap 
## 
## c3net.wrap 
## 
## clr.wrap 
## 
## GeneNet.wrap

## Estimate (local) false discovery rates (partial correlations):

## mrnetb.wrap 
## 
## mutrank.wrap 
## 
## pcit.wrap 
## 
## zscore.wrap 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 3 
## 
## aracne.wrap 
## 
## c3net.wrap 
## 
## clr.wrap 
## 
## GeneNet.wrap

## Estimate (local) false discovery rates (partial correlations):

## mrnetb.wrap 
## 
## mutrank.wrap 
## 
## pcit.wrap 
## 
## zscore.wrap 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 4 
## 
## aracne.wrap 
## 
## c3net.wrap 
## 
## clr.wrap 
## 
## GeneNet.wrap

## Estimate (local) false discovery rates (partial correlations):

## mrnetb.wrap 
## 
## mutrank.wrap 
## 
## pcit.wrap 
## 
## zscore.wrap 
## 
## Spearmancor 
## 
## Pearsoncor 
## 
##  dataset: 5 
## 
## aracne.wrap 
## 
## c3net.wrap 
## 
## clr.wrap 
## 
## GeneNet.wrap

## Estimate (local) false discovery rates (partial correlations):

## mrnetb.wrap 
## 
## mutrank.wrap 
## 
## pcit.wrap 
## 
## zscore.wrap 
## 
## Spearmancor 
## 
## Pearsoncor

    aupr <- comp[[1]][,-(1:2)]

Make a boxplot the \(AUPR_{20}\) results:

    #make the name look prety
    library("tools")
    colnames(aupr) <- sapply(colnames(aupr),file_path_sans_ext)
    boxplot(aupr, main="Syntren300")