Benchmarking immunoinformatic tools

Antibody repertoires reveal insights into the biology of the adaptive immune system and empower diagnostics and therapeutics.

There are currently multiple tools available for the annotation and preprocessing analysis of antibody sequences. However, a thorough comparison of the performance of these tools has not been investigated. We benchmark the performance of commonly used immunoinformatics tools, i.e., IMGT/HighV-QUEST, IgBLAST, and MiXCR, in terms of annotation output, accuracy and speed using simulated and experimental datasets. Using the same testing conditions, we compare the tools’ ability to annotate sequences to V, D, and J genes. Our benchmarking results show that the usage of different tools result in diverging outputs. In terms of alignment accuracy, MiXCR had the highest average frequency of gene mishits, 0.02 mishit frequency, and IgBLAST the lowest, 0.004 mishit frequency. Reproducibility in the output complementarity determining 3 regions (CDR3 a.a.) ranged from 11 to 28%. Run time for the tools was assessed and MiXCR was the fastest tool for annotation output per unit of time. These results indicate that immunoinformatic analysis greatly depend on the choice of bioinformatics tool. This benchmark analysis provides a basis to support informed decision-making for immuno-informaticians. We are currently extending this project to provide a thorough benchmarking for all existing immunoinformatic tools.