Apache Spark Streaming, Kafka and HarmonicIO: A Performance Benchmark and Architecture Comparison for Enterprise and Scientific Computing

Technological advances in automated microscopy now allow rapid acquisition of many images without human intervention, images that can be used for large-scale screens. The main challenge in such screens is the conversion of the raw images into interpretable information and hence discoveries. This post-acquisition component of image-based screens requires computational steps to identify cells, choose the cells of interest, assess their phenotype, and identify statistically significant 'hits'. Designing such an analysis pipeline requires careful consideration of the necessary hardware and software components, image analysis, statistical analysis and data presentation tools. Given the increasing availability of such hardware and software, these types of experiments have come within the reach of individual labs, heralding many interesting new ways of acquiring biological knowledge. Show more