Convolutional Neural Networks for Real-Time Localization and Classification in Feedback Digital Microscopy

We present an adapted single-shot convolutional neural network (YOLOv2) for the real-time localization and classification of particles in optical microscopy. As compared to previous works, we focus on the real-time detection capabilities of the system to allow for manipulation of microscopic objects in large heterogeneous ensembles with the help of feedback control. The network is capable of localizing and classifying several hundreds of microscopic objects even at very low signal-to-noise ratios for images as large as 416x416 pixels with an inference time of about 10 ms. We demonstrate the real-time detection performance by manipulating active particles propelled by laser-induced self-thermophoresis. In order to make our framework readily available for others, we provide all scripts and source code. The network is implemented in Python/Keras using the TensorFlow backend. A C library supporting GPUs is provided for the real-time inference.