Under the hood of in-silico labeling Interpretation of Image-to-Image transformation models

In-silico labeling, the computational inference of multiple organelle localization patterns from label-free microscopy images, has the potential to revolutionize cell biology by overcoming the inherent technical limitations fluorescence tagging impose upon dynamic imaging of multiple organelles in the same living cell. Although several studies have demonstrated proof-of-concept in-silico labeling, the ability to interpret the in-silico model predictions is necessary for routine practical application.

We introduce Mask Interpreter, a method for extracting the essential information from a label-free image used to predict an organelle localization. Mask Interpreter generates a noise mask image given a label-free image and its matching organelle localization prediction from a given specific in-silico labeling model. The optimized noise mask highlights regions, whereas the label-free image can be degraded without affecting organelle localization prediction. According to our preliminary findings, Mask Interpreter gives a more intuitive interpretation than the current state of the art in explainable AI. The ability to manipulate the organelle spatial context for more accurate interpretability is a key advantage.

Using Mask Interpreter, we revealed the organelle-dependent spatial context learned by the in-silico model. The Nucleolus Granular Component, for example, is localized largely based on image-areas around the organelle, whereas the Nuclear Envelope requires the entire nucleus and its surroundings, The Plasma Membrane on the contrary is localized based on wide image regions that exclude the organelle itself. By providing visual and quantitative explanations of the spatial context utilized by the in-silico model, users will be able to disregard inconsistent in-silico localizations and form hypotheses regarding organelle-organelle interactions.