Artificial Intelligence and Confocal Microscopy – What You Need to Know

FAQs about AI software AiviaMotion on the STELLARIS confocal platform

AiviaMotion: Truly simultaneous multicolor imaging of live cells (U2OS) in 3D How_Artificial_Intelligence_Enhances_Confocal_Imaging_teaser.jpg

This list of frequently asked questions provides “hands-on” answers and is a supplement to the introductory article about AiviaMotion How Artificial Intelligence Enhances Confocal Imaging.

What is the benefit of AiviaMotion?

AiviaMotion boosts the denoising method Dynamic Signal Enhancement (DSE) with artificial intelligence (AI). Thus, AiviaMotion even further strengthens the live-cell imaging capabilities of the STELLARIS platform requiring less trade-offs between cell viability, acquisition speed, image quality, and dimensionality. 

How do Dynamic Signal Enhancement (DSE) and AiviaMotion work?

DSE uses a rolling average calculation as its core algorithm. The ratio of a rolling window is that noise between adjacent imaging frames is random while biological signals are preserved. Accordingly, by averaging over several adjacent imaging frames with a rolling window, the relevant biological signal is accentuated while the unwanted noise is “averaged out”. In more expert terms, the DSE increases the signal-to-noise ratio (SNR) and, thus, leads to better image quality.

Video 1: Scheme to depict DSE calculation for vesicles (large red circles) using five raw data frames (upper images) for the rolling window (bracket). Noise (small red dots) gets averaged out while the biological signal (large red circles) is accentuated in the output frames (lower images). Note that, for simplicity, the weight parameter of the DSE is not shown. The weight parameter controls the contribution of raw data frames for rolling window calculation (i.e., central frames are more weighted than peripheral frames) and, thus, helps to optimize the preservation of temporal dynamics.

With AiviaMotion, we now introduce a new option for the DSE which uses AI to interpolate so-called “AiviaMotion frames” between acquired “raw data frames”. Using the additional information from the AiviaMotion frames for rolling average calculation, DSE can now reach a higher SNR and better temporal dynamics at the same time.

Video 2: Scheme to depict DSE calculation with activated AiviaMotion option. As both acquired raw data frames (upper images with white outline) and AiviaMotion frames (upper images with blue outline) are used for the rolling window calculation, the temporal dynamics are better preserved with an activated AiviaMotion option (indicated by well-defined red circles in the lower images). In contrast to the previous example (of the “vanilla” DSE), now an average over only three acquired raw data frames could be used since with the additional two AiviaMotion frames still a total of five frames is employed for the rolling window calculation. On top, because the AiviaMotion frames by themselves also filter noise, not only the temporal dynamics are better preserved, but also the SNR is higher than with the vanilla DSE.

What is the underlying technology of AiviaMotion?

AiviaMotion is based on a specialized type of deep neural network architecture referred to as convolutional neural network (CNN). The very implementation of AiviaMotion is based on the so-called “CAIN” network (Channel Attention Is All You Need for Video Frame Interpolation, Choi et al., 2020).

For which kind of experiments does AiviaMotion work?

  • What kind of movements?

Frame interpolation with AiviaMotion works best for small, linear displacements of objects.

  • What kind of specimens?

AiviaMotion was trained on a large body of training data. We used more than 6,000 image triplets from more than 100 distinct microscopy time-lapses to fine-tune the underlying deep neural network on domain-specific data (ranging from cytoskeleton in cell culture to membrane staining in zebrafish retinae). Accordingly, we presume AiviaMotion to generalize well on a wide range of specimen and applications.

  • What kind of imaging data?

AiviaMotion works on imaging data involving a time dimension, e.g., on xyt and xyczt scans. Furthermore, AiviaMotion works for both FOV (Field of View) and resonant scanners, as well as for different modalities of the STELLARIS platform, e.g., confocal, DIVE and DLS. For the TauSense toolbox, AiviaMotion works on all intensity data, i.e., on images for which life-time information was used for the acquisition but is not part of the resulting image (e.g., it works with TauSeparation, but not with TauContrast).

Can DSE/AiviaMotion be combined with LIGHTNING?

DSE (including AiviaMotion) can be combined with LIGHTNING. This combination is especially rewarding when used in conjunction with the resonant scanner for fast and gentle live-cell imaging with multiple fluorophores. Furthermore, the processing of DSE and LIGHTNING can be chained, i.e., with a single “start” command the calculation of both image processing steps can be initiated.

How many frames are interpolated with different AiviaMotion levels?

AiviaMotion has three different levels: “1”, “2”, and “3”. The AiviaMotion level controls how many iterations of AiviaMotion are applied on the image series, e.g., in case of AiviaMotion level 1, a single iteration of AiviaMotion is applied (and so on).

However, the AiviaMotion level does not directly indicate the number of frames which get inserted between individual acquired raw data frames. In the single iteration for AiviaMotion level 1, only a single image is inserted between individual acquired raw data frames. For AiviaMotion level 2, in the first iteration a single AiviaMotion frame is inserted between two adjacent raw data frames, and in the second iteration two additional AviaMotion frames are added (one between the “left” raw data frame and the AiviaMotion frame from the first iteration, and the other one between the AiviaMotion frame from the first iteration and the “right” raw data frame). So, in total there are three AiviaMotion frames inserted with AiviaMotion level 2. Finally, using the exact same reasoning from above, but with three iterations, AiviaMotion level 3 adds a total of seven frames in between two adjacent raw data frames. In more general terms, the number of frames inserted between two adjacent raw data frames could be calculated by 2i-1 where the parameter i is the number of AiviaMotion iterations.

Along those lines, it is frequently desired to also know the total number of frames used for a given rolling average window size and AiviaMotion level. Thus, here is an example for using a DSE setting with five raw data frames and the resulting number of total frames used for rolling average calculation given different AiviaMotion levels.

AiviaMotion Level (i)

Frames Raw (n)

Frames AiviaMotion

Frames Total (t)

Generalization (Frames Total)

0

5

0 * 4 = 0

5

n + (20 – 1) * (n – 1) = n

1

5

1 * 4 = 4

9

n + (21 – 1) * (n – 1) = 2n - 1

2

5

  3 * 4 = 12

17

n + (22 – 1) * (n – 1) = 4n - 3

3

5

  7 * 4 = 28

33

n + (23 – 1) * (n – 1) = 8n - 7

Table: Total number of frames (t) used for the DSE rolling window calculation as a function of the given AiviaMotion levels (i) and raw frames (n); t = n + (2i – 1) * (n – 1).

The ratio of using higher AiviaMotion levels is that the SNR can potentially be further improved by averaging over more AiviaMotion-interpolated frames. This strategy is especially applicable when the size of the rolling window cannot be increased (i.e., using more raw images for the calculation is not feasible), because the best temporal dynamics is of key importance for the experiment.

Does AiviaMotion increase the frame rate?

No, AiviaMotion does not increase the number of output frames, i.e., the number of output frames from the Dynamic Signal Enhancement (DSE) is unchanged – irrespective whether the AiviaMotion option is activated or not. However, AiviaMotion does transiently increase the number of frames, i.e., the number of frames used as input for calculating the rolling window of the DSE is increased. Since, however, the rolling window is moved in a way that the stride considers the AiviaMotion level (i.e., it is proportional to how many frames were inserted between adjacent raw data frames), the number of output frames is independent of the selected AiviaMotion level and thus invariant. In summary, AiviaMotion helps to better preserve the temporal dynamics of the specimen by enabling to shrink the size of the rolling window (see also FAQ question 2), but without increasing the frame rate.

Can AiviaMotion also interpolate the z-dimension?

For the moment, AiviaMotion can perform frame interpolation for the t-dimension. As AiviaMotion only adds a single parameter to the Dynamic Signal Enhancement (number of t-interpolation iterations), AiviaMotion is easy to use and can be activated by a single click.

Is it scientifically OK to denoise data with Dynamic Signal Enhancement (DSE) and AiviaMotion?

DSE uses a rolling average calculation as its core algorithm. The use of a rolling average is widely established as method for removing noise, with application fields ranging from imaging data to stock charts. The AiviaMotion option adds a deep neural network to DSE processing. In contrast to many other deep learning algorithms for denoising, AiviaMotion does not operate on single images and the spatial domain of the data, but rather on multiple input images and the temporal domain. Accordingly, AiviaMotion excels on “temporal consistency” of the processed data. Furthermore, since AiviaMotion is “only” adding some frames for the DSE rolling window calculation, there is always a blend of “ground-truth” raw-data frames and AiviaMotion frames used for denoising. Finally, the user has always the option to compare DSE results with and without AiviaMotion. Thus, especially for quantitative analysis (e.g., measuring fluorescence intensity levels), the user has easily the possibility to confirm the correct working of AiviaMotion.

Can the AiviaMotion algorithm be trained?

At the moment, we do not provide you with the opportunity to train the AiviaMotion interpolation algorithm. However, we ship AiviaMotion with a pre-trained deep learning model which first had been trained on an extensive publicly available video dataset and then fine-tuned via transfer learning on a domain-specific microscopy dataset. Accordingly, we presume AiviaMotion to generalize well on a wide range of specimen and applications.

Does the weight parameter of Dynamic Signal Enhancement (DSE) have any impact on AiviaMotion frame interpolation?

For DSE, not all frames are counted equally. There is a “weight” parameter used in DSE which determines how much frames close to the center of the rolling window, as well as how much more remote frames, contribute to this calculation.

Yet, the weight parameter is not considered when generating the AiviaMotion frames via frame interpolation, because, in any case, only the two adjacent frames are used from the AiviaMotion algorithm. When applying the rolling window calculation on both raw data frames and AiviaMotion frames, the weight parameter is considered as described above for the rolling window calculation. In summary, the weight parameter is not used for the AiviaMotion frame interpolation, but used later during the rolling average calculation.

How does processing time scale with the number of AiviaMotion iterations?

The time necessary for each AiviaMotion round is not uniform. As every AiviaMotion iteration practically doubles the number of input images for the next iteration, the following AiviaMotion iteration takes twice as long as the previous round. Accordingly, while the second iteration would take twice the first interval, the third iteration would take quadruple the first interval. When combined with LIGHTNING, the processing time of both Dynamic Signal Enhancement (DSE including AiviaMotion) and LIGHTNING simply adds up, because they are executed sequentially. Moreover, since the AiviaMotion frames are only transient (i.e., the number of output frames from the DSE is unchanged – irrespective of the application of the AiviaMotion level), the processing time of LIGHTNING is independent from the employed AiviaMotion level.

Related Pages

Interested to know more?

Talk to our experts. We are happy to answer all your questions and concerns.

Contact Us

Do you prefer personal consulting?

  • Leica Microsystems Inc.
    1700 Leider Lane
    Buffalo Grove, IL 60089 United States
    Office Phone : +1 800 248 0123
    Service Phone : 1 800 248 0223
    Fax : +1 847-236-3009

You will find a more detailed list of local contacts here.