AiviaMotion boosts the denoising method Dynamic Signal Enhancement (DSE) with artificial intelligence (AI). Thus, AiviaMotion even further strengthens the live-cell imaging capabilities of the STELLARIS platform requiring less trade-offs between cell viability, acquisition speed, image quality, and dimensionality.
DSE uses a rolling average calculation as its core algorithm. The ratio of a rolling window is that noise between adjacent imaging frames is random while biological signals are preserved. Accordingly, by averaging over several adjacent imaging frames with a rolling window, the relevant biological signal is accentuated while the unwanted noise is “averaged out”. In more expert terms, the DSE increases the signal-to-noise ratio (SNR) and, thus, leads to better image quality.
With AiviaMotion, we now introduce a new option for the DSE which uses AI to interpolate so-called “AiviaMotion frames” between acquired “raw data frames”. Using the additional information from the AiviaMotion frames for rolling average calculation, DSE can now reach a higher SNR and better temporal dynamics at the same time.
AiviaMotion is based on a specialized type of deep neural network architecture referred to as convolutional neural network (CNN). The very implementation of AiviaMotion is based on the so-called “CAIN” network (Channel Attention Is All You Need for Video Frame Interpolation, Choi et al., 2020).
- What kind of movements?
Frame interpolation with AiviaMotion works best for small, linear displacements of objects.
- What kind of specimens?
AiviaMotion was trained on a large body of training data. We used more than 6,000 image triplets from more than 100 distinct microscopy time-lapses to fine-tune the underlying deep neural network on domain-specific data (ranging from cytoskeleton in cell culture to membrane staining in zebrafish retinae). Accordingly, we presume AiviaMotion to generalize well on a wide range of specimen and applications.
- What kind of imaging data?
AiviaMotion works on imaging data involving a time dimension, e.g., on xyt and xyczt scans. Furthermore, AiviaMotion works for both FOV (Field of View) and resonant scanners, as well as for different modalities of the STELLARIS platform, e.g., confocal, DIVE and DLS. For the TauSense toolbox, AiviaMotion works on all intensity data, i.e., on images for which life-time information was used for the acquisition but is not part of the resulting image (e.g., it works with TauSeparation, but not with TauContrast).
DSE (including AiviaMotion) can be combined with LIGHTNING. This combination is especially rewarding when used in conjunction with the resonant scanner for fast and gentle live-cell imaging with multiple fluorophores. Furthermore, the processing of DSE and LIGHTNING can be chained, i.e., with a single “start” command the calculation of both image processing steps can be initiated.
AiviaMotion has three different levels: “1”, “2”, and “3”. The AiviaMotion level controls how many iterations of AiviaMotion are applied on the image series, e.g., in case of AiviaMotion level 1, a single iteration of AiviaMotion is applied (and so on).
However, the AiviaMotion level does not directly indicate the number of frames which get inserted between individual acquired raw data frames. In the single iteration for AiviaMotion level 1, only a single image is inserted between individual acquired raw data frames. For AiviaMotion level 2, in the first iteration a single AiviaMotion frame is inserted between two adjacent raw data frames, and in the second iteration two additional AiviaMotion frames are added (one between the “left” raw data frame and the AiviaMotion frame from the first iteration, and the other one between the AiviaMotion frame from the first iteration and the “right” raw data frame). So, in total there are three AiviaMotion frames inserted with AiviaMotion level 2. Finally, using the exact same reasoning from above, but with three iterations, AiviaMotion level 3 adds a total of seven frames in between two adjacent raw data frames. In more general terms, the number of frames inserted between two adjacent raw data frames could be calculated by 2i-1 where the parameter i is the number of AiviaMotion iterations.
Along those lines, it is frequently desired to also know the total number of frames used for a given rolling average window size and AiviaMotion level. Thus, here is an example for using a DSE setting with five raw data frames and the resulting number of total frames used for rolling average calculation given different AiviaMotion levels.
AiviaMotion Level (i)
Frames Raw (n)
Frames Total (t)
Generalization (Frames Total)
0 * 4 = 0
n + (20 – 1) * (n – 1) = n
1 * 4 = 4
n + (21 – 1) * (n – 1) = 2n - 1
3 * 4 = 12
n + (22 – 1) * (n – 1) = 4n - 3
7 * 4 = 28
n + (23 – 1) * (n – 1) = 8n - 7
The ratio of using higher AiviaMotion levels is that the SNR can potentially be further improved by averaging over more AiviaMotion-interpolated frames. This strategy is especially applicable when the size of the rolling window cannot be increased (i.e., using more raw images for the calculation is not feasible), because the best temporal dynamics is of key importance for the experiment.
No, AiviaMotion does not increase the number of output frames, i.e., the number of output frames from the Dynamic Signal Enhancement (DSE) is unchanged – irrespective whether the AiviaMotion option is activated or not. However, AiviaMotion does transiently increase the number of frames, i.e., the number of frames used as input for calculating the rolling window of the DSE is increased. Since, however, the rolling window is moved in a way that the stride considers the AiviaMotion level (i.e., it is proportional to how many frames were inserted between adjacent raw data frames), the number of output frames is independent of the selected AiviaMotion level and thus invariant. In summary, AiviaMotion helps to better preserve the temporal dynamics of the specimen by enabling to shrink the size of the rolling window (see also FAQ question 2), but without increasing the frame rate.
For the moment, AiviaMotion can perform frame interpolation for the t-dimension. As AiviaMotion only adds a single parameter to the Dynamic Signal Enhancement (number of t-interpolation iterations), AiviaMotion is easy to use and can be activated by a single click.
Is it scientifically OK to denoise data with Dynamic Signal Enhancement (DSE) and AiviaMotion?
DSE uses a rolling average calculation as its core algorithm. The use of a rolling average is widely established as method for removing noise, with application fields ranging from imaging data to stock charts. The AiviaMotion option adds a deep neural network to DSE processing. In contrast to many other deep learning algorithms for denoising, AiviaMotion does not operate on single images and the spatial domain of the data, but rather on multiple input images and the temporal domain. Accordingly, AiviaMotion excels on “temporal consistency” of the processed data. Furthermore, since AiviaMotion is “only” adding some frames for the DSE rolling window calculation, there is always a blend of “ground-truth” raw-data frames and AiviaMotion frames used for denoising. Finally, the user has always the option to compare DSE results with and without AiviaMotion. Thus, especially for quantitative analysis (e.g., measuring fluorescence intensity levels), the user has easily the possibility to confirm the correct working of AiviaMotion.
Can the AiviaMotion algorithm be trained?
At the moment, we do not provide you with the opportunity to train the AiviaMotion interpolation algorithm. However, we ship AiviaMotion with a pre-trained deep learning model which first had been trained on an extensive publicly available video dataset and then fine-tuned via transfer learning on a domain-specific microscopy dataset. Accordingly, we presume AiviaMotion to generalize well on a wide range of specimen and applications.
Does the weight parameter of Dynamic Signal Enhancement (DSE) have any impact on AiviaMotion frame interpolation?
For DSE, not all frames are counted equally. There is a “weight” parameter used in DSE which determines how much frames close to the center of the rolling window, as well as how much more remote frames, contribute to this calculation.
Yet, the weight parameter is not considered when generating the AiviaMotion frames via frame interpolation, because, in any case, only the two adjacent frames are used from the AiviaMotion algorithm. When applying the rolling window calculation on both raw data frames and AiviaMotion frames, the weight parameter is considered as described above for the rolling window calculation. In summary, the weight parameter is not used for the AiviaMotion frame interpolation, but used later during the rolling average calculation.
How does processing time scale with the number of AiviaMotion iterations?
The time necessary for each AiviaMotion round is not uniform. As every AiviaMotion iteration practically doubles the number of input images for the next iteration, the following AiviaMotion iteration takes twice as long as the previous round. Accordingly, while the second iteration would take twice the first interval, the third iteration would take quadruple the first interval. When combined with LIGHTNING, the processing time of both Dynamic Signal Enhancement (DSE including AiviaMotion) and LIGHTNING simply adds up, because they are executed sequentially. Moreover, since the AiviaMotion frames are only transient (i.e., the number of output frames from the DSE is unchanged – irrespective of the application of the AiviaMotion level), the processing time of LIGHTNING is independent from the employed AiviaMotion level.