Professor pavan ramdya of the school of life sciences at the Federal Institute of Technology (EPFL) in Lausanne, Switzerland, said: "We used two kinds of data to build neuromechfly. First, we photographed a real fruit fly and performed CT scanning to build a realistic biomechanical model. the second data source is the real limb movement of fruit fly, which was obtained using the posture estimation software we developed in the past few years, which allows us to accurately track the movement of animals. "
Ramdya's team, in collaboration with Professor Auke ijspeert of the EPFL biological Robotics Laboratory, published a paper in the journal natural methods on May 11, showing the precise "digital twin" of the first Drosophila melanogaster known as "neuromechfly".
Drosophila is the most commonly used insect in life science and the focus of ramdya's long-term research. He has been committed to digital tracking and modeling of this animal for many years. In 2019, his team published deepfly3d, a motion capture software based on deep learning, which uses multiple camera perspectives to quantify the movement of fruit flies in three-dimensional space.
After continuing the in-depth study, ramdya's team published liftpose3d in 2021, a method to reconstruct 3D animal posture from 2D images taken by a single camera. These types of breakthroughs provide practical tools for the explosive field of neuroscience and animal inspired robotics.
In many ways, neuromechfly represents the culmination of all these efforts. Subject to the morphological and kinematic data of previous studies, the model has an independent calculation part to simulate different parts of the insect body. This includes a biomechanical exoskeleton with articulated body parts, such as head, legs, wings, abdominal segments, antennae, tail (an organ that helps flies measure their direction in flight), and a neural network "controller" with motor output.
Why establish a "digital twin" of Drosophila?
"How do we know we understand a system?" Ramdya said. "One way is to be able to recreate it. We may try to build a robot Drosophila, but building a simulated animal is much faster and easier. Therefore, one of the main motivations behind this work is to start building a model that integrates our understanding of the nervous system and biomechanics of Drosophila to test whether it is sufficient to explain its behavior."
"When we do experiments, we tend to be motivated by assumptions," he added. "Until now, we have relied on intuition and logic to make assumptions and predictions. However, as neuroscience becomes more and more complex, we rely more on models that can bring together many intertwined components, play their role, and predict what will happen if you make an adjustment here or there."
Test platform
Neuromechfly provides a very valuable test platform for promoting the research of biomechanics and biotechnology, but it is limited to accurately representing real animals in the digital environment. Verifying this is one of the main concerns of researchers. "We conducted validation experiments to prove that we can closely replicate the behavior of real animals," ramdya said.
The researchers first made 3D measurements of real walking and combing fruit flies. They then used neuromechfly's biomechanical exoskeleton to replay these behaviors in a physics based simulation environment.
As they showed in their paper, the model can actually predict various motion parameters, which cannot be measured otherwise, such as leg torque and contact reaction force with the ground. Finally, they were able to use all neuromechfly's neuromechanical capabilities to discover neural networks and muscle parameters, enabling fruit flies to "run" in a way that optimizes both speed and stability.
"These case studies build our confidence in the model," ramdya said. "But what we are most interested in is pointing out ways to improve the model when the simulation fails to replicate animal behavior." Therefore, neuromechfly represents a powerful test platform for building an understanding of how the interaction between complex neuromechanical systems and their physical environment produces behavior.
A community effort
Ramdya stressed that neuromechfly has always been and will continue to be a community project. Therefore, the software is open source and can be used and modified by scientists free of charge. "We have built a tool, not only for ourselves, but also for others. Therefore, we open source and modularize it, and provide guidance on how to use and modify it."
He added: "The progress of science depends more and more on the efforts of the community. It is important for the community to use the model and improve it. But one thing neuromechfly has done is to raise standards. In the past, because the model is not very realistic, we didn't ask them how they can get information directly from the data. Here, we have shown how you can do this; you can take the model, replay the behavior, and infer that there are Meaningful information. So I think this is a big step forward. "