Projects
Analysis of behavior across species in the cloud
PI: Pietro Perona (Division of Engineering and Applied Science)
SASE: Anna Ding, Scholar
The analysis of animal behavior is crucial for understanding biological systems, particularly the brain, as well as the effects of pharmacological or genetic interventions. Unfortunately, biologists often resort to manual analysis, a process fraught with issues of time consumption, subjectivity, bias, and potential oversight of crucial ethological aspects of animal behavior. To address these issues, we will build a platform at Caltech for behavioral analysis in the cloud that facilitates all aspects of analysis: data curation and storage, annotation, analysis, visualization, and results sharing. We will save costly hours of human domain experts, increase scientific discovery through unsupervised learning, and increase scientific rigor by increasing the reproducibility of results across labs. Our platform will have an easy-to-use standardized interface that allows users to upload and browse their video data.
The Perona group is collaborating with the Schmidt Academy to develop this ML model platform. Our technical approach includes self-supervised analysis methods to extract behavioral features over time directly from the raw video the user will upload. The Perona group has developed a set of self-supervised algorithms that we will use as the backend of the online platform. This includes an algorithm to extract key points on the animal's body over time, the use of Expert heuristics to project animal trajectories into lower dimensional feature space over time, using key points generated in earlier in the workflow to track multiple individual animals consistently over time, and finally using a Contrastive Learning with Learned Masking (CLLM) algorithm that learns behavioral features over time directly from raw video by learning multiple self-supervised objectives simultaneously.
Overview of our label-free cloud platform: A simple interface that allows users to upload behavioral videos via drag and drop. Subsequently: all data is uploaded to and persistently stored in the cloud. The data is automatically fetched from a computational server that performs training and inference on the videos. After this process is completed, the user can log in again to the platform and view the results: tracked key points over time, tracked animals over time, and behavioral features over time. The resulting data can be exported to standard formats and used in downstream analyses.