• Paper: ➡️📑

Simultaneous 2D Pose-Estimation and Behavior Classification

This repository demonstrates multi-task learning for combined object detection and pose-estimation using YOLO-Pose models. IntegraPose enables robust, simultaneous classification of complex behaviors and precise keypoint tracking from a single model. IntegraPose is a comprehensive suite of tools designed to streamline the entire workflow of behavioral analysis, from initial video annotation to advanced, post-hoc analysis of model outputs. It consists of a primary GUI-based application that integrates various functionalities into a single, cohesive workflow.

There two User Guides to get you started:

• 📖 Comprehensive User Guide: Covers the graphical user interface (GUI), GIFs for workflow steps, advanced YOLO model customization, and example model backbone modules (for user wanting more control over the YOLO model).
• 🚀 Quick start guide: A streamlined tutorial to get you up and running in minutes. Includes screenshots for references.

🚧 Project Status 🚧

This repository is currently under active development. APIs may change, and new features will be added. Feedback and contributions are welcome!

✨ Key Features

1. Simultaneous Tracking & Classification: A single model predicts bounding boxes, keypoint locations, and behavioral class.
2. Modular GUI-Driven Workflow: Easy-to-use graphical interfaces for annotation, training, and analysis, minimizing the need for complex command-line interaction.
3. Integrated Annotation Tool: Interactively label keypoints and assign behavioral classes to subjects in your images.
4. Advanced Bout Analytics: Tools for analyzing, confirming, and scoring behavioral bouts, including manual review and correction.
5. Pose Clustering Analysis: A dedicated tab for exploratory data analysis, clustering, and visualization of pose data.
6. Flexible ROI Management: Define regions of interest dynamically for various analysis modes.
7. Advanced Gait & Kinematic Analysis: A dedicated pipeline offering distinct methods for detecting strides and performing in-depth analysis of movement from keypoint data.
8. HMM-VAE Video Segmentation: An unsupervised workflow to automatically segment continuous pose data into discrete, meaningful behavioral motifs without prior labels.
9. Sub-Behavior Discovery: Employs a Seq2Seq LSTM Autoencoder to discover subtle, stereotyped variations or "sub-behaviors" within a broader action.
10. Tandem YOLO-LSTM Classifier: A powerful approach using YOLO for spatial pose data and an LSTM for robust, temporal behavior classification based on movement patterns. (Does require training two seperate models, but can be run in close to real-time speeds 15-18ms per frame)

🛠️ Prerequisites

Before you begin, ensure you have the following installed:

• Python: Version > 3.11 (tested with 3.11 & 3.12).
• Conda: Recommended for managing dependencies. Core Libraries (can be installed via pip after setting up a Conda environment):

pip install ultralytics opencv-python pandas numpy hdbscan umap-learn matplotlib pyyaml scipy scikit-learn openpyxl XlsxWriter

• Alternatively, you can create a Conda environment using the provided environment.yml file:

conda env create -f environment.yml
conda activate IntegraPose

🚀 Workflow & How to Use

The IntegraPose workflow is managed through a single main GUI application.

To get started, run main_gui_app.py located in BehaviorAnnotation_ModelTraining Folder:

python main_gui_app.py

The application is organized into several tabs, each corresponding to a stage or type of analysis:

1. Setup & Annotation Tab

This tab is for defining your project's keypoints and behaviors, and for launching the integrated annotation tool.

• Define Keypoint Names: Specify the ordered names of your keypoints (e.g., nose, left_ear, right_ear).
• Define Behaviors: List the behaviors you want to classify, each with a unique numeric ID.
• Configure Annotation Directories: Set paths for your source images and where the generated annotation .txt files will be saved.
• Launch Annotator: Click "Launch Annotator" to open the OpenCV-based interactive tool for drawing bounding boxes and labeling keypoints. The annotator supports zoom, pan, and autosave.
• Generate dataset.yaml: After annotation, use this section to create the YOLO-compatible dataset.yaml file, which is crucial for model training.

2. Model Training Tab

Once your dataset is ready, use this tab to configure and initiate the training of your YOLO-Pose model.

• Load Dataset YAML: Point to the dataset.yaml file generated in the Setup tab.
• Select Model Variant: Choose a pre-trained YOLO-Pose model (e.g., yolov8n-pose.pt).
• Adjust Hyperparameters: Configure epochs, learning rate, batch size, and other training parameters.
• Data Augmentation Settings: Control various augmentation techniques like flip, perspective, and mosaic.
• Start Training: Launch the Ultralytics training process directly from the GUI.

3. Inference Tab

Use this tab to run your trained model on new video files or folders containing images.

• Load Trained Model: Select your .pt model file.
• Select Source: Choose a video file or a folder of images for inference.
• Configure Parameters: Adjust confidence thresholds, IOU, and other inference settings.
• Output Options: Define where output videos and detection .txt files will be saved.
• Start Inference: Process your data and generate detection outputs.

4. Webcam Inference Tab

Perform real-time inference using your trained model with a live webcam feed.

• Load Trained Model: Select your .pt model.
• Webcam Index: Specify the index of your webcam (e.g., 0 for the default).
• Live View & Recording: View the live inference and optionally record the annotated video.

5. Bout Analytics Tab

This tab provides tools for analyzing and managing behavioral bouts from your YOLO inference output.

• Load YOLO Output: Load your YOLO detection .txt files (from a folder or individual files).
• ROI Management:
  • Draw New ROI: Interactively draw regions of interest on a video frame. The tool guides you through drawing polygons.
  • Load/Save ROIs: Import and export ROI definitions to/from YAML files for reusability.

• Behavioral Bout Analysis:

  • Calculate bout statistics (duration, frequency, time between bouts).
  • Filter and analyze bouts based on ROI entry/exit.
  • Review & Confirm Detected Bouts: Launch a dedicated tool to manually review and confirm automatically detected bouts.
  • This tool allows for corrections and detailed scoring, saving updated bout information to a CSV file.
  
  • Open Advanced Bout Scorer {Work in-progress}: Access an advanced interface for in-depth manual scoring and verification of behavioral bouts from a video.

6. Pose Clustering Analysis Tab

This tab allows for exploratory data analysis, including normalization, feature engineering, and clustering of pose data.

• Load Data: Import pose data (from YOLO output or other sources).

• Define Keypoints & Skeleton: Specify keypoint names and skeleton connections for feature calculations.

• Normalization & Feature Engineering: Normalize pose data and generate features based on keypoints and skeleton definitions.

• Clustering: Apply UMAP and HDBSCAN clustering to discover patterns in your behavioral data.

• Visualization: Visualize clusters and their relationship to the original data.

# Project Structure The repository is organized as follows:

BehaviorAnnotation_ModelTraining/
├── main_gui_app.py
├── keypoint_annotator_module.py
├── gui/
│   ├── advanced_bout_analyzer.py
│   ├── bout_confirmation_tool.py
│   ├── inference_tab.py
│   ├── log_tab.py
│   ├── pose_clustering_tab.py
│   ├── roi_analytics_tab.py
│   ├── setup_tab.py
│   ├── tooltips.py
│   ├── training_tab.py
│   └── webcam_tab.py
├── utils/
│   ├── bout_analyzer.py
│   ├── command_builder.py
│   ├── config_manager.py
│   ├── roi_drawing_tool.py
│   ├── roi_manager.py
│   ├── video_creator.py
│   └── webcam_manager.py
├── environment.yml
├── requirements.txt
├── README.md
└── bout_tool.log (example log file)

Showcase 🖼️

Here are some examples of IntegraPose in action, classifying distinct behaviors while simultaneously tracking keypoints.

OpenField	Video Source	Behaviors
	Video Source: BehaviorDEPOT	Walking, Wall-Rearing/Supported Rearing
	Video Source: BehaviorDEPOT	Walking, Wall-Rearing/Supported Rearing
	Video Source: BehaviorDEPOT	Walking, Grooming
	Video Source: Temporal_Behavior_Analysis	Exploring/Waling, Wall-Rearing/Supported Rearing
	Video Source: Temporal_Behavior_Analysis	Wall-Rearing/Supported Rearing, Jump
	Video Source: BehaviorDEPOT	Ambulatory/Walking, Object Exploration, Object Mounting
	Video Source: BehaviorDEPOT	Ambulatory/Walking, Object Exploration
	Video Source: Self	Ambulatory/Walking, Nose-Poking, Wall-Rearing/Supported Rear
	Video Source: Self	Ambulatory/Walking, Wall-Rearing/Supported Rear

Citation Suggestions:

Augustine, et al., (2025). Integrapose: A Unified Framework for Simultaneous Pose Estimation and Behavior Classification. https://www.sciencedirect.com/science/article/abs/pii/S0306452225010097

License

IntegraPose is provided under the GNU Affero General Public License v3.0 (AGPL-3.0).

What this means for you

• Using the app (research, analysis, publications): Totally fine. You can run IntegraPose internally and publish results/ licenses however you like; the AGPL doesn’t limit what you learn or publish.
• Modifying or redistributing IntegraPose: If you share the altered program or host it for others (e.g., a web service), you must provide your changes’ source code under AGPL too.
• Integrating Ultralytics: The AGPL choice keeps the alignment with Ultralytics’ AGPL license. If your group has a commercial exception from Ultralytics, you can apply that to IntegraPose as well.

Need more detail? See GNU’s AGPL overview.

Acknowledgments:

IntegraPose is built upon the powerful and flexible Ultralytics YOLOv8 framework. We extend our sincere gratitude to the Ultralytics team for their significant contributions to the open-source community.

Repo Underdevelopment 🚧🏗️👷🏼