Ianhelle/msticpy 3.0 and Python 3.13

[ianhelle]

PR #867: MSticpy 3.0 and Python 3.13 Support

Important

Major Version Release - This PR introduces breaking changes. Review the migration guide before upgrading.

📋 Overview

Major version release modernizing the codebase with Python 3.13 support, development tooling improvements, and removal of deprecated features.

---

🎯 Key Changes

🐍 Python Version Support

• ❌ Dropped: Python 3.8 and 3.9
• ✅ Added: Python 3.13
• ⚡ Minimum: Python 3.10+

Updated CI Matrix

matrix:
  python-version: ["3.10", "3.11", "3.12", "3.13"]

---

🛠️ Development Workflow Modernization

Migrated to Ruff

Replaced multiple tools (black, flake8, isort) with unified ruff for linting and formatting:

# pyproject.toml
[tool.ruff]
line-length = 95
target-version = "py310"

[tool.ruff.lint]
select = ["E", "W", "F", "I", "D", "UP", "B", "C4", "PL"]
ignore = ["E401", "E501", "W503", "D212", "D203", "D417"]

Changes:

• ✅ Updated .pre-commit-config.yaml and GitHub Actions workflows
• ✅ Removed obsolete dev dependencies
• ✅ Increased pylint max line length: 90 → 110 characters

Removed dependencies:

- black>=20.8b1
- flake8>=3.8.4
- isort>=5.10.1
- pydocstyle>=6.0.0
- pep8-naming>=0.10.0

---

🗑️ Deprecated Features Removed

Warning

The following features have been completely removed in v3.0

KqlMagic Integration

• Removed all KqlMagic imports and environment variable handling
• Cleaned up msticpy/__init__.py KqlMagic configuration
• Removed from requirements: KqlmagicCustom[jupyter-extended]

AI Agents Modules

- msticpy/aiagents/config_utils.py
- msticpy/aiagents/mp_docs_rag_magic.py

Legacy Query Drivers

• Removed test_kql_driver.py (254 lines)
• Removed test_kusto_driver.py (372 lines)
• Cleaned up legacy driver tests

Deprecated Pandas Accessors

# ❌ Old (removed)
df.mp_timeline.plot()
df.mp_process_tree.plot()
df.mp_whois.lookup()

# ✅ New (use these)
df.mp_plot.timeline()
df.mp_plot.process_tree()
df.mp.whois()

---

📦 Dependencies & Configuration

Package	Change	Impact
importlib-resources	Removed (Python ≤3.8)	Cleanup
mo-sql-parsing	2.x → 11.x	⚠️ Breaking
ipython	Now >=7.23.1 (unified)	Cleanup

Documentation Updates

• 📝 "Azure Sentinel" → "Microsoft Sentinel" (branding update)
• 📝 Updated README.md with Python 3.10+ requirement
• 📝 Removed outdated badge references

---

💻 Code Quality Improvements

Type Hints Modernization

# Before
from typing import Union, Optional, List, Dict

def foo(x: Optional[str]) -> Union[List[str], Dict[str, str]]:
    pass

# After  
def foo(x: str | None) -> list[str] | dict[str, str]:
    pass

Import Modernization

# Before
from typing import Iterable, Iterator

# After
from collections.abc import Iterable, Iterator

Additional improvements:

• 🔧 Applied Ruff formatting across 350+ files
• 🧪 Added pytest filter warnings for deprecated patterns
• 🐛 Fixed timezone warnings (datetime.utcnow() → datetime.now(timezone.utc))

---

🔧 CI/CD Updates

# GitHub Actions improvements
- name: Free Disk Space
  run: # Added disk cleanup steps
  
- name: Setup Python
  with:
    python-version: ["3.10", "3.11", "3.12", "3.13"]

- name: Ruff Check
  run: ruff check msticpy --ignore PLW0603

Simplified workflows:

• ❌ Removed: black, flake8, pydocstyle steps
• ✅ Added: Unified ruff check and ruff format

---

⚠️ Breaking Changes

Users upgrading to v3.0 should note:

1. Python 3.8/3.9 no longer supported - Upgrade to Python 3.10+
2. KqlMagic functionality removed - No direct replacement
3. Legacy pandas accessors deprecated - Use new mp_plot and mp accessors
4. mo-sql-parsing v2 → v11 - SQL parsing behavior may differ
5. Removed AI agents experimental features - Code removed entirely

---

🚀 Migration Guide

Step 1: Upgrade Python

# Check current version
python --version

# Upgrade to 3.10 or later
conda create -n msticpy-3.0 python=3.10
conda activate msticpy-3.0

Step 2: Update Code

Replace deprecated accessor methods:

# Timeline plotting
df.mp_timeline.plot(...)           # ❌ Old
df.mp_plot.timeline(...)           # ✅ New

# Process tree
df.mp_process_tree.plot(...)       # ❌ Old  
df.mp_plot.process_tree(...)       # ✅ New

# WHOIS lookups
df.mp_whois.lookup(...)            # ❌ Old
df.mp.whois(...)                   # ✅ New

Step 3: Update Dependencies

# Upgrade msticpy
pip install --upgrade msticpy==3.0.0.pre1

# Review pinned dependencies
pip list | grep mo-sql-parsing

---

📊 Statistics

• Files Changed: ~350+
• Lines Added: ~15,000+
• Lines Removed: ~10,000+
• Test Coverage: All checks passing ✅
• Reviewers: @ryan-detect-dot-dev (Approved)

---

📚 Related Links

• 📖 Documentation
• 🐛 Issue Tracker
• 💬 Discussions
• 📦 PyPI Release

---

Reviewer Comments

@ryan-detect-dot-dev:

Only minor questions, otherwise LGTM

Resolved discussions:

• Line length alignment between ruff and pylint
• Python 3.8 dependency cleanup
• Target Python version for Ruff configuration

---

Tip

Status: Open | All checks passing ✅
Ready to merge pending final approval

[ryan-detect-dot-dev]

Only minor questions, otherwise LGTM

[FlorianBracq]

Looks great, well done!