Skip to content

fix(vertex_ai): improve passthrough endpoint url parsing and construction (#17402) #17526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
krisxia0506 wants to merge 3 commits into
base: main
Choose a base branch
from
+650 −7
Open

fix(vertex_ai): improve passthrough endpoint url parsing and construction (#17402) #17526

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ecbe6cc
fix(vertex_ai): improve passthrough endpoint url parsing and construc…
krisxia0506 Dec 5, 2025
5ad09a6
test(proxy): add test for vertex passthrough load balancing
krisxia0506 Dec 8, 2025
5e7bb0c
feat(proxy): add pass-through deployment filtering methods
krisxia0506 Dec 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions litellm/llms/vertex_ai/common_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,16 @@ def get_vertex_location_from_url(url: str) -> Optional[str]:
return match.group(1) if match else None


def get_vertex_model_id_from_url(url: str) -> Optional[str]:
"""
Get the vertex model id from the url

`https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:streamGenerateContent`
"""
match = re.search(r"/models/([^/:]+)", url)
return match.group(1) if match else None


def replace_project_and_location_in_route(
requested_route: str, vertex_project: str, vertex_location: str
) -> str:
Expand Down Expand Up @@ -782,6 +792,15 @@ def construct_target_url(
if "cachedContent" in requested_route:
vertex_version = "v1beta1"

# Check if the requested route starts with a version
# e.g. /v1beta1/publishers/google/models/gemini-3-pro-preview:streamGenerateContent
if requested_route.startswith("/v1/"):
vertex_version = "v1"
requested_route = requested_route.replace("/v1/", "/", 1)
elif requested_route.startswith("/v1beta1/"):
vertex_version = "v1beta1"
requested_route = requested_route.replace("/v1beta1/", "/", 1)

base_requested_route = "{}/projects/{}/locations/{}".format(
vertex_version, vertex_project, vertex_location
)
Expand Down
20 changes: 20 additions & 0 deletions litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1547,6 +1547,7 @@ async def _base_vertex_proxy_route(
from litellm.llms.vertex_ai.common_utils import (
construct_target_url,
get_vertex_location_from_url,
get_vertex_model_id_from_url,
get_vertex_project_id_from_url,
)

Expand Down Expand Up @@ -1576,6 +1577,25 @@ async def _base_vertex_proxy_route(
vertex_location=vertex_location,
)

if vertex_project is None or vertex_location is None:
Copy link
Contributor

@krrishdholakia krrishdholakia Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should make sure user has access to the model before allowing request to go through

can you add the extraction logic here -

litellm/litellm/proxy/auth/user_api_key_auth.py

Line 943 in 5df701d

model = get_model_from_request(request_data, route)

Copy link
Contributor

@krrishdholakia krrishdholakia Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or some version of the logic maybe in your code block?

maybe extract model and just run can_key_call_model - to confirm valid access before proceeding

Copy link
Contributor Author

@krisxia0506 krisxia0506 Dec 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I missed this question "about whether the key has permission to access the model"
I will fix it.

Copy link
Contributor Author

@krisxia0506 krisxia0506 Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the original pass_through_endpoints logic did not check whether the key had permission to call the model. I have fixed this issue in the new PR #17970.

Copy link
Contributor Author

@krisxia0506 krisxia0506 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@krrishdholakia Hi, This permission issue has been merged in pr #18097, please review this PR again, is there anything else to be modified?

# Check if model is in router config
model_id = get_vertex_model_id_from_url(endpoint)
if model_id:
from litellm.proxy.proxy_server import llm_router

if llm_router:
try:
# Use the dedicated pass-through deployment selection method to automatically filter use_in_pass_through=True
deployment = llm_router.get_available_deployment_for_pass_through(model=model_id)
if deployment:
litellm_params = deployment.get("litellm_params", {})
vertex_project = litellm_params.get("vertex_project")
vertex_location = litellm_params.get("vertex_location")
except Exception as e:
verbose_proxy_logger.debug(
f"Error getting available deployment for model {model_id}: {e}"
)

vertex_credentials = passthrough_endpoint_router.get_vertex_credentials(
project_id=vertex_project,
location=vertex_location,
Expand Down
Loading
Loading