Skip to content

Update inference flows #5340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aviruthen merged 1 commit into from
Nov 25, 2025
Merged

Update inference flows #5340

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions v3-examples/inference-examples/huggingface-example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@
"This notebook demonstrates how to deploy HuggingFace models using SageMaker V3 ModelBuilder for text generation tasks."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prerequisites\n",
"Note: Ensure you have sagemaker and ipywidgets installed in your environment. The ipywidgets package is required to monitor endpoint deployment progress in Jupyter notebooks."
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -292,7 +300,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "venv-test",
"language": "python",
"name": "python3"
},
Expand All @@ -306,9 +314,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}
6 changes: 3 additions & 3 deletions v3-examples/inference-examples/in-process-mode-example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "venv-test",
"language": "python",
"name": "python3"
},
Expand All @@ -327,9 +327,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}
14 changes: 11 additions & 3 deletions v3-examples/inference-examples/inference-spec-example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@
"This notebook demonstrates how to create and deploy custom models using InferenceSpec with SageMaker V3 ModelBuilder."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prerequisites\n",
"Note: Ensure you have sagemaker and ipywidgets installed in your environment. The ipywidgets package is required to monitor endpoint deployment progress in Jupyter notebooks.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -311,7 +319,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "venv-test",
"language": "python",
"name": "python3"
},
Expand All @@ -325,9 +333,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}
44 changes: 29 additions & 15 deletions v3-examples/inference-examples/jumpstart-e2e-training-example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@
"This notebook demonstrates how to use SageMaker V3 to train a JumpStart model from scratch and deploy it for inference."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prerequisites\n",
"Note: Ensure you have sagemaker and ipywidgets installed in your environment. The ipywidgets package is required to monitor endpoint deployment progress in Jupyter notebooks.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -41,7 +49,7 @@
"outputs": [],
"source": [
"# Configuration\n",
"MODEL_ID = \"huggingface-llm-falcon-7b-bf16\"\n",
"MODEL_ID = \"huggingface-spc-bert-base-cased\"\n",
"MODEL_NAME_PREFIX = \"js-e2e-example-model\"\n",
"ENDPOINT_NAME_PREFIX = \"js-e2e-example-endpoint\"\n",
"\n",
Expand All @@ -62,7 +70,7 @@
"source": [
"## Step 2: Train the Model\n",
"\n",
"Use ModelTrainer to train the JumpStart model. This step will take significant time."
"Use ModelTrainer to train the JumpStart model. The training job may take 30+ minutes to complete."
]
},
{
Expand All @@ -75,11 +83,15 @@
"jumpstart_config = JumpStartConfig(model_id=MODEL_ID)\n",
"\n",
"# Initialize ModelTrainer from JumpStart config\n",
"model_trainer = ModelTrainer.from_jumpstart_config(jumpstart_config=jumpstart_config)\n",
"model_trainer = ModelTrainer.from_jumpstart_config(\n",
" jumpstart_config=jumpstart_config, \n",
" base_job_name=training_job_name, \n",
" hyperparameters={\"epochs\": 1}\n",
")\n",
"\n",
"# Train the model\n",
"print(\"Starting model training...\")\n",
"model_trainer.train(training_job_name=training_job_name)\n",
"model_trainer.train()\n",
"print(f\"Training completed: {training_job_name}\")"
]
},
Expand All @@ -99,9 +111,10 @@
"outputs": [],
"source": [
"# Build ModelBuilder from trained model\n",
"print(\"Building model from training artifacts...\")\n",
"model_builder = model_trainer.build()\n",
"print(\"ModelBuilder created successfully from training artifacts!\")"
"model_builder = ModelBuilder(\n",
" model=model_trainer,\n",
" dependencies={\"auto\": False}\n",
")"
]
},
{
Expand Down Expand Up @@ -150,7 +163,9 @@
"source": [
"## Step 6: Test the Endpoint\n",
"\n",
"Send a test request to the deployed endpoint."
"This endpoint performs text entailment classification, determining the logical relationship between pairs of sentences. The returned scores indicate the model's confidence for different entailment categories (e.g., entailment, contradiction, neutral) - higher scores indicate stronger predictions for each relationship type. \n",
"\n",
"This sends a test request to the deployed endpoint."
]
},
{
Expand All @@ -160,16 +175,15 @@
"outputs": [],
"source": [
"# Test with a sample query for the trained model\n",
"test_data = {\"inputs\": \"What are falcons?\", \"parameters\": {\"max_new_tokens\": 32}}\n",
"test_data = [\"The weather is sunny today\", \"It is not raining\"]\n",
"\n",
"result = core_endpoint.invoke(\n",
" body=json.dumps(test_data),\n",
" content_type=\"application/json\"\n",
" content_type=\"application/list-text\"\n",
")\n",
"\n",
"# Decode the output of the invocation and print the result\n",
"prediction = json.loads(result.body.read().decode('utf-8'))\n",
"print(f\"Result of invoking trained endpoint: {prediction}\")"
"entailment_scores = json.loads(result.body.read().decode('utf-8'))\n",
"print(f\"Result of invoking trained endpoint: {entailment_scores}\")\n"
]
},
{
Expand Down Expand Up @@ -218,7 +232,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "venv-test",
"language": "python",
"name": "python3"
},
Expand All @@ -232,7 +246,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.11"
}
},
"nbformat": 4,
Expand Down
18 changes: 14 additions & 4 deletions v3-examples/inference-examples/jumpstart-example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@
"This notebook demonstrates how to use SageMaker V3 ModelBuilder with JumpStart models for easy model deployment and inference."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prerequisites\n",
"Note: Ensure you have sagemaker and ipywidgets installed in your environment. The ipywidgets package is required to monitor endpoint deployment progress in Jupyter notebooks."
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -21,7 +29,8 @@
"\n",
"from sagemaker.serve.model_builder import ModelBuilder\n",
"from sagemaker.core.jumpstart.configs import JumpStartConfig\n",
"from sagemaker.core.resources import EndpointConfig"
"from sagemaker.core.resources import EndpointConfig\n",
"from sagemaker.train.configs import Compute"
]
},
{
Expand Down Expand Up @@ -69,8 +78,9 @@
"outputs": [],
"source": [
"# Initialize model_builder object with JumpStart configuration\n",
"compute = Compute(instance_type=\"ml.g5.2xlarge\")\n",
"jumpstart_config = JumpStartConfig(model_id=MODEL_ID)\n",
"model_builder = ModelBuilder.from_jumpstart_config(jumpstart_config=jumpstart_config)\n",
"model_builder = ModelBuilder.from_jumpstart_config(jumpstart_config=jumpstart_config, compute=compute)\n",
"\n",
"print(\"ModelBuilder created successfully from JumpStart config!\")"
]
Expand Down Expand Up @@ -188,7 +198,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "venv-test",
"language": "python",
"name": "python3"
},
Expand All @@ -202,7 +212,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.11"
}
},
"nbformat": 4,
Expand Down
21 changes: 17 additions & 4 deletions v3-examples/inference-examples/local-mode-example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@
"from sagemaker.serve.mode.function_pointers import Mode"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# NOTE: Local mode requires Docker to be installed and running. \n",
"# If Docker is not in your system PATH, you may need to define the Docker path in one of the top cells.\n",
"# Here is an example:\n",
"import os\n",
"os.environ['PATH'] = '/usr/local/bin:/Applications/Docker.app/Contents/Resources/bin:' + os.environ['PATH']"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -217,7 +230,7 @@
"source": [
"## Step 6: Deploy in Local Container\n",
"\n",
"Deploy the model in a local Docker container. This may take a few minutes to pull the container image."
"Deploy the model in a local Docker container. This may take a few minutes to pull the container image and ping the container until it is live. This is a normal part of the deployment process."
]
},
{
Expand Down Expand Up @@ -395,7 +408,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "venv-test",
"language": "python",
"name": "python3"
},
Expand All @@ -409,9 +422,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}
18 changes: 13 additions & 5 deletions v3-examples/inference-examples/optimize-example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@
"This notebook demonstrates how to use SageMaker V3 ModelBuilder to optimize a JumpStart model for improved inference performance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prerequisites\n",
"Note: Ensure you have sagemaker and ipywidgets installed in your environment. The ipywidgets package is required to monitor endpoint deployment progress in Jupyter notebooks."
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -124,7 +132,6 @@
" model=MODEL_ID,\n",
" schema_builder=schema_builder,\n",
" sagemaker_session=sagemaker_session,\n",
" model_name=model_name,\n",
")\n",
"print(\"ModelBuilder created successfully!\")"
]
Expand All @@ -135,7 +142,7 @@
"source": [
"## Step 5: Optimize the Model\n",
"\n",
"Optimize the model using AWQ quantization for improved inference performance."
"Optimize the model using AWQ quantization for improved inference performance. This step may take up to 30 minutes to complete!"
]
},
{
Expand All @@ -152,7 +159,8 @@
" output_path=f\"s3://sagemaker-us-east-2-593793038179/optimize-output/jumpstart-{unique_id}/\",\n",
" quantization_config={\"OverrideEnvironment\": {\"OPTION_QUANTIZE\": \"awq\"}},\n",
" accept_eula=True,\n",
" job_name=job_name\n",
" job_name=job_name,\n",
" model_name=model_name,\n",
")\n",
"print(f\"Model Successfully Optimized: {optimized_model.model_name}\")"
]
Expand Down Expand Up @@ -257,7 +265,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "venv-test",
"language": "python",
"name": "python3"
},
Expand All @@ -271,7 +279,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.11"
}
},
"nbformat": 4,
Expand Down
Loading
Loading