Skip to content

Commit 3e26dfd

Browse files
julianwoodjulianwood
authored
Merge pull request #2362 from enmata/s3-eventbridge-sfn-terraform_enmata
Add s3-eventbridge-sfn-terraform patterrn
2 parents d5ee3d7 + 367cf45 commit 3e26dfd

File tree

4 files changed

+472
-0
lines changed

4 files changed

+472
-0
lines changed

‎s3-eventbridge-sfn-terraform/README.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Amazon S3 to AWS Step Functions with Amazon EventBridge Rule
2+
3+
This pattern demonstrates invoking a Step Functions state machine from an S3 event via EventBridge. Implemented with Terraform.
4+
5+
Learn more about this pattern at Serverless Land Patterns: [erverlessland.com/patterns/s3-eventbridge-sfn-terraform](https://serverlessland.com/patterns/s3-eventbridge-sfn-terraform)
6+
7+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
8+
9+
10+
## Requirements
11+
12+
- [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
13+
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
14+
- [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
15+
* [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli?in=terraform/aws-get-started) installed
16+
17+
18+
## Deployment Instructions
19+
20+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
21+
```
22+
git clone https://github.com/aws-samples/serverless-patterns
23+
```
24+
1. Change directory to the pattern directory:
25+
```
26+
cd s3-eventbridge-sfn-terraform
27+
```
28+
1. From the command line, initialize terraform to to downloads and installs the providers defined in the configuration:
29+
```
30+
terraform init
31+
```
32+
1. From the command line, apply the configuration in the main.tf file:
33+
```
34+
terraform apply
35+
```
36+
1. During the prompts:
37+
* Enter yes
38+
1. Note the outputs from the deployment process. These contain the resource names and/or ARNs which are used for testing.
39+
40+
41+
## How it works
42+
43+
- Upload a file to the newly created S3 bucket
44+
- This sends an `Object Created` event to EventBridge
45+
- Based on the EventBridge rule, the state machine is executed
46+
47+
48+
## Testing
49+
50+
1. Navigate to Amazon S3 console, then choose the S3 bucket created by this Terraform template.
51+
52+
2. Upload a file to the S3 bucket
53+
54+
3. Immediately after the upload completes successfully, navigate to the Step Functions console. Select the state machine created by the template, and observe the state machine execution.
55+
56+
57+
## Documentation and next step
58+
59+
To create a full Step Functions workflow with this pattern, search the example workflows available at [serverlessland.com/workflows](https://serverlessland.com/workflows)
60+
61+
62+
## Cleanup
63+
64+
1. Change directory to the pattern directory:
65+
```
66+
cd s3-eventbridge-sfn-terraform
67+
```
68+
1. Delete all files from the S3 bucket
69+
1. Delete all created resources by terraform
70+
```bash
71+
terraform destroy
72+
```
73+
1. During the prompts:
74+
* Enter yes
75+
1. Confirm all created resources has been deleted
76+
```bash
77+
terraform show
78+
```
79+
80+
----
81+
Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
82+
83+
SPDX-License-Identifier: MIT-0

‎s3-eventbridge-sfn-terraform/example-pattern.json‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
{
2+
"title": "S3 to Step Functions using EventBridge",
3+
"description": "How to create an EventBridge rule with S3 as the event source and Step Functions as target.",
4+
"language": "nodejs",
5+
"level": "200",
6+
"framework": "Terraform",
7+
"introBox": {
8+
"headline": "How to create an EventBridge rule with S3 as the event source and Step Functions as target.",
9+
"text": [
10+
"This sample project demonstrates how you can trigger an execution in Step Functions state machine when an object is created in an S3 bucket.",
11+
"When you uploads an object to the newly created S3 bucket, this will send an `Object Created` event to EventBridge, based on the EventBridge rule, the state machine is executed.",
12+
"This pattern deploys one Step Functions state machine, one S3 bucket, one EventBridge rule."
13+
]
14+
},
15+
"gitHub": {
16+
"template": {
17+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/s3-sqs-lambda-terraform",
18+
"templateURL": "serverless-patterns/s3-sqs-lambda-terraform",
19+
"projectFolder": "s3-sqs-lambda-terraform",
20+
"templateFile": "main.tf"
21+
}
22+
},
23+
"resources": {
24+
"bullets": [
25+
{
26+
"text": "AWS S3 Event Notifications with EventBridge",
27+
"link": "https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge"
28+
},
29+
{
30+
"text": "Using EventBridge with S3",
31+
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html"
32+
}
33+
]
34+
},
35+
"deploy": {
36+
"text": [
37+
"terraform init",
38+
"terraform apply"
39+
]
40+
},
41+
"testing": {
42+
"text": [
43+
"See the GitHub repo for detailed testing instructions."
44+
]
45+
},
46+
"cleanup": {
47+
"text": [
48+
"terraform destroy"
49+
]
50+
},
51+
"authors": [
52+
{
53+
"name": "Oriol Matavacas",
54+
"image": "https://togithub.s3.eu-west-1.amazonaws.com/Oriol.jpg",
55+
"bio": "Oriol Matavacas is a Sr. Solutions Architect at AWS based in Barcelona. Oriol primarily supporting customers on the journey to the Cloud. He enjoys building new solutions with scalability, availability and easy to maintain by using serverless.",
56+
"linkedin": "https://www.linkedin.com/in/oriol-matavacas-rodriguez-b165868a",
57+
"twitter": ""
58+
}
59+
]
60+
}
61+

‎s3-eventbridge-sfn-terraform/main.tf‎

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
terraform {
2+
required_providers {
3+
aws = {
4+
source = "hashicorp/aws"
5+
version = "~> 3.27"
6+
}
7+
}
8+
9+
required_version = ">= 0.14.9"
10+
}
11+
12+
# Storing current Account ID and actual AWS region
13+
data "aws_caller_identity" "current" {}
14+
data "aws_region" "current" {}
15+
16+
17+
#################################################################
18+
# S3 Bucket
19+
#################################################################
20+
# Creating a new S3 bucket
21+
resource "aws_s3_bucket" "MyS3Bucket" {
22+
bucket_prefix = "s3-eventbridge-sfn-tf-"
23+
}
24+
25+
# Sending notifications to EventBridge for all events in the bucket
26+
resource "aws_s3_bucket_notification" "MyS3BucketNotification" {
27+
bucket = aws_s3_bucket.MyS3Bucket.id
28+
eventbridge = true
29+
}
30+
31+
# Blocking s3 bucket non secure access
32+
resource "aws_s3_bucket_policy" "MyS3BucketPolicy" {
33+
bucket = aws_s3_bucket.MyS3Bucket.id
34+
policy = data.aws_iam_policy_document.MyS3BucketPolicyDocument.json
35+
}
36+
37+
data "aws_iam_policy_document" "MyS3BucketPolicyDocument" {
38+
statement {
39+
effect = "Deny"
40+
principals {
41+
type = "AWS"
42+
identifiers = ["*"]
43+
}
44+
actions = [
45+
"s3:*"
46+
]
47+
resources = [
48+
"arn:aws:s3:::${aws_s3_bucket.MyS3Bucket.id}",
49+
"arn:aws:s3:::${aws_s3_bucket.MyS3Bucket.id}/*"
50+
]
51+
condition {
52+
test = "Bool"
53+
variable = "aws:SecureTransport"
54+
values = ["false"]
55+
}
56+
}
57+
}
58+
59+
60+
#################################################################
61+
# EventBridge Rule
62+
#################################################################
63+
# Creating an EventBridge rule
64+
resource "aws_cloudwatch_event_rule" "MyEventRule" {
65+
name = "s3-eventbridge-sfn-tf-eventrule"
66+
description = "Object create events on bucket s3://${aws_s3_bucket.MyS3Bucket.id}"
67+
event_pattern = <<EOF
68+
{
69+
"detail-type": ["Object Created"],
70+
"source": ["aws.s3"],
71+
"detail": {
72+
"bucket": {
73+
"name": ["${aws_s3_bucket.MyS3Bucket.id}"]
74+
}
75+
}
76+
}
77+
EOF
78+
}
79+
80+
# Setting Event Target, DLQ and transformer of the EventBridge rule
81+
resource "aws_cloudwatch_event_target" "MyEventRuleTarget" {
82+
rule = aws_cloudwatch_event_rule.MyEventRule.name
83+
arn = aws_sfn_state_machine.MySTFRoleMachine.arn
84+
role_arn = aws_iam_role.MyEventRuleTargetRole.arn
85+
86+
dead_letter_config {
87+
arn = aws_sqs_queue.MySQSDLQ-Queue.arn
88+
}
89+
90+
input_transformer {
91+
input_paths = {
92+
"detail": "$.detail"
93+
}
94+
input_template = <<EOF
95+
{"input":{"detail":<detail>},"inputType":0}
96+
EOF
97+
}
98+
}
99+
100+
# Creating Event Target IAM Role
101+
resource "aws_iam_role" "MyEventRuleTargetRole" {
102+
name = "s3-eventbridge-sfn-tf-EventRole"
103+
assume_role_policy = jsonencode({
104+
Version = "2012-10-17"
105+
Statement = [
106+
{
107+
Effect = "Allow"
108+
Principal = { Service = "events.amazonaws.com" }
109+
Action = "sts:AssumeRole"
110+
}
111+
]
112+
})
113+
}
114+
115+
resource "aws_iam_role_policy" "MyEventRuleTargetPolicy" {
116+
name = "s3-eventbridge-sfn-tf-EventRolePolicy"
117+
policy = jsonencode(
118+
{
119+
"Version": "2012-10-17",
120+
"Statement": [
121+
{
122+
"Action": "states:StartExecution",
123+
"Resource": "${aws_sfn_state_machine.MySTFRoleMachine.arn}",
124+
"Effect": "Allow"
125+
}
126+
]
127+
}
128+
)
129+
role = aws_iam_role.MyEventRuleTargetRole.name
130+
}
131+
132+
133+
#################################################################
134+
# SQS - Dead Letter Queue (DLQ)
135+
#################################################################
136+
# Creating SQS - Dead Letter Queue (DLQ)
137+
resource "aws_sqs_queue" "MySQSDLQ-Queue" {
138+
name = "s3-eventbridge-sfn-tf-SQSDLQQueue"
139+
}
140+
141+
resource "aws_sqs_queue_policy" "MySQSDLQ-Policy" {
142+
queue_url = aws_sqs_queue.MySQSDLQ-Queue.id
143+
144+
policy = <<POLICY
145+
{
146+
"Version": "2012-10-17",
147+
"Statement": [
148+
{
149+
"Effect": "Deny",
150+
"Principal": {
151+
"AWS": "*"
152+
},
153+
"Action": "sqs:*",
154+
"Resource": "arn:aws:sqs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:${aws_sqs_queue.MySQSDLQ-Queue.name}",
155+
"Condition": {
156+
"Bool": {
157+
"aws:SecureTransport": "false"
158+
}
159+
}
160+
},
161+
{
162+
"Sid": "AllowEventRuleS3EventBridgeSfnStackStackpatternEventsRule",
163+
"Effect": "Allow",
164+
"Principal": {
165+
"Service": "events.amazonaws.com"
166+
},
167+
"Action": "sqs:SendMessage",
168+
"Resource": "arn:aws:sqs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:${aws_sqs_queue.MySQSDLQ-Queue.name}",
169+
"Condition": {
170+
"ArnEquals": {
171+
"aws:SourceArn": "${aws_cloudwatch_event_rule.MyEventRule.arn}"
172+
}
173+
}
174+
}
175+
]
176+
}
177+
POLICY
178+
}
179+
180+
181+
#################################################################
182+
# Step Funcions
183+
#################################################################
184+
# Creating Step Functions Machine
185+
resource "aws_sfn_state_machine" "MySTFRoleMachine" {
186+
name = "s3-eventbridge-sfn-tf-StepFunctions"
187+
role_arn = aws_iam_role.MySTFRole.arn
188+
type = "STANDARD"
189+
190+
definition = <<EOF
191+
{
192+
"StartAt": "state",
193+
"States": {
194+
"state": {
195+
"Type": "Pass",
196+
"End": true
197+
}
198+
}
199+
}
200+
EOF
201+
202+
}
203+
204+
# Creating Step Functions Machine Role
205+
resource "aws_iam_role" "MySTFRole" {
206+
name = "s3-eventbridge-sfn-tf-StepFunctionsRole"
207+
assume_role_policy = jsonencode({
208+
Version = "2012-10-17"
209+
Statement = [
210+
{
211+
Effect = "Allow"
212+
Principal = { Service = "states.${data.aws_region.current.name}.amazonaws.com" }
213+
Action = "sts:AssumeRole"
214+
}
215+
]
216+
})
217+
}
218+
219+
220+
#################################################################
221+
# Outputs
222+
#################################################################
223+
# Displaying the SQS Dead Letter Queue, EventBridge rule, S3 bucket and StepFunctions
224+
output "SQSDeadLetterQueue" {
225+
value = aws_sqs_queue.MySQSDLQ-Queue.arn
226+
description = "The SQS Dead Letter Queue ARN"
227+
}
228+
output "EventBridgeRuleARN" {
229+
value = aws_cloudwatch_event_rule.MyEventRule.arn
230+
description = "The EventBridge Rule ARN"
231+
}
232+
output "S3BucketName" {
233+
value = aws_s3_bucket.MyS3Bucket.id
234+
description = "The S3 Bucket Name"
235+
}
236+
output "StepFunctions" {
237+
value = aws_sfn_state_machine.MySTFRoleMachine.arn
238+
description = "The StepFunctions Machine ARN"
239+
}

0 commit comments

Comments
 (0)