Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ jobs:

env:
AUTHOR: ${{ github.actor }}
BASTION_HOST_IPV4_ALLOW_LIST: ${{ vars.BASTION_HOST_IPV4_ALLOW_LIST }}
CERTIFICATE_ARN: ${{ vars.CERTIFICATE_ARN }}
COMMIT_SHA: ${{ github.sha }}
DB_ALLOCATED_STORAGE: ${{ vars.DB_ALLOCATED_STORAGE }}
Expand Down
12 changes: 4 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,17 @@

[![Tests Status](https://github.com/MAAP-Project/maap-eoapi/actions/workflows/tests.yml/badge.svg)]((https://github.com/MAAP-Project/maap-eoapi/actions?query=workflow:tests))


## Overview

This repository contains the AWS CDK code (written in typescript) used to deploy the MAAP project eoapi infrastructure. It is based on the [eoapi-template example](https://github.com/developmentseed/eoapi-template). For the MAAP use case, we use a subset of the eoapi CDK constructs to define a database, an ingestion API, a STAC API, a raster API (i.e a tiling API) and a pgbouncer instance to manage connections to the database. Here, we deploy all these components into a custom VPC.

This repository contains the AWS CDK code (written in typescript) used to deploy the MAAP project eoapi infrastructure. It is based on the [eoapi-template example](https://github.com/developmentseed/eoapi-template). For the MAAP use case, we use a subset of the eoapi CDK constructs to define a database, an ingestion API, a STAC API, a raster API (i.e a tiling API) and a bastion host for direct connections to the database. Here, we deploy all these components into a custom VPC.

## Automated Deployment

Deployment happens through a github workflow manually triggered and defined in `.github/workflows/deploy.yaml`.


## Networking and accessibility of the database.

Because of security requirements, the networking set up imposes the following constraints :
Expand All @@ -23,13 +25,12 @@ This has three consequences :
2. In addition, because these APIs _also_ sometimes need access to the internet, a NAT gateway must in addition be deployed in that VPC.
3. For direct, administrative connections to the database, one _must_ go through an instance placed in the same VPC as the database.

We approach (3) by re-using the 'bastion host' eoAPI construct, which deploys an EC2 instance that can connect to the database, and can be used to create a tunnel from a user's machine to the database. See the eoAPI docs for more information.


## Ingestion

The term "ingestion" refers to the process of cataloging data in the STAC catalog associated with this deployment.


### Direct ingestion

For a small record ingestion (for example a collection record or just one item), one can directly connect to the database and perform loading. This can be done using the `pypgstac` library. For example, to load an item stored locally in `test_item.json`, with `pypgstac` installed, you can run the following command :
Expand All @@ -44,11 +45,6 @@ or for a collection
pypgstac load --table collections test_collection.json
```

This requires

1. that you are allowed to connect to the database. Because of the security requirements mentioned above, you must go through an SSH tunnel using the bastion host EC2 instance. For this to work, you must be allowed to SSH into this EC@ instance [See these docs for more details](https://developmentseed.org/eoapi-cdk/#bastionhost-)
2. [the configuration](https://stac-utils.github.io/pgstac/pypgstac/) for the database connection is present in your environment.


### Indirect ingestion through the ingestion pipeline deployment

Expand Down
21 changes: 0 additions & 21 deletions cdk/PgStacInfra.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ import {
import { Aws, Duration, RemovalPolicy, Stack, StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import {
BastionHost,
CustomLambdaFunctionProps,
PgStacApiLambda,
PgStacDatabase,
Expand Down Expand Up @@ -322,16 +321,6 @@ export class PgStacInfra extends Stack {
ingestorConfig.dataAccessRoleArn,
);

new BastionHost(this, "bastion-host", {
vpc,
db: pgstacDb.db,
ipv4Allowlist: ingestorConfig.ipv4AllowList,
userData: ec2.UserData.custom(
readFileSync(ingestorConfig.userDataPath, { encoding: "utf-8" }),
),
createElasticIp: ingestorConfig.createElasticIp,
});

new StacIngestor(this, "stac-ingestor", {
vpc,
stacUrl: stacApiLambda.url,
Expand Down Expand Up @@ -658,16 +647,6 @@ export interface Props extends StackProps {
* Where userdata.yaml is found.
*/
userDataPath: string;

/**
* Which IPs to allow to access bastion host.
*/
ipv4AllowList: string[];

/**
* Flag to control whether the Bastion Host should make a non-dynamic elastic IP.
*/
createElasticIp?: boolean;
};
dpsStacItemGenConfig?: {
itemGenRoleArn: string;
Expand Down
5 changes: 1 addition & 4 deletions cdk/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ const {
certificateArn,
dbAllocatedStorage,
dbInstanceType,
bastionHostIpv4AllowList,
ingestorDataAccessRoleArn,
ingestorDomainName,
jwksUrl,
Expand Down Expand Up @@ -82,9 +81,7 @@ new PgStacInfra(app, buildStackName("pgSTAC"), {
jwksUrl,
dataAccessRoleArn: ingestorDataAccessRoleArn,
domainName: ingestorDomainName,
userDataPath: "./userdata.yaml",
ipv4AllowList: bastionHostIpv4AllowList,
createElasticIp: stage === "prod",
userDataPath: "./userdata.yaml"
},
addStactoolsItemGenerator: true,
terminationProtection: false,
Expand Down
13 changes: 0 additions & 13 deletions cdk/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ export class Config {
readonly stacBrowserCertificateArn: string;
readonly pgstacVersion: string;
readonly webAclArn: string;
readonly bastionHostIpv4AllowList: string[];
readonly userStacItemGenRoleArn: string;
readonly userStacAllowedPublisherAccountBucketPairs: Array<{accountId: string; bucketArn: string}> | undefined;

Expand Down Expand Up @@ -117,18 +116,6 @@ export class Config {
process.env.TITILER_PGSTAC_API_CUSTOM_DOMAIN_NAME;
this.pgstacVersion = process.env.PGSTAC_VERSION!;
this.webAclArn = process.env.WEB_ACL_ARN!;

this.bastionHostIpv4AllowList = [];

// Parse IP config from environment variable
// Format: JSON with label-IP pairs
// Example: '{"office":"192.168.1.1", "vpn":"10.0.0.1"}'
if (process.env.BASTION_HOST_IPV4_ALLOW_LIST) {
const parsedConfig = JSON.parse(process.env.BASTION_HOST_IPV4_ALLOW_LIST);

this.bastionHostIpv4AllowList = Object.values(parsedConfig);
}

this.userStacItemGenRoleArn = process.env.USER_STAC_ITEM_GEN_ROLE_ARN!;

if (process.env.USER_STAC_ALLOWED_PUBLISHER_ACCOUNT_BUCKET_PAIRS) {
Expand Down
31 changes: 0 additions & 31 deletions test/config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,37 +93,6 @@ describe("Config", () => {
expect(() => new Config()).toThrow(/Must provide STAGE/);
});

test("parses JSON for bastionHostIpv4AllowList correctly", () => {
// Set JSON string for BASTION_HOST_IPV4_ALLOW_LIST
process.env.BASTION_HOST_IPV4_ALLOW_LIST =
'{"office":"192.168.1.1", "vpn":"10.0.0.1"}';

const config = new Config();

// Should parse the JSON and extract IPs
expect(config.bastionHostIpv4AllowList).toContain("192.168.1.1");
expect(config.bastionHostIpv4AllowList).toContain("10.0.0.1");
expect(config.bastionHostIpv4AllowList.length).toBe(2);
});

test("handles missing bastionHostIpv4AllowList by providing an empty array", () => {
// Make sure the env var doesn't exist
delete process.env.BASTION_HOST_IPV4_ALLOW_LIST;

const config = new Config();

// Should have an empty array when the env var is not provided
expect(config.bastionHostIpv4AllowList).toEqual([]);
});

test("throws error for invalid JSON in bastionHostIpv4AllowList", () => {
// Set invalid JSON string
process.env.BASTION_HOST_IPV4_ALLOW_LIST = "{invalid-json}";

// Should throw a SyntaxError when creating a new Config due to invalid JSON
expect(() => new Config()).toThrow(SyntaxError);
});

test("handles optional environment variables correctly", () => {
// Set optional environment variables
process.env.CERTIFICATE_ARN =
Expand Down