From 102882c6c2c816ab038dc16d2501f6ea0cc60af3 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Wed, 29 Jun 2022 22:06:30 +0530 Subject: [PATCH 01/24] Github Actions EKS provisioning --- .../setup-eks-provisioning-pipeline.asciidoc | 100 ++++++++++++++ .../common/install-ingress-controller.sh | 4 + .../github/templates/eks/eks-pipeline.cfg | 58 ++++++++ .../eks/eks-provisioning.yml.template | 128 ++++++++++++++++++ .../github/templates/eks/install-rancher.sh | 15 ++ .../github/templates/eks/obtain-dns.sh | 8 ++ 6 files changed, 313 insertions(+) create mode 100644 documentation/github/setup-eks-provisioning-pipeline.asciidoc create mode 100644 scripts/pipelines/github/templates/common/install-ingress-controller.sh create mode 100644 scripts/pipelines/github/templates/eks/eks-pipeline.cfg create mode 100644 scripts/pipelines/github/templates/eks/eks-provisioning.yml.template create mode 100644 scripts/pipelines/github/templates/eks/install-rancher.sh create mode 100644 scripts/pipelines/github/templates/eks/obtain-dns.sh diff --git a/documentation/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/github/setup-eks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..d761c4fda --- /dev/null +++ b/documentation/github/setup-eks-provisioning-pipeline.asciidoc @@ -0,0 +1,100 @@ +:toc: macro +toc::[] +:idprefix: +:idseparator: - + += Setting up the AWS EKS provisioning workflow +In this section we will create a workflow which will provision an AWS EKS cluster. This workflow will be configured to be manually triggered by the user. As part of EKS cluster provisioning, a NGINX Ingress controller is deployed and a .env file with the name `eks-variables` is created in .github folder, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently. + +The creation of the workflow will follow the project workflow, so a new branch named `feature/eks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropriate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/github/pipeline_generator.sh` will automatically create this new branch, create the EKS provisioning workflow based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +=== Prerequisites + +* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. + +* A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: +``` +aws s3 mb +# Example: aws s3 mb s3://terraformStateBucket +``` + +* An AWS IAM user with https://github.com/devonfw/hangar/blob/master/documentation/aws/setup-aws-account-iam-for-eks.asciidoc#check-iam-user-permissions[required permissions] to provision the EKS cluster. + +* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). + +== Creating the workflow using provided script + +Before executing the workflow generator, you will need to customize some input variables about the environment. Also, you may want to use existing VPC and subnets instead of creating new ones. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/aws/eks`, which allows you to create or update values for the required variables, passing them as flags. + +Example: creating a new VPC on cluster creation: + +``` +./set-terraform-variables.sh --region --instance_type --vpc_name --vpc_cidr_block +``` +Example: reusing existing VPC and subnets: +``` +./set-terraform-variables.sh --region --instance_type --existing_vpc_id --existing_vpc_private_subnets +``` +* Rancher is installed by default on the cluster after provisioning. If you wish to change this, please update `eks-pipeline.cfg` accordingly. + +=== Usage +``` +pipeline_generator.sh \ + -c \ + -n \ + -d \ + --cluster-name \ + --s3-bucket \ + --s3-key-path \ + [-b ] \ + [-w] +``` + +NOTE: The config file for the EKS provisioning workflow is located at `/scripts/pipelines/github/templates/eks/eks-pipeline.cfg`. + +=== Flags +``` +-c, --config-file [Required] Configuration file containing workflow definition. +-n, --pipeline-name [Required] Name that will be set to the workflow. +-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --cluster-name [Required] Name for the cluster." + --s3-bucket [Required] Name of the S3 bucket where the Terraform state of the cluster will be stored. + --s3-key-path [Required] Path within the S3 bucket where the Terraform state of the cluster will be stored. +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Example + +``` +./pipeline_generator.sh -c ./templates/eks/eks-pipeline.cfg -n eks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name hangar-eks-cluster --s3-bucket terraformStateBucket --s3-key-path eks/state -b develop -w +``` + +== Appendix: Interacting with the cluster + +First, generate a `kubeconfig` file for accessing the AWS EKS cluster: + +``` +aws eks update-kubeconfig --name --region +``` +Now you can use `kubectl` tool to communicate with the cluster. + +To enable an IAM user to connect to the EKS cluster, please refer https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html[here]. + +To get the DNS name of the NGINX Ingress controller on the EKS cluster, run the below command: +``` +kubectl get svc --namespace nginx-ingress nginx-ingress-nginx-ingress-controller -o jsonpath={.status.loadBalancer.ingress[0].hostname} +``` + +Rancher will be available on `https:///dashboard`. + +== Appendix: Rancher resources + +* https://rancher.com/docs/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/[Downloading `kubeconfig`]. +* https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/[RBAC] +* https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/[Monitoring] +* https://rancher.com/docs/rancher/v2.6/en/logging/[Logging] diff --git a/scripts/pipelines/github/templates/common/install-ingress-controller.sh b/scripts/pipelines/github/templates/common/install-ingress-controller.sh new file mode 100644 index 000000000..17dd4ac3c --- /dev/null +++ b/scripts/pipelines/github/templates/common/install-ingress-controller.sh @@ -0,0 +1,4 @@ +#!/bin/bash +helm repo add bitnami "https://charts.bitnami.com/bitnami" +helm repo update +helm upgrade --install nginx-ingress "nginx-ingress-controller" --set ingressClassResource.default=true --set containerSecurityContext.allowPrivilegeEscalation=false --repo "https://charts.bitnami.com/bitnami" --namespace nginx-ingress --create-namespace \ No newline at end of file diff --git a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg new file mode 100644 index 000000000..edb1913b4 --- /dev/null +++ b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg @@ -0,0 +1,58 @@ +# Mandatory flags. +mandatoryFlags="$pipelineName,$configFile,$localDirectory,$s3Bucket,$s3KeyPath,$clusterName," +# Path to the templates. +templatesPath="scripts/pipelines/github/templates/eks" +# YAML file name. +yamlFile="eks-provisioning.yml" +# Script name. +scriptFile="" +# Source branch. +sourceBranch="feature/eks-provisioning-demo" +# Path to terraform templates. +terraformTemplatesPath="scripts/environment-provisioning/aws/eks" +# Path to terraform scripts. +terraformPath=".terraform/eks" +# Installs Rancher on EKS cluster if set to true +installRancher=true +# AWS Region where to provision resources. +region=eu-west-1 + +# Function that copies the script to test the application. +function copyScript { + # Create .terraform/eks folder if it does not exist. + mkdir -p "${localDirectory}/${terraformPath}" + + # Copy the terraform files. + cd "${hangarPath}/${terraformTemplatesPath}" + cp * "${localDirectory}/${terraformPath}" + + # Copy the script for the DNS name into the directory. + cp "${hangarPath}/${commonTemplatesPath}/install-ingress-controller.sh" "${localDirectory}/${scriptFilePath}/install-ingress-controller.sh" + + # Copy the script to install rancher into the directory. + cp "${hangarPath}/${templatesPath}/install-rancher.sh" "${localDirectory}/${scriptFilePath}/install-rancher.sh" + + # Copy the script for the DNS name into the directory. + cp "${hangarPath}/${templatesPath}/obtain-dns.sh" "${localDirectory}/${scriptFilePath}/obtain-dns.sh" + +} + +function addPipelineVariables { + export installRancher="$installRancher" + export region="$region" + envsubst '${pipelineName} ${clusterName} ${s3Bucket} ${s3KeyPath} ${installRancher} ${region}' < "${hangarPath}/${templatesPath}/${yamlFile}.template" > "${localDirectory}/${pipelinePath}/${yamlFile}" +} + + +function commitFiles { + # Add the terraform files. + git add .terraform -f + + # Changing all files to be executable. + find .terraform -type f -name '*.sh' -exec git update-index --chmod=+x {} \; + + # Git commit and push it into the repository. + git commit -m "Adding the terraform files" + git push -u origin ${sourceBranch} +} + diff --git a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template new file mode 100644 index 000000000..165b7c702 --- /dev/null +++ b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template @@ -0,0 +1,128 @@ +name: $pipelineName + +on: + workflow_dispatch: + inputs: + cluster_name: + description: 'Name for the cluster.' + required: true + type: string + default: $clusterName + s3_bucket: + description: 'Name of the S3 bucket where the Terraform state of the cluster will be stored.' + required: true + type: string + default: $s3Bucket + s3_keypath: + description: 'Path within the S3 bucket where the Terraform state of the cluster will be stored.' + required: true + type: string + default: $s3KeyPath + aws_region: + description: 'AWS Region where to provision resources.' + required: true + type: string + default: $region + install_rancher: + description: 'Installs Rancher on EKS when set to true.' + required: true + type: string + default: $installRancher + cluster_operation: + description: 'Operation to be performed on EKS Cluster. create or destroy' + required: false + type: string + default: 'create' + +env: + TERRAFORM_WORKING_DIRECTORY: ".terraform/eks" + +jobs: + terraform-actions: + name: Provisioning EKS + runs-on: ubuntu-latest + steps: + - name: Checkout Repository + uses: actions/checkout@master + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v1 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: ${{ github.event.inputs.aws_region }} + + - name: Uninstall Helm charts + if: ${{ github.event.inputs.cluster_operation == 'destroy' }} + id: uninstallhelmcharts + run: | + aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }} + aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws eks update-kubeconfig --name ${{ github.event.inputs.cluster_name }} --region eu-west-1 + helm list --all-namespaces + helm ls -a --all-namespaces | awk 'NR > 1 { print "-n "$2, $1}' | xargs -L1 helm delete + echo 'LIST OF RELEASES AFTER HELM UNINSTALL..' + helm list --all-namespaces + + - name: Terraform Init + if: + id: init + working-directory: ${{ env.TERRAFORM_WORKING_DIRECTORY }} + run: terraform init -backend-config=bucket=${{ github.event.inputs.s3_bucket }} -backend-config=key=${{ github.event.inputs.s3_keypath }} -backend-config=region=${{ github.event.inputs.aws_region }} + continue-on-error: false + + - name: Terraform Apply + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: apply + working-directory: ${{ env.TERRAFORM_WORKING_DIRECTORY }} + run: terraform apply -var 'cluster_name=${{ github.event.inputs.cluster_name }}' --auto-approve + continue-on-error: false + + - name: Terraform Destroy + if: ${{ github.event.inputs.cluster_operation == 'destroy' }} + id: destroy + working-directory: ${{ env.TERRAFORM_WORKING_DIRECTORY }} + run: terraform apply -destroy -var 'cluster_name=${{ github.event.inputs.cluster_name }}' --auto-approve + continue-on-error: false + + - name: Install NGINX Ingress controller + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: installnginx + run: | + aws eks update-kubeconfig --name ${{ github.event.inputs.cluster_name }} --region ${{ github.event.inputs.aws_region }} + helm repo add bitnami "https://charts.bitnami.com/bitnami" + helm repo update + helm upgrade --install nginx-ingress "nginx-ingress-controller" --set ingressClassResource.default=true --set containerSecurityContext.allowPrivilegeEscalation=false --repo "https://charts.bitnami.com/bitnami" --namespace nginx-ingress --create-namespace + + - name: Obtain Ingress Controller domain name + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: obtaindns + run: | + aws eks update-kubeconfig --name ${{ github.event.inputs.cluster_name }} --region ${{ github.event.inputs.aws_region }} + dnsname=$(./modules/scripts/obtain-dns.sh) + echo "::set-output name=dnsname::$dnsname" + + - name: Publish Ingress Controller domain name + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: publishdns + run: | + cd .github + mkdir vars + cd vars + touch "eks_variables.env" + echo "cluster_name=${{ github.event.inputs.cluster_name }}" >> eks_variables.env + echo "dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> eks_variables.env + + - name: Push EKS variables + if: ${{ github.event.inputs.cluster_operation == 'create' }} + uses: EndBug/add-and-commit@v7 + with: + default_author: github_actions + cwd: '.github/vars/' + + - name: Install Rancher + id: installrancher + if: ${{ github.event.inputs.install_rancher == 'true' && github.event.inputs.cluster_operation == 'create'}} + run: | + ./modules/scripts/install-rancher.sh ${{ steps.obtaindns.outputs.dnsname }} + diff --git a/scripts/pipelines/github/templates/eks/install-rancher.sh b/scripts/pipelines/github/templates/eks/install-rancher.sh new file mode 100644 index 000000000..ab7a48e8b --- /dev/null +++ b/scripts/pipelines/github/templates/eks/install-rancher.sh @@ -0,0 +1,15 @@ +#!/bin/bash +helm repo add rancher-latest "https://releases.rancher.com/server-charts/latest" + +kubectl create namespace cattle-system + +kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml + +helm repo add jetstack https://charts.jetstack.io + +helm repo update + +# Install the cert-manager Helm chart +helm install cert-manager "jetstack/cert-manager" --namespace cert-manager --create-namespace --version v1.5.1 + +helm install rancher "rancher-latest/rancher" --namespace cattle-system --set hostname="$1" --set replicas=3 diff --git a/scripts/pipelines/github/templates/eks/obtain-dns.sh b/scripts/pipelines/github/templates/eks/obtain-dns.sh new file mode 100644 index 000000000..334feffef --- /dev/null +++ b/scripts/pipelines/github/templates/eks/obtain-dns.sh @@ -0,0 +1,8 @@ +#!/bin/bash +dnsName=$(kubectl get svc --namespace nginx-ingress nginx-ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') +while test -z "$dnsName" +do + sleep 5s + dnsName=$(kubectl get svc --namespace nginx-ingress nginx-ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') +done +echo $dnsName \ No newline at end of file From 5943b18edce227e85cbd1e988f27d3f6b65ab6f8 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Wed, 29 Jun 2022 22:12:19 +0530 Subject: [PATCH 02/24] Few edits --- .../github/templates/eks/eks-provisioning.yml.template | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template index 165b7c702..fa5714156 100644 --- a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template @@ -90,16 +90,14 @@ jobs: id: installnginx run: | aws eks update-kubeconfig --name ${{ github.event.inputs.cluster_name }} --region ${{ github.event.inputs.aws_region }} - helm repo add bitnami "https://charts.bitnami.com/bitnami" - helm repo update - helm upgrade --install nginx-ingress "nginx-ingress-controller" --set ingressClassResource.default=true --set containerSecurityContext.allowPrivilegeEscalation=false --repo "https://charts.bitnami.com/bitnami" --namespace nginx-ingress --create-namespace + ../common/install-ingress-controller.sh - name: Obtain Ingress Controller domain name if: ${{ github.event.inputs.cluster_operation == 'create' }} id: obtaindns run: | aws eks update-kubeconfig --name ${{ github.event.inputs.cluster_name }} --region ${{ github.event.inputs.aws_region }} - dnsname=$(./modules/scripts/obtain-dns.sh) + dnsname=$(./scripts/obtain-dns.sh) echo "::set-output name=dnsname::$dnsname" - name: Publish Ingress Controller domain name @@ -124,5 +122,5 @@ jobs: id: installrancher if: ${{ github.event.inputs.install_rancher == 'true' && github.event.inputs.cluster_operation == 'create'}} run: | - ./modules/scripts/install-rancher.sh ${{ steps.obtaindns.outputs.dnsname }} + ./scripts/install-rancher.sh ${{ steps.obtaindns.outputs.dnsname }} From f30147628fced87984228a75cce968246ce89da1 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Wed, 29 Jun 2022 22:18:21 +0530 Subject: [PATCH 03/24] Edits --- scripts/pipelines/github/templates/eks/eks-pipeline.cfg | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg index edb1913b4..c36f0fbd3 100644 --- a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg +++ b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg @@ -7,13 +7,15 @@ yamlFile="eks-provisioning.yml" # Script name. scriptFile="" # Source branch. -sourceBranch="feature/eks-provisioning-demo" +sourceBranch="feature/eks-provisioning" # Path to terraform templates. terraformTemplatesPath="scripts/environment-provisioning/aws/eks" # Path to terraform scripts. terraformPath=".terraform/eks" # Installs Rancher on EKS cluster if set to true -installRancher=true +if test -z ${installRancher} + installRancher=false +fi # AWS Region where to provision resources. region=eu-west-1 From 877773a6e6345b974dc2c8513195433952b7bdc8 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Thu, 7 Jul 2022 11:06:29 +0530 Subject: [PATCH 04/24] adsads adsads --- .../setup-eks-provisioning-pipeline.asciidoc | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc diff --git a/documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..f276c14b5 --- /dev/null +++ b/documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc @@ -0,0 +1,100 @@ +:toc: macro +toc::[] +:idprefix: +:idseparator: - + += Setting up the AWS EKS provisioning {pipeline_type} on {provider} +In this section we will create a {pipeline_type} which will provision an AWS EKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of EKS cluster provisioning, a NGINX Ingress controller is deployed and a .env file with the name `eks-variables` is created in .github folder, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently. + +The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/eks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropriate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the EKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +=== Prerequisites + +* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. + +* A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: +``` +aws s3 mb +# Example: aws s3 mb s3://terraformStateBucket +``` + +* An AWS IAM user with https://github.com/devonfw/hangar/blob/master/documentation/aws/setup-aws-account-iam-for-eks.asciidoc#check-iam-user-permissions[required permissions] to provision the EKS cluster. + +* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). + +== Creating the {pipeline_type} using provided script + +Before executing the workflow generator, you will need to customize some input variables about the environment. Also, you may want to use existing VPC and subnets instead of creating new ones. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/aws/eks`, which allows you to create or update values for the required variables, passing them as flags. + +Example: creating a new VPC on cluster creation: + +``` +./set-terraform-variables.sh --region --instance_type --vpc_name --vpc_cidr_block +``` +Example: reusing existing VPC and subnets: +``` +./set-terraform-variables.sh --region --instance_type --existing_vpc_id --existing_vpc_private_subnets +``` +* Rancher is installed by default on the cluster after provisioning. If you wish to change this, please update `eks-pipeline.cfg` accordingly. + +=== Usage +``` +pipeline_generator.sh \ + -c \ + -n \ + -d \ + --cluster-name \ + --s3-bucket \ + --s3-key-path \ + [-b ] \ + [-w] +``` + +NOTE: The config file for the EKS provisioning workflow is located at `/scripts/pipelines/{path_provider}/templates/eks/eks-pipeline.cfg`. + +=== Flags +``` +-c, --config-file [Required] Configuration file containing workflow definition. +-n, --pipeline-name [Required] Name that will be set to the workflow. +-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --cluster-name [Required] Name for the cluster." + --s3-bucket [Required] Name of the S3 bucket where the Terraform state of the cluster will be stored. + --s3-key-path [Required] Path within the S3 bucket where the Terraform state of the cluster will be stored. +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Example + +``` +./pipeline_generator.sh -c ./templates/eks/eks-pipeline.cfg -n eks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name hangar-eks-cluster --s3-bucket terraformStateBucket --s3-key-path eks/state -b develop -w +``` + +== Appendix: Interacting with the cluster + +First, generate a `kubeconfig` file for accessing the AWS EKS cluster: + +``` +aws eks update-kubeconfig --name --region +``` +Now you can use `kubectl` tool to communicate with the cluster. + +To enable an IAM user to connect to the EKS cluster, please refer https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html[here]. + +To get the DNS name of the NGINX Ingress controller on the EKS cluster, run the below command: +``` +kubectl get svc --namespace nginx-ingress nginx-ingress-nginx-ingress-controller -o jsonpath={.status.loadBalancer.ingress[0].hostname} +``` + +Rancher will be available on `https:///dashboard`. + +== Appendix: Rancher resources + +* https://rancher.com/docs/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/[Downloading `kubeconfig`]. +* https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/[RBAC] +* https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/[Monitoring] +* https://rancher.com/docs/rancher/v2.6/en/logging/[Logging] From bbedb76e310f96376b9467a170c34f0122cc060c Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Thu, 7 Jul 2022 11:07:47 +0530 Subject: [PATCH 05/24] Create setup-eks-provisioning-pipeline.asciidoc --- .../src/github/setup-eks-provisioning-pipeline.asciidoc | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 documentation/src/github/setup-eks-provisioning-pipeline.asciidoc diff --git a/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..8c11ab236 --- /dev/null +++ b/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc @@ -0,0 +1,4 @@ +:provider: GitHub +:pipeline_type: workflow +:path_provider: github +include::../common_templates/setup-eks-provisioning-pipeline.asciidoc[] From 3a6f08f467e23026a45d700219a0f9b5406d2b98 Mon Sep 17 00:00:00 2001 From: isandesh1986 Date: Thu, 7 Jul 2022 05:38:04 +0000 Subject: [PATCH 06/24] Automatic generation of documentation --- .../setup-eks-provisioning-pipeline.asciidoc | 203 +++++++++--------- 1 file changed, 103 insertions(+), 100 deletions(-) diff --git a/documentation/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/github/setup-eks-provisioning-pipeline.asciidoc index d761c4fda..870863d91 100644 --- a/documentation/github/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-eks-provisioning-pipeline.asciidoc @@ -1,100 +1,103 @@ -:toc: macro -toc::[] -:idprefix: -:idseparator: - - -= Setting up the AWS EKS provisioning workflow -In this section we will create a workflow which will provision an AWS EKS cluster. This workflow will be configured to be manually triggered by the user. As part of EKS cluster provisioning, a NGINX Ingress controller is deployed and a .env file with the name `eks-variables` is created in .github folder, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently. - -The creation of the workflow will follow the project workflow, so a new branch named `feature/eks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. - -Then, a Pull Request (PR) will be created in order to merge the new branch into the appropriate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. - -The script located at `/scripts/pipelines/github/pipeline_generator.sh` will automatically create this new branch, create the EKS provisioning workflow based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. - -=== Prerequisites - -* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. - -* A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: -``` -aws s3 mb -# Example: aws s3 mb s3://terraformStateBucket -``` - -* An AWS IAM user with https://github.com/devonfw/hangar/blob/master/documentation/aws/setup-aws-account-iam-for-eks.asciidoc#check-iam-user-permissions[required permissions] to provision the EKS cluster. - -* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). - -== Creating the workflow using provided script - -Before executing the workflow generator, you will need to customize some input variables about the environment. Also, you may want to use existing VPC and subnets instead of creating new ones. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/aws/eks`, which allows you to create or update values for the required variables, passing them as flags. - -Example: creating a new VPC on cluster creation: - -``` -./set-terraform-variables.sh --region --instance_type --vpc_name --vpc_cidr_block -``` -Example: reusing existing VPC and subnets: -``` -./set-terraform-variables.sh --region --instance_type --existing_vpc_id --existing_vpc_private_subnets -``` -* Rancher is installed by default on the cluster after provisioning. If you wish to change this, please update `eks-pipeline.cfg` accordingly. - -=== Usage -``` -pipeline_generator.sh \ - -c \ - -n \ - -d \ - --cluster-name \ - --s3-bucket \ - --s3-key-path \ - [-b ] \ - [-w] -``` - -NOTE: The config file for the EKS provisioning workflow is located at `/scripts/pipelines/github/templates/eks/eks-pipeline.cfg`. - -=== Flags -``` --c, --config-file [Required] Configuration file containing workflow definition. --n, --pipeline-name [Required] Name that will be set to the workflow. --d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - --cluster-name [Required] Name for the cluster." - --s3-bucket [Required] Name of the S3 bucket where the Terraform state of the cluster will be stored. - --s3-key-path [Required] Path within the S3 bucket where the Terraform state of the cluster will be stored. --b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. --w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. -``` - -=== Example - -``` -./pipeline_generator.sh -c ./templates/eks/eks-pipeline.cfg -n eks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name hangar-eks-cluster --s3-bucket terraformStateBucket --s3-key-path eks/state -b develop -w -``` - -== Appendix: Interacting with the cluster - -First, generate a `kubeconfig` file for accessing the AWS EKS cluster: - -``` -aws eks update-kubeconfig --name --region -``` -Now you can use `kubectl` tool to communicate with the cluster. - -To enable an IAM user to connect to the EKS cluster, please refer https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html[here]. - -To get the DNS name of the NGINX Ingress controller on the EKS cluster, run the below command: -``` -kubectl get svc --namespace nginx-ingress nginx-ingress-nginx-ingress-controller -o jsonpath={.status.loadBalancer.ingress[0].hostname} -``` - -Rancher will be available on `https:///dashboard`. - -== Appendix: Rancher resources - -* https://rancher.com/docs/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/[Downloading `kubeconfig`]. -* https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/[RBAC] -* https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/[Monitoring] -* https://rancher.com/docs/rancher/v2.6/en/logging/[Logging] +:provider: GitHub +:pipeline_type: workflow +:path_provider: github +:toc: macro +toc::[] +:idprefix: +:idseparator: - + += Setting up the AWS EKS provisioning {pipeline_type} on {provider} +In this section we will create a {pipeline_type} which will provision an AWS EKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of EKS cluster provisioning, a NGINX Ingress controller is deployed and a .env file with the name `eks-variables` is created in .github folder, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently. + +The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/eks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropriate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the EKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +=== Prerequisites + +* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. + +* A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: +``` +aws s3 mb +# Example: aws s3 mb s3://terraformStateBucket +``` + +* An AWS IAM user with https://github.com/devonfw/hangar/blob/master/documentation/aws/setup-aws-account-iam-for-eks.asciidoc#check-iam-user-permissions[required permissions] to provision the EKS cluster. + +* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). + +== Creating the {pipeline_type} using provided script + +Before executing the workflow generator, you will need to customize some input variables about the environment. Also, you may want to use existing VPC and subnets instead of creating new ones. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/aws/eks`, which allows you to create or update values for the required variables, passing them as flags. + +Example: creating a new VPC on cluster creation: + +``` +./set-terraform-variables.sh --region --instance_type --vpc_name --vpc_cidr_block +``` +Example: reusing existing VPC and subnets: +``` +./set-terraform-variables.sh --region --instance_type --existing_vpc_id --existing_vpc_private_subnets +``` +* Rancher is installed by default on the cluster after provisioning. If you wish to change this, please update `eks-pipeline.cfg` accordingly. + +=== Usage +``` +pipeline_generator.sh \ + -c \ + -n \ + -d \ + --cluster-name \ + --s3-bucket \ + --s3-key-path \ + [-b ] \ + [-w] +``` + +NOTE: The config file for the EKS provisioning workflow is located at `/scripts/pipelines/{path_provider}/templates/eks/eks-pipeline.cfg`. + +=== Flags +``` +-c, --config-file [Required] Configuration file containing workflow definition. +-n, --pipeline-name [Required] Name that will be set to the workflow. +-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --cluster-name [Required] Name for the cluster." + --s3-bucket [Required] Name of the S3 bucket where the Terraform state of the cluster will be stored. + --s3-key-path [Required] Path within the S3 bucket where the Terraform state of the cluster will be stored. +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Example + +``` +./pipeline_generator.sh -c ./templates/eks/eks-pipeline.cfg -n eks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name hangar-eks-cluster --s3-bucket terraformStateBucket --s3-key-path eks/state -b develop -w +``` + +== Appendix: Interacting with the cluster + +First, generate a `kubeconfig` file for accessing the AWS EKS cluster: + +``` +aws eks update-kubeconfig --name --region +``` +Now you can use `kubectl` tool to communicate with the cluster. + +To enable an IAM user to connect to the EKS cluster, please refer https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html[here]. + +To get the DNS name of the NGINX Ingress controller on the EKS cluster, run the below command: +``` +kubectl get svc --namespace nginx-ingress nginx-ingress-nginx-ingress-controller -o jsonpath={.status.loadBalancer.ingress[0].hostname} +``` + +Rancher will be available on `https:///dashboard`. + +== Appendix: Rancher resources + +* https://rancher.com/docs/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/[Downloading `kubeconfig`]. +* https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/[RBAC] +* https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/[Monitoring] +* https://rancher.com/docs/rancher/v2.6/en/logging/[Logging] From 9711450e845f0106bb4cb15e9e206aeb2d7327d4 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Thu, 7 Jul 2022 11:09:44 +0530 Subject: [PATCH 07/24] Create setup-eks-provisioning-pipeline.asciidoc --- .../src/azure-devops/setup-eks-provisioning-pipeline.asciidoc | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc diff --git a/documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..cb0a8e4ba --- /dev/null +++ b/documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc @@ -0,0 +1,4 @@ +:provider: Azure DevOps +:pipeline_type: pipeline +:path_provider: azure-devops +include::../common_templates/setup-eks-provisioning-pipeline.asciidoc[] From 2f53b3b3e891ecbbbccf336d5ccedf0577b8a53c Mon Sep 17 00:00:00 2001 From: isandesh1986 Date: Thu, 7 Jul 2022 05:40:05 +0000 Subject: [PATCH 08/24] Automatic generation of documentation --- .../setup-eks-provisioning-pipeline.asciidoc | 45 ++++++++----------- 1 file changed, 18 insertions(+), 27 deletions(-) diff --git a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc index a8fb9863c..deec87316 100644 --- a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc @@ -1,22 +1,23 @@ +:provider: Azure DevOps +:pipeline_type: pipeline +:path_provider: azure-devops :toc: macro toc::[] :idprefix: :idseparator: - -= Setting up a AWS EKS provisioning pipeline on Azure DevOps += Setting up the AWS EKS provisioning {pipeline_type} on {provider} +In this section we will create a {pipeline_type} which will provision an AWS EKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of EKS cluster provisioning, a NGINX Ingress controller is deployed and a .env file with the name `eks-variables` is created in .github folder, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently. -In this section we will create a pipeline which will provision an AWS EKS cluster. This pipeline will be configured to be manually triggered by the user. As part of EKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `eks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently. +The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/eks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/eks-provisioning` will be created, the YAML file for the pipeline and the terraform files for creating the cluster will be pushed to it. +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropriate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. -Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. - -The script located at `/scripts/pipelines/azure-devops/pipeline_generator.sh` will automatically create this new branch, create the EKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the EKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. === Prerequisites -* Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. -* Create a service connection for connecting to an AWS account (as explained in the above Terraform extension link) and name it `AWS-Terraform-Connection`. If you already have a service connection available or you need a specific connection name, please update `eks-pipeline.cfg` accordingly. +* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. * A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: ``` @@ -28,9 +29,9 @@ aws s3 mb * This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). -== Creating the pipeline using provided script +== Creating the {pipeline_type} using provided script -Before executing the pipeline generator, you will need to customize some input variables about the environment. Also, you may want to use existing VPC and subnets instead of creating new ones. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/aws/eks`, which allows you to create or update values for the required variables, passing them as flags. +Before executing the workflow generator, you will need to customize some input variables about the environment. Also, you may want to use existing VPC and subnets instead of creating new ones. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/aws/eks`, which allows you to create or update values for the required variables, passing them as flags. Example: creating a new VPC on cluster creation: @@ -41,6 +42,7 @@ Example: reusing existing VPC and subnets: ``` ./set-terraform-variables.sh --region --instance_type --existing_vpc_id --existing_vpc_private_subnets ``` +* Rancher is installed by default on the cluster after provisioning. If you wish to change this, please update `eks-pipeline.cfg` accordingly. === Usage ``` @@ -48,24 +50,23 @@ pipeline_generator.sh \ -c \ -n \ -d \ - --cluster-name \ + --cluster-name \ --s3-bucket \ --s3-key-path \ [-b ] \ [-w] ``` -NOTE: The config file for the EKS provisioning pipeline is located at `/scripts/pipelines/azure-devops/templates/eks/eks-pipeline.cfg`. +NOTE: The config file for the EKS provisioning workflow is located at `/scripts/pipelines/{path_provider}/templates/eks/eks-pipeline.cfg`. === Flags ``` --c, --config-file [Required] Configuration file containing pipeline definition. --n, --pipeline-name [Required] Name that will be set to the pipeline. +-c, --config-file [Required] Configuration file containing workflow definition. +-n, --pipeline-name [Required] Name that will be set to the workflow. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). --cluster-name [Required] Name for the cluster." --s3-bucket [Required] Name of the S3 bucket where the Terraform state of the cluster will be stored. --s3-key-path [Required] Path within the S3 bucket where the Terraform state of the cluster will be stored. - --rancher Install Rancher to manage the cluster. -b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. -w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. ``` @@ -73,11 +74,9 @@ NOTE: The config file for the EKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/eks/eks-pipeline.cfg -n eks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name hangar-eks-cluster --s3-bucket terraformStateBucket --s3-key-path eks/state --rancher -b develop -w +./pipeline_generator.sh -c ./templates/eks/eks-pipeline.cfg -n eks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name hangar-eks-cluster --s3-bucket terraformStateBucket --s3-key-path eks/state -b develop -w ``` -NOTE: Rancher is installed on the cluster after provisioning when using the above command. - == Appendix: Interacting with the cluster First, generate a `kubeconfig` file for accessing the AWS EKS cluster: @@ -94,15 +93,7 @@ To get the DNS name of the NGINX Ingress controller on the EKS cluster, run the kubectl get svc --namespace nginx-ingress nginx-ingress-nginx-ingress-controller -o jsonpath={.status.loadBalancer.ingress[0].hostname} ``` -Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: - -``` -kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' -``` - -== Appendix: Destroying the cluster - -To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. +Rancher will be available on `https:///dashboard`. == Appendix: Rancher resources From da83b8eeaa06f611a7dc1c565d67fd411fdc1dc0 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Thu, 7 Jul 2022 11:38:24 +0530 Subject: [PATCH 09/24] Update setup-eks-provisioning-pipeline.asciidoc --- .../src/github/setup-eks-provisioning-pipeline.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc index 8c11ab236..dfb017906 100644 --- a/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc @@ -1,4 +1,5 @@ :provider: GitHub :pipeline_type: workflow :path_provider: github +:trigger_sentence: This workflow will be configured to be executed inside a CI pipeline include::../common_templates/setup-eks-provisioning-pipeline.asciidoc[] From 522d28661cd109b7c0c3810b02f7bbe72201dbd9 Mon Sep 17 00:00:00 2001 From: isandesh1986 Date: Thu, 7 Jul 2022 06:08:47 +0000 Subject: [PATCH 10/24] Automatic generation of documentation --- documentation/github/setup-eks-provisioning-pipeline.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/github/setup-eks-provisioning-pipeline.asciidoc index 870863d91..4565f235d 100644 --- a/documentation/github/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-eks-provisioning-pipeline.asciidoc @@ -1,6 +1,7 @@ :provider: GitHub :pipeline_type: workflow :path_provider: github +:trigger_sentence: This workflow will be configured to be executed inside a CI pipeline :toc: macro toc::[] :idprefix: From bcbe0fea07ba962279ce01074ba4bcfcd69008dc Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Thu, 7 Jul 2022 11:39:45 +0530 Subject: [PATCH 11/24] Update setup-eks-provisioning-pipeline.asciidoc --- .../setup-eks-provisioning-pipeline.asciidoc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc index f276c14b5..190f7163a 100644 --- a/documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-eks-provisioning-pipeline.asciidoc @@ -14,7 +14,10 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` === Prerequisites -* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. +ifdef::trigger_sentence_github[ * Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository.] + +ifdef::trigger_sentence_azure[ * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.] +ifdef::trigger_sentence_azure[ * Create a service connection for connecting to an AWS account (as explained in the above Terraform extension link) and name it `AWS-Terraform-Connection`. If you already have a service connection available or you need a specific connection name, please update `eks-pipeline.cfg` accordingly.] * A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: ``` From a78fb0ef84886bcdd6b34905f8bb36b840fa05ea Mon Sep 17 00:00:00 2001 From: isandesh1986 Date: Thu, 7 Jul 2022 06:10:07 +0000 Subject: [PATCH 12/24] Automatic generation of documentation --- .../azure-devops/setup-eks-provisioning-pipeline.asciidoc | 2 +- documentation/github/setup-eks-provisioning-pipeline.asciidoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc index deec87316..5ce6a178d 100644 --- a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc @@ -17,7 +17,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` === Prerequisites -* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. + * A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: ``` diff --git a/documentation/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/github/setup-eks-provisioning-pipeline.asciidoc index 4565f235d..8e94633b0 100644 --- a/documentation/github/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-eks-provisioning-pipeline.asciidoc @@ -18,7 +18,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` === Prerequisites -* Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. + * A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: ``` From 21aeb3b8b17b9538896219eb28eed0acefd27657 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Thu, 7 Jul 2022 11:40:45 +0530 Subject: [PATCH 13/24] Update setup-eks-provisioning-pipeline.asciidoc --- .../src/github/setup-eks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc index dfb017906..b957d5b36 100644 --- a/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/src/github/setup-eks-provisioning-pipeline.asciidoc @@ -1,5 +1,5 @@ :provider: GitHub :pipeline_type: workflow :path_provider: github -:trigger_sentence: This workflow will be configured to be executed inside a CI pipeline +:trigger_sentence_github: This workflow will be configured to be executed inside a CI pipeline include::../common_templates/setup-eks-provisioning-pipeline.asciidoc[] From e3e21904ec7c2a7476e4bc757c59c1399c81c5f0 Mon Sep 17 00:00:00 2001 From: isandesh1986 Date: Thu, 7 Jul 2022 06:11:07 +0000 Subject: [PATCH 14/24] Automatic generation of documentation --- documentation/github/setup-eks-provisioning-pipeline.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/documentation/github/setup-eks-provisioning-pipeline.asciidoc b/documentation/github/setup-eks-provisioning-pipeline.asciidoc index 8e94633b0..1851d14da 100644 --- a/documentation/github/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-eks-provisioning-pipeline.asciidoc @@ -1,7 +1,7 @@ :provider: GitHub :pipeline_type: workflow :path_provider: github -:trigger_sentence: This workflow will be configured to be executed inside a CI pipeline +:trigger_sentence_github: This workflow will be configured to be executed inside a CI pipeline :toc: macro toc::[] :idprefix: @@ -18,6 +18,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` === Prerequisites + * Add AWS credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository. * A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: From c6f91e2075953e1d816c8d043f1d8707db87a489 Mon Sep 17 00:00:00 2001 From: "Sandesh, Pallapati Immanuel Prabhu" Date: Thu, 7 Jul 2022 11:41:27 +0530 Subject: [PATCH 15/24] Update setup-eks-provisioning-pipeline.asciidoc --- .../src/azure-devops/setup-eks-provisioning-pipeline.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc index cb0a8e4ba..02eb49794 100644 --- a/documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/src/azure-devops/setup-eks-provisioning-pipeline.asciidoc @@ -1,4 +1,5 @@ :provider: Azure DevOps :pipeline_type: pipeline :path_provider: azure-devops +:trigger_sentence_azure: This workflow will be configured to be executed inside a CI pipeline include::../common_templates/setup-eks-provisioning-pipeline.asciidoc[] From c45bfaa45adbcfdc4588f5adab3fef74db73ede8 Mon Sep 17 00:00:00 2001 From: isandesh1986 Date: Thu, 7 Jul 2022 06:11:55 +0000 Subject: [PATCH 16/24] Automatic generation of documentation --- .../azure-devops/setup-eks-provisioning-pipeline.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc index 5ce6a178d..776e1fa9a 100644 --- a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc @@ -1,6 +1,7 @@ :provider: Azure DevOps :pipeline_type: pipeline :path_provider: azure-devops +:trigger_sentence_azure: This workflow will be configured to be executed inside a CI pipeline :toc: macro toc::[] :idprefix: @@ -18,6 +19,8 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` === Prerequisites + * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. + * Create a service connection for connecting to an AWS account (as explained in the above Terraform extension link) and name it `AWS-Terraform-Connection`. If you already have a service connection available or you need a specific connection name, please update `eks-pipeline.cfg` accordingly. * A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: ``` From c515838000e1458e935efb96e3653c752ebef123 Mon Sep 17 00:00:00 2001 From: ultymatom <90399827+ultymatom@users.noreply.github.com> Date: Wed, 20 Jul 2022 09:31:34 +0200 Subject: [PATCH 17/24] Update eks-pipeline.cfg --- scripts/pipelines/github/templates/eks/eks-pipeline.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg index c36f0fbd3..0b98c3cd8 100644 --- a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg +++ b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg @@ -14,6 +14,7 @@ terraformTemplatesPath="scripts/environment-provisioning/aws/eks" terraformPath=".terraform/eks" # Installs Rancher on EKS cluster if set to true if test -z ${installRancher} +then installRancher=false fi # AWS Region where to provision resources. From d5bac0aa10cb308ebce2afa91af9717f45ea22f7 Mon Sep 17 00:00:00 2001 From: ultymatom <90399827+ultymatom@users.noreply.github.com> Date: Fri, 5 Aug 2022 21:13:17 +0200 Subject: [PATCH 18/24] Update eks-pipeline.cfg --- scripts/pipelines/github/templates/eks/eks-pipeline.cfg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg index 0b98c3cd8..5d2cb410d 100644 --- a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg +++ b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg @@ -16,6 +16,8 @@ terraformPath=".terraform/eks" if test -z ${installRancher} then installRancher=false +else + installRancher=true fi # AWS Region where to provision resources. region=eu-west-1 From 4177209d69ff8c3643cc13d5ce2f1c5580999ffa Mon Sep 17 00:00:00 2001 From: ultymatom <90399827+ultymatom@users.noreply.github.com> Date: Fri, 5 Aug 2022 21:14:20 +0200 Subject: [PATCH 19/24] Update eks-pipeline.cfg --- scripts/pipelines/github/templates/eks/eks-pipeline.cfg | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg index 5d2cb410d..b3f932e12 100644 --- a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg +++ b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg @@ -43,9 +43,12 @@ function copyScript { } function addPipelineVariables { - export installRancher="$installRancher" - export region="$region" - envsubst '${pipelineName} ${clusterName} ${s3Bucket} ${s3KeyPath} ${installRancher} ${region}' < "${hangarPath}/${templatesPath}/${yamlFile}.template" > "${localDirectory}/${pipelinePath}/${yamlFile}" + export installRancher + export region + export s3Bucket + export s3KeyPath + export clusterName + specificEnvSubstList='${clusterName} ${s3Bucket} ${s3KeyPath} ${installRancher} ${region}' } From af5913d385200d66ee387699d2ab23ab2cc425b6 Mon Sep 17 00:00:00 2001 From: ultymatom <90399827+ultymatom@users.noreply.github.com> Date: Fri, 5 Aug 2022 21:17:12 +0200 Subject: [PATCH 20/24] Update eks-provisioning.yml.template --- .../github/templates/eks/eks-provisioning.yml.template | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template index fa5714156..9709bb2a4 100644 --- a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template @@ -27,7 +27,7 @@ on: description: 'Installs Rancher on EKS when set to true.' required: true type: string - default: $installRancher + default: '$installRancher' cluster_operation: description: 'Operation to be performed on EKS Cluster. create or destroy' required: false @@ -65,7 +65,6 @@ jobs: helm list --all-namespaces - name: Terraform Init - if: id: init working-directory: ${{ env.TERRAFORM_WORKING_DIRECTORY }} run: terraform init -backend-config=bucket=${{ github.event.inputs.s3_bucket }} -backend-config=key=${{ github.event.inputs.s3_keypath }} -backend-config=region=${{ github.event.inputs.aws_region }} @@ -90,14 +89,14 @@ jobs: id: installnginx run: | aws eks update-kubeconfig --name ${{ github.event.inputs.cluster_name }} --region ${{ github.event.inputs.aws_region }} - ../common/install-ingress-controller.sh + .github/workflows/scripts/install-ingress-controller.sh - name: Obtain Ingress Controller domain name if: ${{ github.event.inputs.cluster_operation == 'create' }} id: obtaindns run: | aws eks update-kubeconfig --name ${{ github.event.inputs.cluster_name }} --region ${{ github.event.inputs.aws_region }} - dnsname=$(./scripts/obtain-dns.sh) + dnsname=$(.github/workflows/scripts/obtain-dns.sh) echo "::set-output name=dnsname::$dnsname" - name: Publish Ingress Controller domain name @@ -122,5 +121,5 @@ jobs: id: installrancher if: ${{ github.event.inputs.install_rancher == 'true' && github.event.inputs.cluster_operation == 'create'}} run: | - ./scripts/install-rancher.sh ${{ steps.obtaindns.outputs.dnsname }} + .github/workflows/scripts/install-rancher.sh ${{ steps.obtaindns.outputs.dnsname }} From ab11b2a590eaf832f24ae044a360aa5e27cb3569 Mon Sep 17 00:00:00 2001 From: ultymatom Date: Mon, 22 Aug 2022 10:45:50 +0000 Subject: [PATCH 21/24] Automatic generation of documentation --- .../azure-devops/setup-eks-provisioning-pipeline.asciidoc | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc index 0b9f963f0..776e1fa9a 100644 --- a/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-eks-provisioning-pipeline.asciidoc @@ -25,7 +25,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` * A S3 Bucket. You can use an existing one or https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-buckets-creating[create a new one] with the following command: ``` aws s3 mb -# Example: aws s3 mb s3://terraform-state-bucket +# Example: aws s3 mb s3://terraformStateBucket ``` * An AWS IAM user with https://github.com/devonfw/hangar/blob/master/documentation/aws/setup-aws-account-iam-for-eks.asciidoc#check-iam-user-permissions[required permissions] to provision the EKS cluster. @@ -56,10 +56,6 @@ pipeline_generator.sh \ --cluster-name \ --s3-bucket \ --s3-key-path \ - [--aws-access-key ] \ - [--aws-secret-access-key ] \ - [--aws-region ] \ - [--rancher] \ [-b ] \ [-w] ``` From 8db1caf0ce37530a6cfba5d154182ad9b7e7fbad Mon Sep 17 00:00:00 2001 From: ultymatom <90399827+ultymatom@users.noreply.github.com> Date: Tue, 23 Aug 2022 08:52:19 +0200 Subject: [PATCH 22/24] Update eks-pipeline.cfg --- scripts/pipelines/github/templates/eks/eks-pipeline.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg index b3f932e12..18d25ec98 100644 --- a/scripts/pipelines/github/templates/eks/eks-pipeline.cfg +++ b/scripts/pipelines/github/templates/eks/eks-pipeline.cfg @@ -60,7 +60,7 @@ function commitFiles { find .terraform -type f -name '*.sh' -exec git update-index --chmod=+x {} \; # Git commit and push it into the repository. - git commit -m "Adding the terraform files" + git commit -m "Adding the terraform files [skip ci]" git push -u origin ${sourceBranch} } From 3a294488c69590a951207467c8b3fc8296180c56 Mon Sep 17 00:00:00 2001 From: ultymatom <90399827+ultymatom@users.noreply.github.com> Date: Tue, 23 Aug 2022 10:04:18 +0200 Subject: [PATCH 23/24] adding -p when creating var folder --- .../github/templates/eks/eks-provisioning.yml.template | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template index 9709bb2a4..4f909b638 100644 --- a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template @@ -104,7 +104,7 @@ jobs: id: publishdns run: | cd .github - mkdir vars + mkdir -p vars cd vars touch "eks_variables.env" echo "cluster_name=${{ github.event.inputs.cluster_name }}" >> eks_variables.env From fc50959270064796aaf6f815241f81207719f983 Mon Sep 17 00:00:00 2001 From: ultymatom <90399827+ultymatom@users.noreply.github.com> Date: Thu, 25 Aug 2022 11:14:12 +0200 Subject: [PATCH 24/24] Update eks-provisioning.yml.template --- .../github/templates/eks/eks-provisioning.yml.template | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template index 4f909b638..5369f481d 100644 --- a/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/eks/eks-provisioning.yml.template @@ -108,7 +108,7 @@ jobs: cd vars touch "eks_variables.env" echo "cluster_name=${{ github.event.inputs.cluster_name }}" >> eks_variables.env - echo "dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> eks_variables.env + echo "eks_dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> eks_variables.env - name: Push EKS variables if: ${{ github.event.inputs.cluster_operation == 'create' }}