Skip to content

Add travis test for DockerHub images #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tomcli wants to merge 7 commits into from
Closed

Add travis test for DockerHub images #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e9f6d17
Add travis test for DockerHub images
Tomcli Mar 28, 2018
28fdd67
Merge branch 'master' into travis
Tomcli Mar 28, 2018
ee6249b
modify travis to tag pulled images as latest for helm install
Tomcli Mar 28, 2018
03cc57b
Merge branch 'master' into travis
Tomcli Apr 2, 2018
071e20a
Merge branch 'master' into travis
Tomcli Apr 9, 2018
f5acf3d
update script and makefile as suggested
Tomcli Apr 9, 2018
6d94dcf
update script and makefile as suggested
Tomcli Apr 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,12 @@ before_script:
# install aws CLI
- (sudo pip install -q awscli; which aws > /dev/null) &

script:
# fail fast
- set -e
- export MAKE_ARGS=--no-print-directory
# compile and build Docker images
- glide -q install
- make $MAKE_ARGS build
- make $MAKE_ARGS docker-build
# deploy services
- make $MAKE_ARGS deploy
# submit a test job
- make $MAKE_ARGS test-submit
jobs:
include:
- stage: test
script: ./etc/travis/test-images-local.sh
- if: type IN (push, api, cron)
script: ./etc/travis/test-images-dockerhub.sh

notifications:
email: false
23 changes: 23 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ UI_REPO = git@github.com:IBM/FfDL-dashboard.git
CLI_CMD = $(shell pwd)/cli/bin/ffdl-$(UNAME_SHORT)
CLUSTER_NAME ?= mycluster
PUBLIC_IP ?= 127.0.0.1
TRAVIS_IMAGE_VERSION ?= 0.0.1-master
TEST_IMAGES = $(addprefix $(DOCKER_NAMESPACE)/, $(TEST_IMAGES_SUFFIX))
TEST_IMAGES_SUFFIX = ffdl-lcm ffdl-trainer ffdl-metrics ffdl-databroker_s3 ffdl-ui ffdl-restapi ffdl-jobmonitor ffdl-controller tensorboard_extract_1.3-py3 log_collector ffdl-databroker_objectstorage tensorboard_extract


IMAGE_DIR := $(IMAGE_NAME)
ifneq ($(filter $(IMAGE_NAME),controller ),)
Expand Down Expand Up @@ -203,8 +207,27 @@ test-submit: ## Submit test training job
kubectl get pods | grep learner- | awk '{print $$1}' | xargs -I '{}' kubectl logs '{}' -c load-data; \
exit 1);

pull-dockerhub-images: ## Pull FfDL images from dockerhub
pull-dockerhub-images: $(addprefix pull-, $(TEST_IMAGES))

$(addprefix pull-, $(TEST_IMAGES)): pull-%: %
@TRAVIS_IMAGE=$< make .pull-dockerhub-images

$(TEST_IMAGES): ;

tag-dockerhub-images-to-latest: $(addprefix tag-, $(TEST_IMAGES))

$(addprefix tag-, $(TEST_IMAGES)): tag-%: %
@TRAVIS_IMAGE=$< make .tag-dockerhub-latest

# Helper targets

.pull-dockerhub-images:
docker pull $(TRAVIS_IMAGE):$(TRAVIS_IMAGE_VERSION)

.tag-dockerhub-latest:
docker tag $(TRAVIS_IMAGE):$(TRAVIS_IMAGE_VERSION) $(TRAVIS_IMAGE):latest

.build-service:
(cd ./$(SERVICE_NAME)/ && (test ! -e main.go || CGO_ENABLED=0 GOOS=linux go build -ldflags "-s -w" -a -installsuffix cgo -o bin/$(BINARY_NAME)))

Expand Down
12 changes: 12 additions & 0 deletions etc/travis/test-images-dockerhub.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash
# fail fast
Copy link
Contributor

@whummer whummer Apr 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given how we're calling these scripts from .travis.yml, do we need to add shebangs at the top of these scripts?

#!/bin/bash
...

set -e
export MAKE_ARGS=--no-print-directory
# pull images from dockerhub
make $MAKE_ARGS pull-dockerhub-images
make $MAKE_ARGS tag-dockerhub-images-to-latest
Copy link
Contributor

@whummer whummer Apr 7, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good workaround solution for now, but I guess at some point we should make the Docker tags parameterizable in the deployment script, so we don't need to overwrite latest tag here (as Animesh mentioned).


# deploy services
make $MAKE_ARGS deploy
# submit a test job
make $MAKE_ARGS test-submit
12 changes: 12 additions & 0 deletions etc/travis/test-images-local.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash
# fail fast
set -e
export MAKE_ARGS=--no-print-directory
# compile and build Docker images
glide -q install
make $MAKE_ARGS build
make $MAKE_ARGS docker-build
# deploy services
make $MAKE_ARGS deploy
# submit a test job
make $MAKE_ARGS test-submit