Unit testing: design notes

Goal: to add unit testing functionality to Lambda Tools.

Proposed usage:

ltools test

It should have the same common arguments as ltools build.

Points of note:

ltools build deletes the bundle folder after creating the deployment package. This will be problematic if we want to run the tests. However, we need to be cautious about changing this behaviour.

I therefore propose the following:

• If a test section has not been defined, the bundle folder should be deleted as before.
• If a test section has been defined:
  • The bundle folder should NOT be deleted. However, the zip file should still be created.
  • Testing packages should only be added when ltools test is run.
  • An additional command, ltools cleanup should remove the bundle folder.
  • It should be possible to run ltools build twice in succession to make testing easier.
• An optional configuration setting, bundle, should be added to the build section specifying where to create the bundle.

Questions that we need to consider when creating the tests.

• Where should the tests be run? Locally, in a Docker container, or in a deployed lambda function on AWS?
• Can we run the tests locally without running ltools build first?

Proposed test section:

functions:
  my_function:
    # snip
    test:
      # Contains the test directory.
      # This will be copied into the package as a new subdirectory
      source: tests/hello-world
      # Requirements are specified identically to build requirements.
      requirements:
        - file: test-requirements.txt
      runner: py.test

Test runners to consider supporting:

• Nose: http://nose.readthedocs.io/en/latest/
• Nose2: http://nose2.readthedocs.io/en/latest/
• py.test: https://docs.pytest.org/en/latest/
• unittest/unittest2

Where should we run the tests?

• Locally, in the same environment as ltools?
• Locally, in a new virtualenv?
• In a docker container?
• On AWS itself, in a separate lambda function?

To do:

• Add test configuration section
• Add bundle configuration setting
• Use the specified bundle folder
• Don't delete the bundle folder if we have a test section
• Create ltools clean command to remove the bundle
• Create ltools test command to run the tests
• Copy the test dependencies into the bundle
• Deploy to test runner platform (see above)
• Run the tests