Mock AWS & Pytest Click

Published in

Analytics Vidhya

4 min readJun 1, 2020

As part of the DevOps team, I get to work on multiple different tools and applications. Recently, I had the opportunity to work on a click app in python which connects to AWS to get data. I am using setup.py build for the deployment(setup.py is a build script for setuptools. It tells setuptools about the package (such as the name and version) as well as which code files to include). A problem that I encountered was how to write test cases that will mock AWS calls. There is good documentation around each component like how to test click app, how to test setup.py, and how to do Mock AWS calls. In this blog, I am trying to combine all of them.

Where to start?

In this repo (link), I am putting a small subset of files for someone to try out how all these components work.

Main function:

get_value_from_paramter_store() in src/demo/manage-secrets.pyaccepts name of the parameter to get from SSM Parameter Store.

click cli :

src/demo/cli.py It accepts one option --name which is required. We can add more parameters as needed. I kept it one for understanding.

setup.py:

This took me some time to finalize, there are multiple ways to create this file.

Let's review it:

setup is part of setuptools package.
VERSION: There are many techniques to maintain a single source of truth for the version number of a project. I prefer it this way. You can also create a version file and specify it.
ROOT_PATH: Specifies the parent directory of the setup.py file. Sometimes, I face issues with the path.
You notice two print statements, this is just to ensure that the setup.py was working. They can be removed.
description and long_description: A short desc of your project, while long_description points to a README.md file.
package_dir: {"":"src"}, If you want to put modules in directories not named for their package, then you need to use the package_dir. All distutils packages are under src
packages: Include all packages under src
tests_require: If your project’s tests need one or more additional packages besides those needed to install it.
install_requires: A string or list of strings specifying what other distributions need to be installed when this one is. I have put a function to handle this.
entry_points: Entry points are a mechanism for an installed distribution to advertise components it provides to be discovered and used by other code. Basically, it specifies the command to run.

test_cli.py:

@pytest.fixture — The scope of this is limited to this module, available options are: function, class, module, package or session

As per documentation of moto, due to changes with botocore, it's recommended to specify dummy AWS credentials, so moto does not use actual AWS creds specified in the credentials file. Another recommendation is not to import the modules at the top if not needed.

In order to useget_ssm I need to do put_ssm as it does not contain existing data.

Note: response = runner.invoke(entrypoint,["get", "--name", “demo_parameter"]) , entrypoint is the name of the function which starts the cli, get and name are the parameters passed to the cli command.

How to run:

Once you have cloned the repo, run:

python setup.py develop to deploy in development mode.

Then run: demo and you should see the output below. I am using develop as it will create links and not install the actual packages, it really comes in handy when you are developing. Good explanation on when to use develop and install: here

demo
Usage: demo [OPTIONS] COMMAND [ARGS]...Options:
  --help  Show this message and exit.Commands:
  get  get secrets

How to run test(s):

There are multiple ways to run tests and multiple arguments are supported.pytest will run all tests: pytest -v -s -v for verbose and -s for printing any print statements specified in the tests.

Publish to PyPI:

Now if you need to publish your package to PyPI and share it with others. Create an account on TestPyPI and PyPI. Inside your virtualenv:

python3 -m pip install --user --upgrade setuptools wheel
pip install wheel
pip install twine

From the base of the repo:

python3 setup.py sdist bdist_wheel

Then to check with twine:

twine check dist/*
Checking dist/demo-0.0.1.dev1-py3-none-any.whl: PASSED
Checking dist/demo-0.0.1.dev1.tar.gz: PASSED

Upload:

python3 -m twine upload dist/* --verbose
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: demo
Enter your password: 
Uploading demo_python_aws_click-0.0.1.dev1-py3-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.99k/5.99k [00:02<00:00, 2.27kB/s]
Uploading demo-python-aws-click-0.0.1.dev1.tar.gz
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.63k/4.63k [00:01<00:00, 3.23kB/s]View at:
https://pypi.org/project/demo-python-aws-click/0.0.1.dev1/

How to test PyPI package:

Create a new virtualenv, download using pip and run the demo command:

- python3 -m venv ~/Documents/virtualenv/demo-venv
- source ~/Documents/virtualenv/demo-venv/bin/activate
- pip install demo-python-aws-click
- demo

Note:

Use of virtualenv is recommended
Tested package with Python 3.8
The package name must be unique when uploading to PyPI.

Conclusion:

Click is one of the most popular packages to create a command-line interface in python. When integrating multiple components it's sometimes hard to find out what fits where. I hope it helps. Feel free to correct or comment as needed.