diff --git a/README.md b/README.md index 0e4a781..2905c9e 100644 --- a/README.md +++ b/README.md @@ -1,273 +1,4 @@ Python 3.7 container image -=================== +========================= -This container image includes Python 3.7 as a [S2I](https://github.com/openshift/source-to-image) base image for your Python 3.7 applications. -Users can choose between RHEL and CentOS based builder images. -The RHEL images are available in the [Red Hat Container Catalog](https://access.redhat.com/containers/), -the CentOS images are available on [Docker Hub](https://hub.docker.com/r/centos/), -and the Fedora images are available in [Fedora Registry](https://registry.fedoraproject.org/). -The resulting image can be run using [podman](https://github.com/containers/libpod) or -[docker](http://docker.io). - -Note: while the examples in this README are calling `podman`, you can replace any such calls by `docker` with the same arguments - -Description ------------ - -Python 3.7 available as container is a base platform for -building and running various Python 3.7 applications and frameworks. -Python is an easy to learn, powerful programming language. It has efficient high-level -data structures and a simple but effective approach to object-oriented programming. -Python's elegant syntax and dynamic typing, together with its interpreted nature, -make it an ideal language for scripting and rapid application development in many areas -on most platforms. - -This container image includes an npm utility -(see [base image repository](https://github.com/sclorg/s2i-base-container/tree/master/base)), -so users can use it to install JavaScript -modules for their web applications. There is no guarantee for any specific npm or nodejs -version, that is included in the image; those versions can be changed anytime and -the nodejs itself is included just to make the npm work. - -Usage ---------------------- - -For this, we will assume that you are using the supported image, available via `python:3.7` imagestream tag in Openshift. -Building a simple [python-sample-app](https://github.com/sclorg/s2i-python-container/tree/master/3.7/test/setup-test-app) application -in Openshift can be achieved with the following step: - - ``` - oc new-app python:3.7~https://github.com/sclorg/s2i-python-container.git --context-dir=3.7/test/setup-test-app/ - ``` - -The same application can also be built using the standalone [S2I](https://github.com/openshift/source-to-image) application on systems that have it available: - - ``` - $ s2i build https://github.com/sclorg/s2i-python-container.git --context-dir=3.7/test/setup-test-app/ python-sample-app - ``` - -Where `` is the s2i-python image you [downloaded from RHEL, Centos or Fedora registry](../README.md#Download) or [built](../README.md#Build) from these sources. For example ubi8/python-36, centos/python-36-centos7 or f31/python3. - -**Accessing the application:** -``` -$ curl 127.0.0.1:8080 -``` - -Environment variables ---------------------- - -To set these environment variables, you can place them as a key value pair into a `.s2i/environment` -file inside your source code repository. - -* **APP_SCRIPT** - - Used to run the application from a script file. - This should be a path to a script file (defaults to `app.sh` unless set to null) that will be - run to start the application. - -* **APP_FILE** - - Used to run the application from a Python script. - This should be a path to a Python file (defaults to `app.py` unless set to null) that will be - passed to the Python interpreter to start the application. - -* **APP_MODULE** - - Used to run the application with Gunicorn, as documented - [here](http://docs.gunicorn.org/en/latest/run.html#gunicorn). - This variable specifies a WSGI callable with the pattern - `MODULE_NAME:VARIABLE_NAME`, where `MODULE_NAME` is the full dotted path - of a module, and `VARIABLE_NAME` refers to a WSGI callable inside the - specified module. - Gunicorn will look for a WSGI callable named `application` if not specified. - - If `APP_MODULE` is not provided, the `run` script will look for a `wsgi.py` - file in your project and use it if it exists. - - If using `setup.py` for installing the application, the `MODULE_NAME` part - can be read from there. For an example, see - [setup-test-app](https://github.com/sclorg/s2i-python-container/tree/master/3.7/test/setup-test-app). - -* **APP_HOME** - - This variable can be used to specify a sub-directory in which the application to be run is contained. - The directory pointed to by this variable needs to contain `wsgi.py` (for Gunicorn) or `manage.py` (for Django). - - If `APP_HOME` is not provided, the `assemble` and `run` scripts will use the application's root - directory. - -* **APP_CONFIG** - - Path to a valid Python file with a - [Gunicorn configuration](http://docs.gunicorn.org/en/latest/configure.html#configuration-file) file. - -* **DISABLE_MIGRATE** - - Set this variable to a non-empty value to inhibit the execution of 'manage.py migrate' - when the produced image is run. This only affects Django projects. See - "Handling Database Migrations" section of [Django blogpost on OpenShift blog]( - https://blog.openshift.com/migrating-django-applications-openshift-3/) on suggestions - how/when to run DB migrations in OpenShift environment. Most importantly, - note that running DB migrations from two or more pods might corrupt your database. - -* **DISABLE_COLLECTSTATIC** - - Set this variable to a non-empty value to inhibit the execution of - 'manage.py collectstatic' during the build. This only affects Django projects. - -* **DISABLE_SETUP_PY_PROCESSING** - - Set this to a non-empty value to skip processing of setup.py script if you - use `-e .` in requirements.txt to trigger its processing or you don't want - your application to be installed into site-packages directory. - -* **ENABLE_PIPENV** - - Set this variable to use [Pipenv](https://github.com/pypa/pipenv), - the higher-level Python packaging tool, to manage dependencies of the application. - This should be used only if your project contains properly formated Pipfile - and Pipfile.lock. -* **PIN_PIPENV_VERSION** - - Set this variable together with `ENABLE_PIPENV` to use a specific version of Pipenv. - If not set, the latest stable version from PyPI is installed. - For example `PIN_PIPENV_VERSION=2018.11.26` installs `pipenv==2018.11.26`. - -* **ENABLE_MICROPIPENV** - - Set this variable to use [micropipenv](https://github.com/thoth-station/micropipenv), - a lightweight wrapper for pip to support requirements.txt, Pipenv and Poetry lock - files or converting them to pip-tools compatible output. Designed for containerized Python applications. - Available only for Python 3 images. - -* **ENABLE_INIT_WRAPPER** - - Set this variable to a non-empty value to make use of an init wrapper. - This is useful for servers that are not capable of reaping zombie - processes, such as Django development server or Tornado. This option can - be used together with **APP_SCRIPT** or **APP_FILE**. It never applies - to Gunicorn used through **APP_MODULE** as Gunicorn reaps zombie - processes correctly. - -* **PIP_INDEX_URL** - - Set this variable to use a custom index URL or mirror to download required - packages during build process. This affects packages listed in - requirements.txt. It also affects the installation of pipenv and - micropipenv and the update of pip in the container, though if not found in - the custom index, the container will try to install/update them from - upstream PyPI afterwards. - -* **UPGRADE_PIP_TO_LATEST** - - Set this variable to a non-empty value to have the 'pip' program and related - python packages (setuptools and wheel) be upgraded to the most recent version - before any Python packages are installed. If not set it will use whatever - the default version is included by the platform for the Python version being used. - -* **WEB_CONCURRENCY** - - Set this to change the default setting for the number of - [workers](http://docs.gunicorn.org/en/stable/settings.html#workers). By - default, this is set to the number of available cores times 2, capped - at 12. - - -Source repository layout ------------------------- - -You do not need to change anything in your existing Python project's repository. -However, if these files exist they will affect the behavior of the build process: - -* **requirements.txt** - - List of dependencies to be installed with `pip`. The format is documented - [here](https://pip.pypa.io/en/latest/user_guide.html#requirements-files). - - -* **Pipfile** - - The replacement for requirements.txt, project is currently under active - design and development, as documented [here](https://github.com/pypa/pipfile). - Set `ENABLE_PIPENV` environment variable to true in order to process this file. - - -* **setup.py** - - Configures various aspects of the project, including installation of - dependencies, as documented - [here](https://packaging.python.org/en/latest/distributing.html#setup-py). - For most projects, it is sufficient to simply use `requirements.txt` or - `Pipfile`. Set `DISABLE_SETUP_PY_PROCESSING` environment variable to true - in order to skip processing of this file. - -Run strategies --------------- - -The container image produced by s2i-python executes your project in one of the -following ways, in precedence order: - -* **Gunicorn** - - The Gunicorn WSGI HTTP server is used to serve your application in the case that it - is installed. It can be installed by listing it either in the `requirements.txt` - file or in the `install_requires` section of the `setup.py` file. - - If a file named `wsgi.py` is present in your repository, it will be used as - the entry point to your application. This can be overridden with the - environment variable `APP_MODULE`. - This file is present in Django projects by default. - - If you have both Django and Gunicorn in your requirements, your Django project - will automatically be served using Gunicorn. - -* **Django development server** - - If you have Django in your requirements but don't have Gunicorn, then your - application will be served using Django's development web server. However, this is not - recommended for production environments. - -* **Python script** - - This would be used where you provide a Python code file for running you - application. It will be used in the case where you specify a path to a - Python script via the `APP_FILE` environment variable, defaulting to a - file named `app.py` if it exists. The script is passed to a regular - Python interpreter to launch your application. - -* **Application script file** - - This is the most general way of executing your application. It will be - used in the case where you specify a path to an executable script file - via the `APP_SCRIPT` environment variable, defaulting to a file named - `app.sh` if it exists. The script is executed directly to launch your - application. - -Hot deploy ---------------------- - -If you are using Django, hot deploy will work out of the box. - -To enable hot deploy while using Gunicorn, make sure you have a Gunicorn -configuration file inside your repository with the -[`reload`](https://gunicorn-docs.readthedocs.org/en/latest/settings.html#reload) -option set to `true`. Make sure to specify your config via the `APP_CONFIG` -environment variable. - -To change your source code in running container, use podman's (or docker's) -[exec](https://github.com/containers/libpod/blob/master/docs/podman-exec.1.md) command: - -``` -podman exec -it /bin/bash -``` - -After you enter into the running container, your current directory is set -to `/opt/app-root/src`, where the source code is located. - - -See also --------- -Dockerfile and other sources are available on https://github.com/sclorg/s2i-python-container. -In that repository you also can find another versions of Python environment Dockerfiles. -Dockerfile for CentOS is called `Dockerfile`, Dockerfile for RHEL7 is called `Dockerfile.rhel7`, -for RHEL8 it's `Dockerfile.rhel8` and the Fedora Dockerfile is called `Dockerfile.fedora`. +**The Python 3.7 image based on Fedora 31 is deprecated.** diff --git a/s2i/bin/usage b/s2i/bin/usage index b1c7448..df8c1a5 100755 --- a/s2i/bin/usage +++ b/s2i/bin/usage @@ -6,13 +6,13 @@ NAMESPACE=centos cat < -- curl 127.0.0.1:8080 EOF diff --git a/test/check_imagestreams.py b/test/check_imagestreams.py new file mode 100755 index 0000000..b0eb11b --- /dev/null +++ b/test/check_imagestreams.py @@ -0,0 +1,95 @@ +#!/bin/env python3 + +# MIT License +# +# Copyright (c) 2018-2019 Red Hat, Inc. + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +import sys +import json +import logging +import os + +from pathlib import Path +from typing import Dict + +IMAGESTREAMS_DIR: str = "imagestreams" + + +class ImageStreamChecker(object): + version: str = "" + results: Dict = {} + + def __init__(self, version: str): + self.version = version + + def load_json_file(self, filename: Path): + with open(str(filename)) as f: + return json.load(f) + + def check_version(self, json_dict: Dict): + res = [] + for tags in json_dict["spec"]["tags"]: + # The name can be"" or "-elX" or "-ubiX" + if (tags["name"] == self.version or + tags["name"].startswith(self.version + '-')): + res.append(tags) + return res + + def check_latest_tag(self, json_dict: Dict): + latest_tag_correct: bool = False + for tags in json_dict["spec"]["tags"]: + if tags["name"] != "latest": + continue + # The latest can link to either "" or "-elX" or "-ubiX" + if tags["from"]["name"] == self.version or tags["from"]["name"].startswith(self.version + '-'): + latest_tag_correct = True + return latest_tag_correct + + def check_imagestreams(self): + p = Path(".") + json_files = p.glob(f"{IMAGESTREAMS_DIR}/*.json") + if not json_files: + print(f"No json files present in {IMAGESTREAMS_DIR}.") + return 0 + for f in json_files: + if os.environ.get("TARGET") in ("rhel7", "centos7") and "aarch64" in str(f): + print("Imagestream aarch64 is not supported on rhel7") + continue + print(f"Checking file {str(f)}.") + json_dict = self.load_json_file(f) + if not (self.check_version(json_dict) and self.check_latest_tag(json_dict)): + print(f"The latest version is not present in {str(f)} or in latest tag.") + self.results[f] = False + if self.results: + return 1 + print("Imagestreams contains the latest version.") + return 0 + + +if __name__ == "__main__": + if len(sys.argv) != 2: + logging.fatal("%s: %s", sys.argv[0], "VERSION as an argument was not provided") + sys.exit(1) + + print(f"Version to check is {sys.argv[1]}.") + isc = ImageStreamChecker(version=sys.argv[1]) + sys.exit(isc.check_imagestreams()) + diff --git a/test/from-dockerfile/Dockerfile.tpl b/test/from-dockerfile/Dockerfile.tpl new file mode 100644 index 0000000..a44d41d --- /dev/null +++ b/test/from-dockerfile/Dockerfile.tpl @@ -0,0 +1,14 @@ +FROM #IMAGE_NAME# # Replaced by sed in tests, see test_from_dockerfile in test/run + +# Add application sources to a directory that the assemble script expects them +# and set permissions so that the container runs without root access +USER 0 +ADD app-src /tmp/src +RUN /usr/bin/fix-permissions /tmp/src +USER 1001 + +# Install the dependencies +RUN /usr/libexec/s2i/assemble + +# Set the default command for the resulting image +CMD /usr/libexec/s2i/run diff --git a/test/from-dockerfile/Dockerfile_no_s2i.tpl b/test/from-dockerfile/Dockerfile_no_s2i.tpl new file mode 100644 index 0000000..0b9bea7 --- /dev/null +++ b/test/from-dockerfile/Dockerfile_no_s2i.tpl @@ -0,0 +1,16 @@ +FROM #IMAGE_NAME# # Replaced by sed in tests, see test_from_dockerfile in test/run + +# Add application sources with correct permissions for OpenShift +USER 0 +ADD app-src . +RUN chown -R 1001:0 ./ +USER 1001 + +# Install the dependencies +RUN pip install -U "pip>=19.3.1{% if spec.version == "2.7" %},<21.0{% endif %}" && \ + pip install -r requirements.txt && \ + python manage.py collectstatic --noinput && \ + python manage.py migrate + +# Run the application +CMD python manage.py runserver 0.0.0.0:8080 diff --git a/test/imagestreams/python-centos.json b/test/imagestreams/python-centos.json new file mode 100644 index 0000000..1be8538 --- /dev/null +++ b/test/imagestreams/python-centos.json @@ -0,0 +1,173 @@ +{ + "kind": "ImageStream", + "apiVersion": "v1", + "metadata": { + "name": "python", + "annotations": { + "openshift.io/display-name": "Python" + } + }, + "spec": { + "tags": [ + { + "name": "latest", + "annotations": { + "openshift.io/display-name": "Python (Latest)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python applications on UBI. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.9/README.md.\n\nWARNING: By selecting this tag, your application will automatically update to use the latest version of Python available on OpenShift, including major version updates.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "ImageStreamTag", + "name": "3.9-ubi8" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.9-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.9 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.9 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.9/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.9,python", + "version": "3.9", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.access.redhat.com/ubi8/python-39:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.8-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.8 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.8 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.8/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.8,python", + "version": "3.8", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.access.redhat.com/ubi8/python-38:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.8-ubi7", + "annotations": { + "openshift.io/display-name": "Python 3.8 (UBI 7)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.8 applications on UBI 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.8/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.8,python", + "version": "3.8", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.access.redhat.com/ubi7/python-38:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.6-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.6 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.6 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.6/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.6,python", + "version": "3.6", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.access.redhat.com/ubi8/python-36:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "2.7-ubi8", + "annotations": { + "openshift.io/display-name": "Python 2.7 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 2.7 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/2.7/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:2.7,python", + "version": "2.7", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.access.redhat.com/ubi8/python-27:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "2.7-ubi7", + "annotations": { + "openshift.io/display-name": "Python 2.7 (UBI 7)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 2.7 applications on UBI 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/2.7/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:2.7,python", + "version": "2.7", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.access.redhat.com/ubi7/python-27:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "2.7", + "annotations": { + "openshift.io/display-name": "Python 2.7", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 2.7 applications on CentOS 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/2.7/README.md.", + "iconClass": "icon-python", + "tags": "builder,python,hidden", + "supports":"python:2.7,python", + "version": "2.7", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "quay.io/centos7/python-27-centos7:latest" + }, + "referencePolicy": { + "type": "Local" + } + } + ] + } +} diff --git a/test/imagestreams/python-rhel-aarch64.json b/test/imagestreams/python-rhel-aarch64.json new file mode 100644 index 0000000..a390935 --- /dev/null +++ b/test/imagestreams/python-rhel-aarch64.json @@ -0,0 +1,113 @@ +{ + "kind": "ImageStream", + "apiVersion": "v1", + "metadata": { + "name": "python", + "annotations": { + "openshift.io/display-name": "Python" + } + }, + "spec": { + "tags": [ + { + "name": "latest", + "annotations": { + "openshift.io/display-name": "Python (Latest)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python applications on UBI. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.9/README.md.\n\nWARNING: By selecting this tag, your application will automatically update to use the latest version of Python available on OpenShift, including major version updates.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "ImageStreamTag", + "name": "3.9-ubi8" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.9-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.9 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.9 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.9/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.9,python", + "version": "3.9", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-39:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.8-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.8 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.8 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.8/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.8,python", + "version": "3.8", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-38:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.6-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.6 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.6 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.6/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.6,python", + "version": "3.6", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-36:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "2.7-ubi8", + "annotations": { + "openshift.io/display-name": "Python 2.7 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 2.7 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/2.7/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:2.7,python", + "version": "2.7", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-27:latest" + }, + "referencePolicy": { + "type": "Local" + } + } + ] + } +} diff --git a/test/imagestreams/python-rhel.json b/test/imagestreams/python-rhel.json new file mode 100644 index 0000000..7522c80 --- /dev/null +++ b/test/imagestreams/python-rhel.json @@ -0,0 +1,193 @@ +{ + "kind": "ImageStream", + "apiVersion": "v1", + "metadata": { + "name": "python", + "annotations": { + "openshift.io/display-name": "Python" + } + }, + "spec": { + "tags": [ + { + "name": "latest", + "annotations": { + "openshift.io/display-name": "Python (Latest)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python applications on UBI. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.9/README.md.\n\nWARNING: By selecting this tag, your application will automatically update to use the latest version of Python available on OpenShift, including major version updates.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "ImageStreamTag", + "name": "3.9-ubi8" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.9-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.9 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.9 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.9/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.9,python", + "version": "3.9", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-39:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.8-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.8 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.8 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.8/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.8,python", + "version": "3.8", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-38:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.8-ubi7", + "annotations": { + "openshift.io/display-name": "Python 3.8 (UBI 7)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.8 applications on UBI 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.8/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.8,python", + "version": "3.8", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi7/python-38:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.8", + "annotations": { + "openshift.io/display-name": "Python 3.8", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.8 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.8/README.md.", + "iconClass": "icon-python", + "tags": "builder,python,hidden", + "supports":"python:3.8,python", + "version": "3.8", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/rhscl/python-38-rhel7:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "3.6-ubi8", + "annotations": { + "openshift.io/display-name": "Python 3.6 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 3.6 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/3.6/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:3.6,python", + "version": "3.6", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-36:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "2.7-ubi8", + "annotations": { + "openshift.io/display-name": "Python 2.7 (UBI 8)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 2.7 applications on UBI 8. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/2.7/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:2.7,python", + "version": "2.7", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi8/python-27:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "2.7-ubi7", + "annotations": { + "openshift.io/display-name": "Python 2.7 (UBI 7)", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 2.7 applications on UBI 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/2.7/README.md.", + "iconClass": "icon-python", + "tags": "builder,python", + "supports":"python:2.7,python", + "version": "2.7", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/ubi7/python-27:latest" + }, + "referencePolicy": { + "type": "Local" + } + }, + { + "name": "2.7", + "annotations": { + "openshift.io/display-name": "Python 2.7", + "openshift.io/provider-display-name": "Red Hat, Inc.", + "description": "Build and run Python 2.7 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-python-container/blob/master/2.7/README.md.", + "iconClass": "icon-python", + "tags": "builder,python,hidden", + "supports":"python:2.7,python", + "version": "2.7", + "sampleRepo": "https://github.com/sclorg/django-ex.git" + }, + "from": { + "kind": "DockerImage", + "name": "registry.redhat.io/rhscl/python-27-rhel7:latest" + }, + "referencePolicy": { + "type": "Local" + } + } + ] + } +} diff --git a/test/npm-virtualenv-uwsgi-test-app/app.sh b/test/npm-virtualenv-uwsgi-test-app/app.sh index 2659e06..5b7a273 100755 --- a/test/npm-virtualenv-uwsgi-test-app/app.sh +++ b/test/npm-virtualenv-uwsgi-test-app/app.sh @@ -13,7 +13,7 @@ packages=("pip" "setuptools" "wheel") for pkg in ${packages[@]}; do # grep returns exit code 1 if the output contains only one line starting # with "Requirement already …" which means that the package is updated - python -m pip install -U --no-python-version-warning $pkg 2>&1 | grep -v "^Requirement already up-to-date: " + python -m pip install -U --no-deps --no-python-version-warning $pkg 2>&1 | grep -Ev "^Requirement already (up-to-date|satisfied): " if [ $? -eq 0 ]; then echo "ERROR: Failed to upgrade '$pkg' to the latest version." exit 1 diff --git a/test/run b/test/run index d3829de..42a52d5 100755 --- a/test/run +++ b/test/run @@ -187,6 +187,12 @@ test_application() { cleanup_app } +test_from_dockerfile(){ + info "Test from Dockerfile" + sed "s@#IMAGE_NAME#@${IMAGE_NAME}@" $test_dir/from-dockerfile/Dockerfile.tpl > $test_dir/from-dockerfile/Dockerfile + ct_test_app_dockerfile $test_dir/from-dockerfile/Dockerfile 'https://github.com/sclorg/django-ex.git' 'Welcome to your Django application on OpenShift' app-src + check_result $? +} # Since we built the candidate image locally, we don't want S2I attempt to pull # it from Docker hub @@ -228,4 +234,20 @@ for app in ${@:-${WEB_APPS[@]}}; do cleanup ${app} done +if [ "$OS" == "rhel7" ] || [ "$OS" == "centos7" ]; then + # autocleanup only enabled here as only the following tests so far use it + CID_FILE_DIR=$(mktemp -d) + ct_enable_cleanup + + info "Testing variable presence during \`docker exec\`" + ct_check_exec_env_vars + check_result $? + + info "Checking if all scl variables are defined in Dockerfile" + ct_check_scl_enable_vars + check_result $? +fi + +test_from_dockerfile + info "All tests finished successfully." diff --git a/test/run-openshift b/test/run-openshift index 4a09812..76d36e8 100755 --- a/test/run-openshift +++ b/test/run-openshift @@ -7,6 +7,7 @@ # The image has to be available before this script is executed. THISDIR=$(dirname ${BASH_SOURCE[0]}) +test_dir="$(readlink -f $(dirname ${BASH_SOURCE[0]}))" source "${THISDIR}/test-lib.sh" source "${THISDIR}/test-lib-openshift.sh" @@ -16,6 +17,14 @@ set -eo nounset trap ct_os_cleanup EXIT SIGINT +test_latest_imagestreams() { + info "Testing the latest version in imagestreams" + # Switch to root directory of a container + pushd "${test_dir}/../.." >/dev/null + ct_check_latest_imagestreams + popd >/dev/null +} + ct_os_check_compulsory_vars ct_os_enable_print_logs @@ -37,13 +46,16 @@ for template in $EPHEMERAL_TEMPLATES; do "$template" \ python \ 'Welcome to your Django application on OpenShift' \ - 8080 http 200 "-p SOURCE_REPOSITORY_REF=master -p PYTHON_VERSION=${VERSION} -p POSTGRESQL_VERSION=9.6 -p NAME=python-testing" \ - "centos/postgresql-96-centos7|postgresql:9.6" + 8080 http 200 "-p SOURCE_REPOSITORY_REF=master -p PYTHON_VERSION=${VERSION} -p POSTGRESQL_VERSION=10 -p NAME=python-testing" \ + "quay.io/centos7/postgresql-10-centos7|postgresql:10" done # Check the imagestream test_python_imagestream +# check if latest imagestream version is correct +test_latest_imagestreams + OS_TESTSUITE_RESULT=0 ct_os_cluster_down diff --git a/test/run-openshift-remote-cluster b/test/run-openshift-remote-cluster new file mode 100755 index 0000000..b810752 --- /dev/null +++ b/test/run-openshift-remote-cluster @@ -0,0 +1,56 @@ +#!/bin/bash +# +# Test the Python image in the OpenShift. +# +# IMAGE_NAME specifies a name of the candidate image used for testing. +# VERSION specifies a version of the python in the candidate image. +# The image has to be available before this script is executed. + +THISDIR=$(dirname ${BASH_SOURCE[0]}) + +source "${THISDIR}/test-lib.sh" +source "${THISDIR}/test-lib-openshift.sh" +source "${THISDIR}/test-lib-python.sh" +source "${THISDIR}/test-lib-remote-openshift.sh" + +set -eo nounset +ct_os_set_ocp4 + + +trap ct_os_cleanup EXIT SIGINT + +ct_os_check_compulsory_vars + +oc status || false "It looks like oc is not properly logged in." + +# For testing on OpenShift 4 we use internal registry +export CT_EXTERNAL_REGISTRY=true + +test -n "${IMAGE_NAME-}" || false 'make sure $IMAGE_NAME is defined' +test -n "${VERSION-}" || false 'make sure $VERSION is defined' + +if [ -z "${EPHEMERAL_TEMPLATES:-}" ]; then + EPHEMERAL_TEMPLATES=" +https://raw.githubusercontent.com/sclorg/django-ex/master/openshift/templates/django-postgresql.json \ +https://raw.githubusercontent.com/openshift/origin/master/examples/quickstarts/django-postgresql.json" +fi + +ct_os_test_s2i_app "${IMAGE_NAME}" "https://github.com/sclorg/s2i-python-container.git" "examples/standalone-test-app" "Hello World from standalone WSGI application!" +ct_os_test_s2i_app "${IMAGE_NAME}" "https://github.com/sclorg/django-ex.git" . 'Welcome to your Django application on OpenShift' + +for template in $EPHEMERAL_TEMPLATES; do + ct_os_test_template_app "$IMAGE_NAME" \ + "$template" \ + python \ + 'Welcome to your Django application on OpenShift' \ + 8080 http 200 "-p SOURCE_REPOSITORY_REF=master -p PYTHON_VERSION=${VERSION} -p POSTGRESQL_VERSION=10 -p NAME=python-testing" \ + "rhscl/postgresql-10-rhel7|postgresql:10" +done + +# Check the imagestream +test_python_imagestream + +OS_TESTSUITE_RESULT=0 + +# vim: set tabstop=2:shiftwidth=2:expandtab: + diff --git a/test/test-lib-openshift.sh b/test/test-lib-openshift.sh index b92e950..1135ad7 100644 --- a/test/test-lib-openshift.sh +++ b/test/test-lib-openshift.sh @@ -145,7 +145,7 @@ function ct_os_get_pod_status() { # Arguments: pod_prefix - prefix or whole ID of the pod function ct_os_get_build_pod_status() { local pod_prefix="${1}" ; shift - local query="custom-columns=NAME:.metadata.name,Ready:status.containerStatuses[0].state.terminated.reason" + local query="custom-columns=NAME:.metadata.name,Ready:status.phase" oc get pods -o "$query" | grep -e "${pod_prefix}" | grep -E "\-build\s" \ | sort -u | awk '{print $2}' | tail -n 1 } @@ -193,7 +193,7 @@ function ct_os_wait_pod_ready() { if ct_os_get_all_pods_name | grep -E "${pod_prefix}.*-build"; then SECONDS=0 echo -n "Waiting for ${pod_prefix} build pod to finish ..." - while ! [ "$(ct_os_get_build_pod_status "${pod_prefix}")" == "Completed" ] ; do + while ! [ "$(ct_os_get_build_pod_status "${pod_prefix}")" == "Succeeded" ] ; do echo -n "." [ "${SECONDS}" -gt "${timeout}0" ] && echo " FAIL" && return 1 sleep 3 @@ -219,7 +219,7 @@ function ct_os_wait_rc_ready() { local pod_prefix="${1}" ; shift local timeout="${1}" ; shift SECONDS=0 - echo -n "Waiting for ${pod_prefix} pod becoming ready ..." + echo -n "Waiting for ${pod_prefix} having desired numbers of replicas ..." while ! test "$( (oc get --no-headers statefulsets; oc get --no-headers rc) 2>/dev/null \ | grep "^${pod_prefix}" | awk '$2==$3 {print "ready"}')" == "ready" ; do echo -n "." @@ -413,6 +413,7 @@ function ct_os_install_in_centos() { bash-completion origin-clients docker origin-clients } + # ct_os_cluster_up [DIR, IS_PUBLIC, CLUSTER_VERSION] # -------------------- # Runs the local OpenShift cluster using 'oc cluster up' and logs in as developer. @@ -604,8 +605,7 @@ function ct_os_test_s2i_app_func() { local check_command=${4} local oc_args=${5:-} local image_name_no_namespace=${image_name##*/} - local service_name="${image_name_no_namespace}-testing" - local image_tagged="${image_name_no_namespace}:${VERSION}" + local service_name="${image_name_no_namespace%%:*}-testing" local namespace if [ $# -lt 4 ] || [ -z "${1}" ] || [ -z "${2}" ] || [ -z "${3}" ] || [ -z "${4}" ]; then @@ -617,17 +617,22 @@ function ct_os_test_s2i_app_func() { ct_os_new_project namespace=${CT_NAMESPACE:-"$(oc project -q)"} + local image_tagged="${image_name_no_namespace%:*}:${VERSION}" - # Create a specific imagestream tag for the image so that oc cannot use anything else - if [ "${CT_SKIP_UPLOAD_IMAGE:-false}" == 'true' ] ; then - echo "Importing image ${image_name} as ${namespace}/${image_tagged}" - # Use --reference-policy=local to pull remote image content to the cluster - # Works around the issue of builder pods not having access to registry.redhat.io - oc tag --source=docker "${image_name}" "${namespace}/${image_tagged}" --insecure=true --reference-policy=local - ct_os_wait_stream_ready "${image_tagged}" "${namespace}" + if [ "${CT_EXTERNAL_REGISTRY:-false}" == 'true' ] ; then + ct_os_import_image_ocp4 "${image_name}" "${image_tagged}" else - echo "Uploading image ${image_name} as ${image_tagged}" - ct_os_upload_image "${image_name}" "${image_tagged}" + # Create a specific imagestream tag for the image so that oc cannot use anything else + if [ "${CT_SKIP_UPLOAD_IMAGE:-false}" == 'true' ] ; then + echo "Importing image ${image_name} as ${namespace}/${image_tagged}" + # Use --reference-policy=local to pull remote image content to the cluster + # Works around the issue of builder pods not having access to registry.redhat.io + oc tag --source=docker "${image_name}" "${namespace}/${image_tagged}" --insecure=true --reference-policy=local + ct_os_wait_stream_ready "${image_tagged}" "${namespace}" + else + echo "Uploading image ${image_name} as ${image_tagged}" + ct_os_upload_image "${image_name}" "${image_tagged}" + fi fi local app_param="${app}" @@ -752,27 +757,35 @@ function ct_os_test_template_app_func() { ct_os_new_project namespace=${CT_NAMESPACE:-"$(oc project -q)"} - # Create a specific imagestream tag for the image so that oc cannot use anything else - if [ "${CT_SKIP_UPLOAD_IMAGE:-false}" == 'true' ] ; then - echo "Importing image ${image_name} as ${image_tagged}" - # Use --reference-policy=local to pull remote image content to the cluster - # Works around the issue of builder pods not having access to registry.redhat.io - oc tag --source=docker "${image_name}" "${namespace}/${image_tagged}" --insecure=true --reference-policy=local - ct_os_wait_stream_ready "${image_tagged}" "${namespace}" + if [ "${CT_EXTERNAL_REGISTRY:-false}" == 'true' ] ; then + ct_os_import_image_ocp4 "${image_name}" "${image_tagged}" else - echo "Uploading image ${image_name} as ${image_tagged}" - ct_os_upload_image "${image_name}" "${image_tagged}" - - # upload also other images, that template might need (list of pairs in the format | - local image_tag_a - local i_t - for i_t in ${other_images} ; do - echo "${i_t}" - IFS='|' read -ra image_tag_a <<< "${i_t}" - docker pull "${image_tag_a[0]}" - ct_os_upload_image "${image_tag_a[0]}" "${image_tag_a[1]}" - done + if [ "${CT_SKIP_UPLOAD_IMAGE:-false}" == 'true' ] ; then + echo "Importing image ${image_name} as ${image_tagged}" + # Use --reference-policy=local to pull remote image content to the cluster + # Works around the issue of builder pods not having access to registry.redhat.io + oc tag --source=docker "${image_name}" "${namespace}/${image_tagged}" --insecure=true --reference-policy=local + ct_os_wait_stream_ready "${image_tagged}" "${namespace}" + else + echo "Uploading image ${image_name} as ${image_tagged}" + ct_os_upload_image "${image_name}" "${image_tagged}" + fi + fi + if [ "${CT_SKIP_UPLOAD_IMAGE:-false}" == 'false' ] ; then + # upload also other images, that template might need (list of pairs in the format | + local image_tag_a + local i_t + for i_t in ${other_images} ; do + echo "${i_t}" + IFS='|' read -ra image_tag_a <<< "${i_t}" + docker pull "${image_tag_a[0]}" + if [ "${CT_EXTERNAL_REGISTRY:-false}" == 'true' ] ; then + ct_os_import_image_ocp4 "${image_tag_a[0]}" "${image_tag_a[1]}" + else + ct_os_upload_image "${image_tag_a[0]}" "${image_tag_a[1]}" + fi + done fi # get the template file from remote or local location; if not found, it is @@ -781,6 +794,8 @@ function ct_os_test_template_app_func() { local local_template local_template=$(ct_obtain_input "${template}" 2>/dev/null || echo "--template=${template}") + + echo "Creating a new-app with name ${name_in_template} in namespace ${namespace} with args ${oc_args}." # shellcheck disable=SC2086 oc new-app "${local_template}" \ --name "${name_in_template}" \ @@ -916,7 +931,7 @@ ct_os_test_image_update() { function ct_os_deploy_cmd_image() { local image_name=${1} oc get pod command-app &>/dev/null && echo "command POD already running" && return 0 - echo "command POD not running yet, will start one called command-app" + echo "command POD not running yet, will start one called command-app ${image_name}" oc create -f - </bin directory, + +# Arguments: oc_version - X.Y part of the version of OSE (e.g. 3.9) +function ct_os_set_path_oc_4() { + echo "Setting OCP4 client" + local oc_version=$1 + local installed_oc_path="/usr/local/oc-v${oc_version}/bin" + echo "PATH ${installed_oc_path}" + if [ -x "${installed_oc_path}/oc" ] ; then + oc_path="${installed_oc_path}" + echo "Binary oc found in ${installed_oc_path}" >&2 + else + echo "OCP4 not found" + return 1 + fi + export PATH="${oc_path}:${PATH}" + oc version + if ! oc version | grep -q "Client Version: ${oc_version}." ; then + echo "ERROR: something went wrong, oc located at ${oc_path}, but oc of version ${oc_version} not found in PATH ($PATH)" >&1 + return 1 + else + echo "PATH set correctly, binary oc found in version ${oc_version}: $(command -v oc)" + fi +} + +# ct_os_prepare_ocp4 +# ------------------ +# Prepares environment for testing images in OpenShift 4 environment +# +# +function ct_os_set_ocp4() { + local login + OS_OC_CLIENT_VERSION=${OS_OC_CLIENT_VERSION:-4.4} + ct_os_set_path_oc_4 "${OS_OC_CLIENT_VERSION}" + + oc version + + login=$(cat "$KUBEPASSWORD") + oc login -u kubeadmin -p "$login" + echo "Login to OpenShift ${OS_OC_CLIENT_VERSION} is DONE" + # let openshift cluster to sync to avoid some race condition errors + sleep 3 +} + +function ct_os_upload_image_external_registry() { + local input_name="${1}" ; shift + local image_name=${input_name##*/} + local imagestream=${1:-$image_name:latest} + local output_name + + ct_os_login_external_registry + + output_name="${INTERNAL_DOCKER_REGISTRY}/rhscl-ci-testing/$imagestream" + + docker images + docker tag "${input_name}" "${output_name}" + docker push "${output_name}" +} + + +function ct_os_login_external_registry() { + local docker_token + # docker login fails with "404 page not found" error sometimes, just try it more times + # shellcheck disable=SC2034 + echo "loging" + [ -z "${INTERNAL_DOCKER_REGISTRY:-}" ] && "INTERNAL_DOCKER_REGISTRY has to be set for working with Internal registry" && return 1 + # shellcheck disable=SC2034 + for i in $(seq 12) ; do + # shellcheck disable=SC2015 + docker_token=$(cat "$DOCKER_UPSHIFT_TOKEN") + # shellcheck disable=SC2015 + docker login -u rhscl-ci-testing -p "$docker_token" "${INTERNAL_DOCKER_REGISTRY}" && return 0 || : + sleep 5 + done + return 1 +} + +function ct_os_import_image_ocp4() { + local image_name="${1}"; shift + local imagestream=${1:-$image_name:latest} + local namespace + + namespace=${CT_NAMESPACE:-"$(oc project -q)"} + deploy_image_name="${INTERNAL_DOCKER_REGISTRY}/rhscl-ci-testing/${imagestream}" + echo "Uploading image ${image_name} as ${deploy_image_name} , ${imagestream} into external registry." + ct_os_upload_image_external_registry "${image_name}" "${imagestream}" + if [ "${CT_TAG_IMAGE:-false}" == 'true' ]; then + echo "Tag ${deploy_image_name} to ${namespace}/${imagestream}" + oc tag --source=docker "${deploy_image_name}" "${namespace}/${imagestream}" --insecure=true --reference-policy=local + else + echo "Import image into OpenShift 4 environment ${namespace}/${imagestream} from ${deploy_image_name}" + oc import-image "${namespace}/${imagestream}" --from="${deploy_image_name}" --confirm --reference-policy=local + fi +} diff --git a/test/test-lib.sh b/test/test-lib.sh index 52bee15..cc046a9 100644 --- a/test/test-lib.sh +++ b/test/test-lib.sh @@ -25,7 +25,9 @@ EXPECTED_EXIT_CODE=0 # Uses: $CID_FILE_DIR - path to directory containing cid_files # Uses: $EXPECTED_EXIT_CODE - expected container exit code function ct_cleanup() { + ct_show_resources for cid_file in "$CID_FILE_DIR"/* ; do + [ -f "$cid_file" ] || continue local container container=$(cat "$cid_file") @@ -68,6 +70,7 @@ function ct_check_envs_set { loop_envs=$1; shift env_format=${1:-"*VALUE*"} while read -r variable; do + [ -z "$variable" ] && continue var_name=$(echo "$variable" | awk -F= '{ print $1 }') stripped=$(echo "$variable" | awk -F= '{ print $2 }') filtered_envs=$(echo "$check_envs" | grep "^$var_name=") @@ -81,11 +84,12 @@ function ct_check_envs_set { if [ -n "${filtered_envs##${env_format//VALUE/$value}}" ]; then echo " Value $value is missing from variable $var_name" echo "$filtered_envs" + IFS=$old_IFS return 1 fi done + IFS=$old_IFS done <<< "$(echo "$loop_envs" | grep "$env_filter" | grep -v "^PWD=")" - IFS=$old_IFS } # ct_get_cid [name] @@ -339,7 +343,7 @@ function ct_binary_found_from_df() { # Create Dockerfile that looks for the binary cat <"$tmpdir/Dockerfile" FROM $IMAGE_NAME -RUN which $binary | grep "$binary_path" +RUN command -v $binary | grep "$binary_path" EOF # Build an image, looking for expected path in the output if ! docker build -f "$tmpdir/Dockerfile" --no-cache "$tmpdir"; then @@ -557,7 +561,7 @@ ct_registry_from_os() { registry=registry.redhat.io ;; *) - registry=docker.io + registry=quay.io ;; esac echo "$registry" @@ -583,7 +587,9 @@ ct_get_public_image_name() { elif [ "x$os" == "xrhel8" ]; then public_image_name=$registry/rhel8/$base_image_name-${version//./} elif [ "x$os" == "xcentos7" ]; then - public_image_name=$registry/centos/$base_image_name-${version//./}-centos7 + public_image_name=$registry/centos7/$base_image_name-${version//./}-centos7 + elif [ "x$os" == "xcentos8" ]; then + public_image_name=$registry/centos8/$base_image_name-${version//./}-centos8 fi echo "$public_image_name" @@ -773,4 +779,135 @@ ct_check_image_availability() { fi } +# ct_check_latest_imagestreams +# ----------------------------- +# Check if the latest version present in Makefile in the variable VERSIONS +# is present in all imagestreams. +# Also the latest tag in the imagestreams has to contain the latest version +ct_check_latest_imagestreams() { + local latest_version= + local test_lib_dir= + + # We only maintain imagestreams for RHEL and CentOS (Community) + if [[ "$OS" =~ ^fedora.* ]] ; then + echo "Imagestreams for Fedora are not maintained, skipping ct_check_latest_imagestreams" + return 0 + fi + + # Check only lines which starts with VERSIONS + latest_version=$(grep '^VERSIONS' Makefile | rev | cut -d ' ' -f 1 | rev ) + # Fall back to previous version if the latest is excluded for this OS + [ -f "$latest_version/.exclude-$OS" ] && latest_version=$(grep '^VERSIONS' Makefile | rev | cut -d ' ' -f 2 | rev ) + # Only test the imagestream once, when the version matches + # ignore the SC warning, $VERSION is always available + # shellcheck disable=SC2153 + if [ "$latest_version" == "$VERSION" ]; then + test_lib_dir=$(dirname "$(readlink -f "$0")") + python3 "${test_lib_dir}/check_imagestreams.py" "$latest_version" + else + echo "Image version $VERSION is not latest, skipping ct_check_latest_imagestreams" + fi +} + +# ct_show_resources +# ---------------- +# Prints the available resources +ct_show_resources() +{ + echo "Resources info:" + echo "Memory:" + free -h + echo "Storage:" + df -h || : + echo "CPU" + lscpu +} + +# ct_test_app_dockerfile +# ----------------------------- +# Argument: dockerfile - path to a Dockerfile that will be used for building an image +# (must work with an application directory called 'app-src') +# Argument: app_url - git URI with a testing application, supports "@" to indicate a different branch +# Argument: body_regexp - PCRE regular expression that must match the response body +# Argument: app_dir - name of the application directory that is used in the Dockerfile +# Argument: port - Optional port number (default: 8080) +ct_test_app_dockerfile() { + local dockerfile=$1 + local app_url=$2 + local expected_text=$3 + local app_dir=$4 # this is a directory that must match with the name in the Dockerfile + local port=${5:-8080} + local app_image_name=myapp + local ret + local cname=app_dockerfile + + if [ -z "$app_dir" ] ; then + echo "ERROR: Option app_dir not set. Terminating the Dockerfile build." + return 1 + fi + + if ! [ -r "${dockerfile}" ] || ! [ -s "${dockerfile}" ] ; then + echo "ERROR: Dockerfile ${dockerfile} does not exist or is empty." + echo "Terminating the Dockerfile build." + return 1 + fi + + CID_FILE_DIR=${CID_FILE_DIR:-$(mktemp -d)} + local dockerfile_abs + dockerfile_abs=$(readlink -f "${dockerfile}") + tmpdir=$(mktemp -d) + pushd "$tmpdir" >/dev/null + cp "${dockerfile_abs}" Dockerfile + + # Rewrite the source image to what we test + sed -i -e "s|^FROM.*$|FROM $IMAGE_NAME|" Dockerfile + # a bit more verbose, but should help debugging failures + echo "Using this Dockerfile:" + cat Dockerfile + + # If app_url contains @, the string after @ is considered + # as a name of a branch to clone instead of the main/master branch + IFS='@' read -ra git_url_parts <<< "${app_url}" + + if [ -n "${git_url_parts[1]}" ]; then + git_clone_cmd="git clone --branch ${git_url_parts[1]} ${git_url_parts[0]} ${app_dir}" + else + git_clone_cmd="git clone ${app_url} ${app_dir}" + fi + + if ! $git_clone_cmd ; then + echo "ERROR: Git repository ${app_url} cannot be cloned into ${app_dir}." + echo "Terminating the Dockerfile build." + return 1 + fi + + echo "Building '${app_image_name}' image using docker build" + if ! docker build --no-cache=true -t "${app_image_name}" . ; then + echo "ERROR: The image cannot be built from ${dockerfile} and application ${app_url}." + echo "Terminating the Dockerfile build." + return 1 + fi + + if ! docker run -d --cidfile="${CID_FILE_DIR}/app_dockerfile" --rm "${app_image_name}" ; then + echo "ERROR: The image ${app_image_name} cannot be run for ${dockerfile} and application ${app_url}." + echo "Terminating the Dockerfile build." + return 1 + fi + echo "Waiting for ${app_image_name} to start" + ct_wait_for_cid "${CID_FILE_DIR}/app_dockerfile" + + ip="$(ct_get_cip "${cname}")" + ct_test_response "http://$ip:${port}" 200 "${expected_text}" + ret=$? + + # cleanup + docker kill "$(ct_get_cid "${cname}")" + sleep 2 + docker rmi "${app_image_name}" + popd >/dev/null + rm -rf "${tmpdir}" + rm -f "${CID_FILE_DIR}/${cname}" + return $ret +} + # vim: set tabstop=2:shiftwidth=2:expandtab: