diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..55002be --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +/mrack-1.11.0.tar.gz +/mrack-1.12.0.tar.gz +/mrack-1.12.1.tar.gz +/mrack-1.12.2.tar.gz diff --git a/.packit.yaml b/.packit.yaml new file mode 100644 index 0000000..1fe5c0d --- /dev/null +++ b/.packit.yaml @@ -0,0 +1,51 @@ +--- +# See the documentation for more information: +# https://packit.dev/docs/configuration/ +specfile_path: mrack.spec + +# add or remove files that should be synced +files_to_sync: + - mrack.spec + - README.md + - .packit.yaml + +# name in upstream package repository/registry (e.g. in PyPI) +upstream_package_name: mrack +# downstream (Fedora) RPM package name +downstream_package_name: mrack +# mrack uses tag v(ver_number) and we need to make sure packit know what to expect +# see: https://packit.dev/docs/configuration/#upstream_tag_template +upstream_tag_template: "v{version}" + +actions: + create-archive: + - "python3 setup.py sdist --dist-dir ." + - "sh -c 'echo mrack-$(python3 setup.py --version).tar.gz'" + get-current-version: + - "python3 setup.py --version" + +jobs: + - job: copr_build + trigger: pull_request + targets: [fedora-all] + + - job: copr_build + trigger: release + owner: "@freeipa" + project: neoave + targets: [fedora-all] + + - job: propose_downstream + trigger: release + dist_git_branches: + - fedora-all + + - job: koji_build + trigger: commit + dist_git_branches: + - fedora-all + + - job: bodhi_update + trigger: commit + dist_git_branches: + - fedora-branched # rawhide updates are created automatically diff --git a/README.md b/README.md index 3b5b028..2895974 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,304 @@ # mrack -The mrack package +![pypi_badge](https://img.shields.io/pypi/v/mrack?label=PyPI&logo=pypi) ![readthedocs_badge](https://img.shields.io/readthedocs/mrack?label=Read%20the%20Docs&logo=read-the-docs) ![badge](https://copr.fedorainfracloud.org/coprs/g/freeipa/neoave/package/mrack/status_image/last_build.png) + +Provisioning library for CI and local multi-host testing supporting multiple +provisioning providers e.g. OpenStack, libvirt, containers, Beaker). + +But in comparison to multi-cloud libraries, the aim is to be able to describe +host from application perspective. E.g.: + +```yaml +network: IPv4 +hosts: +- name: master.testdomain.test + role: master + group: ipaserver + os: fedora-31 +- name: client.testdomain.test + role: client + group: ipaclient + os: fedora-30 +``` + +This is then combined with provisioning configuration where each provider has +defined meaning for `role`, `group`, `os` params which e.g. translates to +flavors, images, ... + +```yaml +provider: openstack # default provider +openstack: + images: + fedora-30: Fedora-Cloud-Base-30-compose-latest + fedora-31: Fedora-Cloud-Base-31 + flavors: + ipaserver: ci.m1.medium + ipaclient: ci.m1.micro + networks: + IPv4: net_ci_4 + IPv6: net_ipv6_only + dual: net_cci_4_6 + keypair: ipa_key +beaker: + distros: + fedora-30: FEDORA-30% + fodora-31: FEDORA-31% +``` + +So then user can quickly change provisioning location just by changing provider +name without needing to redefine the provider/cloud specifics. This is +especially useful for a lot of jobs sharing the same app specific provisioner +configuration. + +```yaml +provider: beaker +``` + +Or each host can use a different provider: + +```yaml +hosts: +- name: master.testdomain.test + role: master + group: ipaserver + os: fedora-31 + provider: openstack + network: dual +- name: client.testdomain.test + role: client + group: ipaclient + os: fedora-30 + provider: beaker +``` + +The output is then Ansible inventory with correct group mapping and information +which allows to SSH to the machines. + +```yaml +all: + children: + ipaserver: + hosts: + f30-1.mrack.test: {} + f33-2.mrack.test: {} + hosts: + f30-1.mrack.test: + ansible_host: 10.0.154.21 + ansible_python_interpreter: /usr/bin/python3 + ansible_ssh_private_key_file: config/id_rsa + ansible_user: fedora + meta_dc_record: DC=mrack,DC=test + meta_domain: mrack.test + meta_fqdn: f30-1.mrack.test + meta_ip: 10.0.154.21 + meta_os: fedora-30 + meta_provider_id: 7c3c28f9-4674-4f7f-b413-00bc0b00d711 + meta_restraint_id: 9 + meta_role: master + f33-2.mrack.test: + ansible_host: ec2-13-15-16-1.eu-central-1.compute.amazonaws.com + ansible_python_interpreter: /usr/bin/python3 + ansible_ssh_private_key_file: config/id_rsa + ansible_user: fedora + meta_dc_record: DC=mrack,DC=test + meta_domain: mrack.test + meta_fqdn: f33-2.mrack.test + meta_ip: 13.15.16.1 + meta_os: fedora-33 + meta_provider_id: i-08c0d4a86c4b7f7ef + meta_restraint_id: 1 + meta_role: master +``` + +## Installation + +mrack can be installed via pip, from [PyPI](https://pypi.org/project/mrack/): + +``` +pip install mrack +``` + +It is also available for Fedora 32+ via [COPR](https://copr.fedorainfracloud.org/coprs/g/freeipa/neoave/package/mrack/): + +``` +# enable the copr at your system +sudo dnf copr enable @freeipa/neoave + +# install mrack package with all supported providers and cli +sudo dnf install mrack + +# install mrack command and e.g. AWS provider +sudo dnf install mrack-cli python3-mrack-aws + +# install mrack library to be later used as an import with support of OpenStack provider: +sudo dnf install python3-mrack-openstack + +``` +--- +**NOTE** + +Above command will install `mrack` package which depends on (contains) following packages: +- mrack-cli +- python3-mracklib +- python3-mrack-aws +- python3-mrack-beaker +- python3-mrack-openstack +- python3-mrack-podman +- python3-mrack-virt + +The `python3-mracklib` is mandatory as it contains core mrack library. +To install only some of `mrack` providers you should install `python3-mrack-{provider_name}`. +To install mrack command a package `mrack-cli` is required. + +--- + +## Run mrack + +In order to use the mrack utility a mrack.conf (e.g. [mrack.conf] from the repository(repo/blob/main/src/mrack/data/mrack.conf)) is needed. + +mrack looks for the config file in following order: +- `./` actual directory +- `~/.mrack/` home directory +- `/etc/mrack/` system directory + +Values from the configuration file could be overriden using mrack utility +options `--mrack-config` `--provisioning-config` `--db` (for more see `mrack --help`). +``` +Usage: mrack [OPTIONS] COMMAND [ARGS]... + + Multihost human friendly provisioner. + +Options: + -c, --mrack-config PATH + -p, --provisioning-config PATH + -d, --db PATH + --debug + --help Show this message and exit. + +Commands: + destroy Destroy provisioned hosts. + list List host tracked by. + output Create outputs - such as Ansible inventory. + ssh SSH to host. + up Provision hosts. +``` + +### Provisioning with mrack + +To provision system from the metadata.yaml either run: +``` +mrack up +``` +or use `up` command's option `--metadata`/`-m` to override path to the metadata file. +``` +mrack up --metadata other-metadata.yaml +``` + +To return resources using mrack run: +``` +mrack destroy +``` +or use `destroy` command's option `--metadata`/`-m` to override path to the metadata file. +``` +mrack destroy --metadata other-metadata.yaml +``` + +### mrack as python library + +```python +# first set up authentication for each provider +# $ export AWS_CONFIG_FILE=`readlink -f ./aws.key` +from mrack.providers import providers +# import all supported providers: +from mrack.providers.aws import PROVISIONER_KEY as AWS +from mrack.providers.aws import AWSProvider +from mrack.providers.beaker import PROVISIONER_KEY as BEAKER +from mrack.providers.beaker import BeakerProvider +from mrack.providers.openstack import PROVISIONER_KEY as OPENSTACK +from mrack.providers.openstack import OpenStackProvider +from mrack.providers.podman import PROVISIONER_KEY as PODMAN +from mrack.providers.podman import PodmanProvider +from mrack.providers.static import PROVISIONER_KEY as STATIC +from mrack.providers.static import StaticProvider +from mrack.providers.virt import PROVISIONER_KEY as VIRT +from mrack.providers.virt import VirtProvider + +# register all supported providers: +providers.register(AWS, AWSProvider) +providers.register(OPENSTACK, OpenStackProvider) +providers.register(BEAKER, BeakerProvider) +providers.register(PODMAN, PodmanProvider) +providers.register(STATIC, StaticProvider) +providers.register(VIRT, VirtProvider) + + +# use global context class +import mrack +global_context = mrack.context.global_context + +# init global context with paths to files +mrack_config = "mrack.conf" +provisioning_config = "provisioning-config.yaml" +db_file = "mrackdb.json" +global_context.init(mrack_config, provisioning_config, db_file) + +# load the metadata to global_context +metadata = "./metadata-f34.yaml" +global_context.init_metadata(metadata) + +# pick default provider: +provider = "aws" + +# create Up action + +from mrack.actions.up import Up + +up_action = Up() +await up_action.init(provider) + +# provision machines: +await up_action.provision() + +# store the output to db or just use the db later on +from mrack.actions.output import Output +output_action = Output() +await output_action.generate_outputs() + +# work with machines... + +# cleanup the machines: +from mrack.actions.destroy import Destroy +destroy_action = Destroy() +await destroy_action.destroy() +``` + +## Contribute + +Projects is using [black](https://github.com/psf/black) formater and [isort](https://github.com/PyCQA/isort) to keep consistent +formatting, [flake8](https://flake8.pycqa.org/en/latest/) and +[pydocstyle](http://pycodestyle.pycqa.org/en/latest/intro.html) to ensure following +Python good practices. + +Contributions (Pull Requests) are welcome. It is expected that they will pass tox tests and code checkers. +Inclusion of the unit tests for the new code is recommended. +Because of that we have configured [pre-commit](https://pre-commit.com/) hook. +Please enable the feature on your local system and use it before sending a patch. +It could save us lot of re-pushing to the PR. + +### Black formatting and isort +Expected formatting can be achieved by running: +``` +$ make format +``` + +Look into [black](https://github.com/psf/black) documentation for possible integration +in editors/IDEs. + +### Testing +Just run tox to execute all tests and linters + +``` +$ tox +# or use make +$ make test +``` diff --git a/README.packit b/README.packit new file mode 100644 index 0000000..bdc4243 --- /dev/null +++ b/README.packit @@ -0,0 +1,3 @@ +This repository is maintained by packit. +https://packit.dev/ +The file was generated using packit 0.63.2.dev8+g33fd64a. diff --git a/mrack.spec b/mrack.spec new file mode 100644 index 0000000..e8ba4b4 --- /dev/null +++ b/mrack.spec @@ -0,0 +1,275 @@ +Name: mrack +Version: 1.12.2 +Release: 1%{?dist} +Summary: Multicloud use-case based multihost async provisioner + +License: Apache-2.0 +URL: https://github.com/neoave/mrack +Source0: %{URL}/releases/download/v%{version}/%{name}-%{version}.tar.gz +BuildArch: noarch + +BuildRequires: python3-devel +BuildRequires: python3-click +BuildRequires: python3-pyyaml +BuildRequires: python3-setuptools + +# coma separated list of provider plugins +%global provider_plugins aws,beaker,openstack,podman,virt + +Requires: %{name}-cli = %{version}-%{release} +Requires: python3-%{name}lib = %{version}-%{release} +Requires: python3-%{name}-aws = %{version}-%{release} +Requires: python3-%{name}-beaker = %{version}-%{release} +Requires: python3-%{name}-openstack = %{version}-%{release} +Requires: python3-%{name}-podman = %{version}-%{release} +Requires: python3-%{name}-virt = %{version}-%{release} + +# We filter out the asyncopenstackclient dependency of this package +# so it is not forcing installation of missing dependencies in Fedora +# Once python3-AsyncOpenStackClient is in fedora we can drop this line +%global __requires_exclude asyncopenstackclient +%{?python_disable_dependency_generator} + +%description +mrack is a provisioning tool and a library for CI and local multi-host +testing supporting multiple provisioning providers (e.g. AWS, Beaker, +Openstack). But in comparison to other multi-cloud libraries, +the aim is to be able to describe host from application perspective. + +%package cli +Summary: Command line interface for mrack +Requires: python3-%{name}lib = %{version}-%{release} +Requires: python3-click + +%package -n python3-%{name}lib +Summary: Core mrack libraries +Requires: python3-pyyaml +Recommends: python3-gssapi +Requires: sshpass + +%{?python_provide:%python_provide python3-%{name}lib} + +%package -n python3-%{name}-aws +Summary: AWS provider plugin for mrack +Requires: python3-%{name}lib = %{version}-%{release} +Requires: python3-boto3 +Requires: python3-botocore + +%{?python_provide:%python_provide python3-%{name}-aws} + + +%package -n python3-%{name}-beaker +Summary: Beaker provider plugin for mrack +Requires: python3-%{name}lib = %{version}-%{release} +Requires: beaker-client + +%{?python_provide:%python_provide python3-%{name}-beaker} + + +%package -n python3-%{name}-openstack +Summary: Openstack provider plugin for mrack +Requires: python3-%{name}lib = %{version}-%{release} +Recommends: python3-AsyncOpenStackClient + +%{?python_provide:%python_provide python3-%{name}-openstack} + + +%package -n python3-%{name}-podman +Summary: Podman provider plugin for mrack +Requires: python3-%{name}lib = %{version}-%{release} +Requires: podman + +%{?python_provide:%python_provide python3-%{name}-podman} + +%package -n python3-%{name}-virt +Summary: Virtualization provider plugin for mrack using testcloud +Requires: python3-%{name}lib = %{version}-%{release} +Requires: testcloud + +%{?python_provide:%python_provide python3-%{name}-virt} + +%description cli +%{name}-cli contains mrack command which functionality +can be extended by installing mrack plugins + +%description -n python3-%{name}lib +python3-%{name}lib contains core mrack functionalities +and static provider which can be used as a library + +%description -n python3-%{name}-aws +%{name}-aws is an additional plugin with AWS provisioning +library extending mrack package + +%description -n python3-%{name}-beaker +%{name}-beaker is an additional plugin with Beaker provisioning +library extending mrack package + +%description -n python3-%{name}-openstack +%{name}-openstack is an additional plugin with OpenStack provisioning +library extending mrack package + +%description -n python3-%{name}-podman +%{name}-podman is an additional plugin with Podman provisioning +library extending mrack package + +%description -n python3-%{name}-virt +%{name}-virt is an additional plugin with Virualization provisioning +library extending mrack package using testcloud + +%prep +%autosetup -p1 -n %{name}-%{version} +# Remove bundled egg-info +rm -r src/%{name}.egg-info + +%build +%py3_build + +%install +%py3_install + +%files +%license LICENSE +%doc README.md +%doc CHANGELOG.md + +%files cli +# the mrack man page RFE: https://github.com/neoave/mrack/issues/197 +%license LICENSE +%doc README.md +%doc CHANGELOG.md +%{_bindir}/%{name} +%{python3_sitelib}/%{name}/{,__pycache__/}run.* + +%files -n python3-%{name}lib +%license LICENSE +%doc README.md +%doc CHANGELOG.md +%{python3_sitelib}/%{name} +%{python3_sitelib}/%{name}-%{version}-py%{python3_version}.egg-info +%exclude %{python3_sitelib}/%{name}/{,__pycache__/}run.* +%exclude %{python3_sitelib}/%{name}/providers/utils/{,__pycache__/}osapi.* +%exclude %{python3_sitelib}/%{name}/providers/utils/{,__pycache__/}testcloud.* +%exclude %{python3_sitelib}/%{name}/providers/utils/{,__pycache__/}podman.* +%exclude %{python3_sitelib}/%{name}/providers/{,__pycache__/}{%{provider_plugins}}.* +%exclude %{python3_sitelib}/%{name}/transformers/{,__pycache__/}{%{provider_plugins}}.* + +%files -n python3-%{name}-aws +%{python3_sitelib}/%{name}/transformers/{,__pycache__/}aws.* +%{python3_sitelib}/%{name}/providers/{,__pycache__/}aws.* + +%files -n python3-%{name}-beaker +%{python3_sitelib}/%{name}/transformers/{,__pycache__/}beaker.* +%{python3_sitelib}/%{name}/providers/{,__pycache__/}beaker.* + +%files -n python3-%{name}-openstack +%{python3_sitelib}/%{name}/transformers/{,__pycache__/}openstack.* +%{python3_sitelib}/%{name}/providers/{,__pycache__/}openstack.* +%{python3_sitelib}/%{name}/providers/utils/{,__pycache__/}osapi.* + +%files -n python3-%{name}-podman +%{python3_sitelib}/%{name}/transformers/{,__pycache__/}podman.* +%{python3_sitelib}/%{name}/providers/{,__pycache__/}podman.* +%{python3_sitelib}/%{name}/providers/utils/{,__pycache__/}podman.* + +%files -n python3-%{name}-virt +%{python3_sitelib}/%{name}/transformers/{,__pycache__/}virt.* +%{python3_sitelib}/%{name}/providers/{,__pycache__/}virt.* +%{python3_sitelib}/%{name}/providers/utils/{,__pycache__/}testcloud.* + +%changelog +* Fri Dec 02 2022 Packit - 1.12.2-1 +- chore: Release version 1.12.2 (github-actions) +- chore: Use python 3.10 in GH actions (Tibor Dudlák) +- refactor: pylint fixes related to Python 3.10 (Tibor Dudlák) +- test: Fix test_utils.py to be included in pytest run (Tibor Dudlák) +- chore(pytest): add missing python_path when using pytest >=7.0.0 (Tibor Dudlák) +- test: Add test for value_to_bool util function (Tibor Dudlák) +- fix: Owner requirement boolean parsing from string (Tibor Dudlák) +- chore(Packit): Add upstream_tag_template to .packit.yaml (Tibor Dudlák) + +* Thu Nov 24 2022 Tibor Dudlák - 1.12.1-1 +- Released upstream version 1.12.1 + +* Mon Nov 14 2022 Tibor Dudlák - 1.12.0-1 +- Released upstream version 1.12.0 + +* Thu Nov 03 2022 Tibor Dudlák - 1.11.0-1 +- Released upstream version 1.11.0 + +* Wed Oct 26 2022 Tibor Dudlák - 1.10.0-1 +- Released upstream version 1.10.0 + +* Thu Oct 20 2022 Tibor Dudlák - 1.9.1-1 +- Released upstream version 1.9.1 + +* Wed Oct 12 2022 Tibor Dudlák - 1.9.0-1 +- Released upstream version 1.9.0 + +* Mon Oct 10 2022 Tibor Dudlák - 1.8.1-1 +- Released upstream version 1.8.1 + +* Mon Oct 10 2022 Tibor Dudlák - 1.8.0-1 +- Released upstream version 1.8.0 + +* Tue Sep 20 2022 Tibor Dudlák - 1.7.0-1 +- Released upstream version 1.7.0 + +* Wed Jul 27 2022 Tibor Dudlák - 1.6.0-1 +- Released upstream version 1.6.0 + +* Fri Jul 08 2022 Tibor Dudlák - 1.5.0-1 +- Released upstream version 1.5.0 + +* Fri Jun 17 2022 David Pascual Hernandez - 1.4.1-1 +- Released upstream version 1.4.1 + +* Thu May 05 2022 Tibor Dudlák - 1.4.0-1 +- Released upstream version 1.4.0 + +* Tue Apr 05 2022 Tibor Dudlák - 1.3.1-1 +- Released upstream version 1.3.1 + +* Fri Apr 01 2022 David Pascual Hernandez - 1.3.0-1 +- Released upstream version 1.3.0 + +* Wed Dec 15 2021 Tibor Dudlák - 1.2.0-1 +- Released upstream version 1.2.0 + +* Thu Nov 25 2021 Tibor Dudlák - 1.1.1-1 +- Released upstream version 1.1.1 + +* Tue Nov 23 2021 Tibor Dudlák - 1.1.0-1 +- Released upstream version 1.1.0 + +* Fri Sep 03 2021 Tibor Dudlák - 1.0.0-1 +- Released upstream version 1.0.0 + +* Thu Jul 01 2021 Tibor Dudlák - 0.14.0-1 +- Released upstream version 0.14.0 + +* Tue Jun 08 2021 Francisco Triviño - 0.13.0-1 +- Released upstream version 0.13.0 + +* Thu May 13 2021 Tibor Dudlák - 0.12.0-1 +- Released upstream version 0.12.0 + +* Fri May 07 2021 Tibor Dudlák - 0.11.0-1 +- Released upstream version 0.11.0 + +* Fri Apr 30 2021 Bhavik Bhavsar - 0.10.0-1 +- Released upstream version 0.10.0 + +* Mon Apr 19 2021 Armando Neto - 0.9.0-1 +- Released upstream version 0.9.0 + +* Thu Apr 15 2021 Armando Neto - 0.8.0-1 +- Released upstream version 0.8.0 + +* Tue Mar 23 2021 Armando Neto - 0.7.1-1 +- Released upstream version 0.7.1 + +* Mon Mar 22 2021 Tibor Dudlák - 0.7.0-1 +- Released upstream version 0.7.0 + +* Thu Feb 04 2021 Armando Neto - 0.6.0-1 +- Initial package. diff --git a/sources b/sources new file mode 100644 index 0000000..7ce0828 --- /dev/null +++ b/sources @@ -0,0 +1 @@ +SHA512 (mrack-1.12.2.tar.gz) = 7ac6ec24398d58cb092e4a12c15c75758485a5e430c74cbe3312a48dde905f0350029b70245d8b21e6bb060b3022a16657b25bd080ced9de1f193159a2752405