add new module with integration tests to manage gitlab project variables (#56574)

* add new module with integration tests to manage gitlab project variables

* fix invalid yaml in DOCUMENTATION variable and don't import * from ansible module

* remove extends_documentation_fragment and put imports after DOCUMENTATION/EXAMPLES/RETURN/ANSIBLE_METADATA

* fix author in documentation and remove import from display

* add alias file for integration test

* split long lines and try to fix the author key

remove tailing whitespace

* replace email address with github username

* adding the at style to username

* add metaclass and future import

* add state variable to be able to delete selected variables

* add test with state = absent

* update documentation. scheme is necessary

* use singular in exmaple section

* use key purge instead of purge_vars

use purge instead of purge_vars also in the integration test

* create gitlab object in the ansible main function

* remove usedless .format

* follow best practice fail message

* add return documentation, return information about which variables were added, updated or removed and catch gitlab api auth error

* use module_utils.api with api_url and api_token

* use dict instead of list for vars

* use project name instead of name as playbook key

* add ansible checkmode_support, reduce variables in gitlab_project_variables class, remove wrong/duplicated HAS_GITLAB_PACKAGE check

* use extends_documentation_fragment and don't pop elements from basic_auth_argument_spec

* use just project_variable as output variable

* update mutually_exclusive as suggested

* re-add api_token documentation, because it is not included in api basic auth

* remove useless statement

remove unnecessary if

* add one test with a changing value

* put type at first position

* keep item to reduce api calls, build array and keep indexes by replacing with None instead of poping

* more asserts

* Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

* remove unused return key from documentation

msg is only returned when failed

* Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

* remove error key, because it is not returned

* change also documentation from purged_vars to purge

* Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update test/integration/targets/gitlab_project_variable/tasks/main.yml

Co-Authored-By: Felix Fontein <felix@fontein.de>

Update test/integration/targets/gitlab_project_variable/tasks/main.yml

Co-Authored-By: Felix Fontein <felix@fontein.de>

* remove extra spaces

fix wrong spelling

* expand return value documentation with examples

* add check_mode test

reorder tests. first the check_mode test, later all other tests

* Update lib/ansible/modules/source_control/gitlab_project_variable.py

Co-Authored-By: Felix Fontein <felix@fontein.de>

* fix existing keys in 'present' array

rework key handling (reduce code)

fix integration tests

use untouched instead of present to identify unchanged variable keys

fix wrong replacement

minor fixes on request

set aliases to unsupported, because the test succeed

remove posix group1 because it conflicts with unsupported

remove useless item from aliases

* rework gitlab connection

lib/ansible/modules/source_control/gitlab_project_variable.py

@@ -0,0 +1,254 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# Copyright: (c) 2019, Markus Bergholz (markuman@gmail.com)
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+ANSIBLE_METADATA = {'metadata_version': '1.1',
+                    'status': ['preview'],
+                    'supported_by': 'community'}
+
+
+DOCUMENTATION = '''
+module: gitlab_project_variable
+short_description: Creates/updates/deletes GitLab Projects Variables
+description:
+  - When a project variable does not exist, it will be created.
+  - When a project variable does exist, its value will be updated when the values are different.
+  - Variables which are untouched in the playbook, but are not untouched in the GitLab project,
+    they stay untouched (I(purge) is C(false)) or will be deleted (I(purge) is C(true)).
+version_added: "2.9"
+author:
+  - "Markus Bergholz (@markuman)"
+requirements:
+  - python >= 2.7
+  - python-gitlab python module
+extends_documentation_fragment:
+  - auth_basic
+options:
+  state:
+    description:
+      - Create or delete project variable.
+      - Possible values are present and absent.
+    default: present
+    type: str
+    choices: ["present", "absent"]
+  api_token:
+    description:
+      - GitLab access token with API permissions.
+    required: true
+    type: str
+  project:
+    description:
+      - The path and name of the project.
+    required: true
+    type: str
+  purge:
+    description:
+      - When set to true, all variables which are not untoucheded in the task will be deleted.
+    default: false
+    type: bool
+  vars:
+    description:
+      - A list of key value pairs.
+    default: {}
+    type: dict
+'''
+
+
+EXAMPLES = '''
+- name: Set or update some CI/CD variables
+  gitlab_project_variable:
+    api_url: https://gitlab.com
+    api_token: secret_access_token
+    project: markuman/dotfiles
+    purge: false
+    vars:
+      ACCESS_KEY_ID: abc123
+      SECRET_ACCESS_KEY: 321cba
+
+- name: Delete one variable
+  gitlab_project_variable:
+    api_url: https://gitlab.com
+    api_token: secret_access_token
+    project: markuman/dotfiles
+    state: absent
+    vars:
+      ACCESS_KEY_ID: abc123
+'''
+
+RETURN = '''
+project_variable:
+  description: Four lists of the variablenames which were added, updated, removed or exist.
+  returned: always
+  type: dict
+  contains:
+    added:
+      description: A list of variables which were created.
+      returned: always
+      type: list
+      sample: "['ACCESS_KEY_ID', 'SECRET_ACCESS_KEY']"
+    untouched:
+      description: A list of variables which exist.
+      returned: always
+      type: list
+      sample: "['ACCESS_KEY_ID', 'SECRET_ACCESS_KEY']"
+    removed:
+      description: A list of variables which were deleted.
+      returned: always
+      type: list
+      sample: "['ACCESS_KEY_ID', 'SECRET_ACCESS_KEY']"
+    updated:
+      description: A list of variables whose values were changed.
+      returned: always
+      type: list
+      sample: "['ACCESS_KEY_ID', 'SECRET_ACCESS_KEY']"
+'''
+
+import traceback
+
+from ansible.module_utils.basic import AnsibleModule, missing_required_lib
+from ansible.module_utils._text import to_native
+from ansible.module_utils.api import basic_auth_argument_spec
+
+
+GITLAB_IMP_ERR = None
+try:
+    import gitlab
+    HAS_GITLAB_PACKAGE = True
+except Exception:
+    GITLAB_IMP_ERR = traceback.format_exc()
+    HAS_GITLAB_PACKAGE = False
+
+
+class GitlabProjectVariables(object):
+
+    def __init__(self, module, gitlab_instance):
+        self.repo = gitlab_instance
+        self.project = self.get_project(module.params['project'])
+        self._module = module
+
+    def get_project(self, project_name):
+        return self.repo.projects.get(project_name)
+
+    def list_all_project_variables(self):
+        return self.project.variables.list()
+
+    def create_variable(self, key, value):
+        if self._module.check_mode:
+            return
+        return self.project.variables.create({"key": key, "value": value})
+
+    def update_variable(self, var, value):
+        if var.value == value:
+            return False
+        if self._module.check_mode:
+            return True
+        var.value = value
+        var.save()
+        return True
+
+    def delete_variable(self, key):
+        if self._module.check_mode:
+            return
+        return self.project.variables.delete(key)
+
+
+def native_python_main(this_gitlab, purge, var_list, state):
+
+    change = False
+    return_value = dict(added=list(), updated=list(), removed=list(), untouched=list())
+
+    gitlab_keys = this_gitlab.list_all_project_variables()
+    existing_variables = [x.get_id() for x in gitlab_keys]
+
+    for key in var_list:
+        if key in existing_variables:
+            index = existing_variables.index(key)
+            existing_variables[index] = None
+
+            if state == 'present':
+                single_change = this_gitlab.update_variable(
+                    gitlab_keys[index], var_list[key])
+                change = single_change or change
+                if single_change:
+                    return_value['updated'].append(key)
+                else:
+                    return_value['untouched'].append(key)
+
+            elif state == 'absent':
+                this_gitlab.delete_variable(key)
+                change = True
+                return_value['removed'].append(key)
+
+        elif key not in existing_variables and state == 'present':
+            this_gitlab.create_variable(key, var_list[key])
+            change = True
+            return_value['added'].append(key)
+
+    existing_variables = list(filter(None, existing_variables))
+    if purge:
+        for item in existing_variables:
+            this_gitlab.delete_variable(item)
+            change = True
+            return_value['removed'].append(item)
+    else:
+        return_value['untouched'].extend(existing_variables)
+
+    return change, return_value
+
+
+def main():
+    argument_spec = basic_auth_argument_spec()
+    argument_spec.update(
+        api_token=dict(type='str', required=True, no_log=True),
+        project=dict(type='str', required=True),
+        purge=dict(type='bool', required=False, default=False),
+        vars=dict(type='dict', required=False, default=dict()),
+        state=dict(type='str', default="present", choices=["absent", "present"])
+    )
+
+    module = AnsibleModule(
+        argument_spec=argument_spec,
+        mutually_exclusive=[
+            ['api_username', 'api_token'],
+            ['api_password', 'api_token'],
+        ],
+        required_together=[
+            ['api_username', 'api_password'],
+        ],
+        required_one_of=[
+            ['api_username', 'api_token']
+        ],
+        supports_check_mode=True
+    )
+
+    api_url = module.params['api_url']
+    gitlab_token = module.params['api_token']
+    purge = module.params['purge']
+    var_list = module.params['vars']
+    state = module.params['state']
+
+    if not HAS_GITLAB_PACKAGE:
+        module.fail_json(msg=missing_required_lib("python-gitlab"), exception=GITLAB_IMP_ERR)
+
+    try:
+        gitlab_instance = gitlab.Gitlab(url=api_url, private_token=gitlab_token)
+        gitlab_instance.auth()
+    except (gitlab.exceptions.GitlabAuthenticationError, gitlab.exceptions.GitlabGetError) as e:
+        module.fail_json(msg="Failed to connect to GitLab server: %s" % to_native(e))
+    except (gitlab.exceptions.GitlabHttpError) as e:
+        module.fail_json(msg="Failed to connect to Gitlab server: %s. \
+            Gitlab remove Session API now that private tokens are removed from user API endpoints since version 10.2" % to_native(e))
+
+    this_gitlab = GitlabProjectVariables(module=module, gitlab_instance=gitlab_instance)
+
+    change, return_value = native_python_main(this_gitlab, purge, var_list, state)
+
+    module.exit_json(changed=change, project_variable=return_value)
+
+
+if __name__ == '__main__':
+    main()

test/integration/targets/gitlab_project_variable/aliases

@@ -0,0 +1 @@
+unsupported

test/integration/targets/gitlab_project_variable/tasks/main.yml

@@ -0,0 +1,250 @@
+- name: Install required libs
+  pip:
+    name: python-gitlab
+    state: present
+
+- name: purge all variables for check_mode test
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    purge: True
+
+- name: add a variable value in check_mode
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: checkmode
+  check_mode: yes
+  register: gitlab_project_variable_state
+
+- name: check_mode state must be changed
+  assert:
+    that:
+      - gitlab_project_variable_state is changed
+
+- name: apply add value from check_mode test
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: checkmode
+  register: gitlab_project_variable_state
+
+- name: state must be changed
+  assert:
+    that:
+      - gitlab_project_variable_state is changed
+
+- name: change a variable value in check_mode again
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: checkmode
+  check_mode: yes
+  register: gitlab_project_variable_state
+
+- name: check_mode state must not be changed
+  assert:
+    that:
+      - gitlab_project_variable_state is not changed
+
+- name: apply again the value change from check_mode test
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: checkmode
+  register: gitlab_project_variable_state
+
+- name: state must not be changed
+  assert:
+    that:
+      - gitlab_project_variable_state is not changed
+
+- name: purge all variables at the beginning
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    purge: True
+
+- name: set two test variables
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: abc123
+      SECRET_ACCESS_KEY: 321cba
+  register: gitlab_project_variable_state
+
+- name: set two test variables state must be changed
+  assert:
+    that:
+      - gitlab_project_variable_state is changed
+      - gitlab_project_variable_state.project_variable.added|length == 2
+      - gitlab_project_variable_state.project_variable.untouched|length == 0
+      - gitlab_project_variable_state.project_variable.removed|length == 0
+      - gitlab_project_variable_state.project_variable.updated|length == 0
+
+- name: re-set two test variables
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: abc123
+      SECRET_ACCESS_KEY: 321cba
+  register: gitlab_project_variable_state
+
+- name: re-set two test variables state must not be changed
+  assert:
+    that:
+      - gitlab_project_variable_state is not changed
+      - gitlab_project_variable_state.project_variable.added|length == 0
+      - gitlab_project_variable_state.project_variable.untouched|length == 2
+      - gitlab_project_variable_state.project_variable.removed|length == 0
+      - gitlab_project_variable_state.project_variable.updated|length == 0
+
+- name: edit one variable
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: changed
+    purge: False
+  register: gitlab_project_variable_state
+
+- name: edit one variable state must be changed
+  assert:
+    that:
+      - gitlab_project_variable_state.changed
+      - gitlab_project_variable_state.project_variable.added|length == 0
+      - gitlab_project_variable_state.project_variable.untouched|length == 1
+      - gitlab_project_variable_state.project_variable.removed|length == 0
+      - gitlab_project_variable_state.project_variable.updated|length == 1
+      - gitlab_project_variable_state.project_variable.updated[0] == "ACCESS_KEY_ID"
+
+- name: append one variable
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      some: value
+    purge: False
+  register: gitlab_project_variable_state
+
+- name: append one variable state must be changed
+  assert:
+    that:
+      - gitlab_project_variable_state.changed
+      - gitlab_project_variable_state.project_variable.added|length == 1
+      - gitlab_project_variable_state.project_variable.untouched|length == 2
+      - gitlab_project_variable_state.project_variable.removed|length == 0
+      - gitlab_project_variable_state.project_variable.updated|length == 0
+      - gitlab_project_variable_state.project_variable.added[0] == "some"
+
+- name: re-set all variables
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      ACCESS_KEY_ID: changed
+      SECRET_ACCESS_KEY: 321cba
+      some: value
+  register: gitlab_project_variable_state
+
+- name: re-set all variables state must not be changed
+  assert:
+    that:
+      - not gitlab_project_variable_state.changed
+      - gitlab_project_variable_state.project_variable.added|length == 0
+      - gitlab_project_variable_state.project_variable.untouched|length == 3
+      - gitlab_project_variable_state.project_variable.removed|length == 0
+      - gitlab_project_variable_state.project_variable.updated|length == 0
+
+- name: set one variables and purge all others
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      some: value
+    purge: True
+  register: gitlab_project_variable_state
+
+- name: set one variables and purge all others state must be changed
+  assert:
+    that:
+      - gitlab_project_variable_state.changed
+      - gitlab_project_variable_state.project_variable.added|length == 0
+      - gitlab_project_variable_state.project_variable.untouched|length == 1
+      - gitlab_project_variable_state.project_variable.removed|length == 2
+      - gitlab_project_variable_state.project_variable.updated|length == 0
+
+- name: only one variable is left
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    vars:
+      some: value
+    purge: False
+  register: gitlab_project_variable_state
+
+- name: only one variable is left state must not be changed
+  assert:
+    that:
+      - not gitlab_project_variable_state.changed
+      - gitlab_project_variable_state.project_variable.added|length == 0
+      - gitlab_project_variable_state.project_variable.untouched|length == 1
+      - gitlab_project_variable_state.project_variable.removed|length == 0
+      - gitlab_project_variable_state.project_variable.updated|length == 0
+      - gitlab_project_variable_state.project_variable.untouched[0] == "some"
+
+- name: delete the last left variable
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    state: absent
+    vars:
+      some: value
+  register: gitlab_project_variable_state
+
+- name: no variable is left state must be changed
+  assert:
+    that:
+      - gitlab_project_variable_state.changed
+      - gitlab_project_variable_state.project_variable.added|length == 0
+      - gitlab_project_variable_state.project_variable.untouched|length == 0
+      - gitlab_project_variable_state.project_variable.removed|length == 1
+      - gitlab_project_variable_state.project_variable.updated|length == 0
+      - gitlab_project_variable_state.project_variable.removed[0] == "some"
+
+- name: check that no variables are left
+  gitlab_project_variable:
+    api_url: "{{ gitlab_host }}"
+    api_token: "{{ gitlab_login_token }}"
+    project: "{{ gitlab_project_name }}"
+    purge: True
+  register: gitlab_project_variable_state
+
+- name: check that no variables are untoucheded state must be changed
+  assert:
+    that:
+      - not gitlab_project_variable_state.changed
+      - gitlab_project_variable_state.project_variable.added|length == 0
+      - gitlab_project_variable_state.project_variable.untouched|length == 0
+      - gitlab_project_variable_state.project_variable.removed|length == 0
+      - gitlab_project_variable_state.project_variable.updated|length == 0