From 2916adc88a2686ac1785813dd1e0c7cc21a526a1 Mon Sep 17 00:00:00 2001 From: Alex Stephen Date: Fri, 6 Jul 2018 07:09:13 -0700 Subject: [PATCH] New doc guide for GCP (#41753) * New doc guide for GCP * Changes per PR * Addressing PR comments on GCE scenario guide --- .../docsite/rst/scenario_guides/guide_gce.rst | 343 ++++++++---------- 1 file changed, 156 insertions(+), 187 deletions(-) diff --git a/docs/docsite/rst/scenario_guides/guide_gce.rst b/docs/docsite/rst/scenario_guides/guide_gce.rst index a15f01f9dd2..6864a7ab19f 100644 --- a/docs/docsite/rst/scenario_guides/guide_gce.rst +++ b/docs/docsite/rst/scenario_guides/guide_gce.rst @@ -4,220 +4,230 @@ Google Cloud Platform Guide .. gce_intro: Introduction ------------- +-------------------------- -.. note:: This section of the documentation is under construction. We are in the process of adding more examples about all of the GCE modules and how they work together. Upgrades via github pull requests are welcomed! +Ansible + Google have been working together on a set of auto-generated +Ansible modules designed to consistently and comprehensively cover the entirety +of the Google Cloud Platform. -Ansible contains modules for managing Google Compute Engine resources, including creating instances, controlling network access, working with persistent disks, and managing -load balancers. Additionally, there is an inventory plugin that can automatically suck down all of your GCE instances into Ansible dynamic inventory, and create groups by tag and other properties. +Ansible contains modules for managing Google Cloud Platform resources, +including creating instances, controlling network access, working with +persistent disks, managing load balancers, and a lot more. -The GCE modules all require the apache-libcloud module which you can install from pip: +These new modules can be found under a new consistent name scheme "gcp_*" +(Note: gcp_target_proxy and gcp_url_map are legacy modules, despite the "gcp_*" +name. Please use gcp_compute_target_proxy and gcp_compute_url_map instead). + +Additionally, the gcp_compute inventory plugin can discover all GCE instances +and make them automatically available in your Ansible inventory. + +You may see a collection of other GCP modules that do not conform to this +naming convention. These are the original modules primarily developed by the +Ansible community. You will find some overlapping functionality such as with +the the "gce" module and the new "gcp_compute_instance" module. Either can be +used, but you may experience issues trying to use them together. + +While the community GCP modules are not going away, Google is investing effort +into the new "gcp_*" modules. Google is committed to ensuring the Ansible +community has a great experience with GCP and therefore recommends that begin +adopting these new modules if possible. + + +Introduction +--------------- +The Google Cloud Platform (GCP) modules require both the ``requests`` and the +``google-auth`` libraries to be installed. .. code-block:: bash - $ pip install apache-libcloud + $ pip install requests google-auth -.. note:: If you're using Ansible on macOS, libcloud also needs to access a CA cert chain. You'll need to download one (you can get one for `here `_.) Credentials ----------- +It's easy to create a GCP account with credentials for Ansible. You have multiple options to +get your credentials - here are two of the most common options: + +* Service Accounts (Recommended): Use JSON service accounts with specific permissions. +* Machine Accounts: Use the permissions associated with the GCP Instance you're using Ansible on. -To work with the GCE modules, you'll first need to get some credentials in the +For the following examples, we'll be using service account credentials. + +To work with the GCP modules, you'll first need to get some credentials in the JSON format: 1. `Create a Service Account `_ 2. `Download JSON credentials `_ -There are three different ways to provide credentials to Ansible so that it can talk with Google Cloud for provisioning and configuration actions: - -.. note:: If you would like to use JSON credentials you must have libcloud >= 0.17.0 +Once you have your credentials, there are two different ways to provide them to Ansible: -* by providing to the modules directly -* by populating a ``secrets.py`` file +* by specifying them directly as module parameters * by setting environment variables -Calling Modules By Passing Credentials -`````````````````````````````````````` +Providing Credentials as Module Parameters +`````````````````````````````````````````` For the GCE modules you can specify the credentials as arguments: +* ``auth_kind``: type of authentication being used (choices: machineaccount, serviceaccount, application) * ``service_account_email``: email associated with the project -* ``credentials_file``: path to the JSON credentials file -* ``project_id``: id of the project +* ``service_account_file``: path to the JSON credentials file +* ``project``: id of the project +* ``scopes``: The specific scopes that you want the actions to use. -For example, to create a new instance using the cloud module, you can use the following configuration: +For example, to create a new IP address using the `gcp_compute_address` module, +you can use the following configuration: .. code-block:: yaml - - name: Create instance(s) + - name: Create IP address hosts: localhost connection: local gather_facts: no vars: - service_account_email: unique-id@developer.gserviceaccount.com - credentials_file: /path/to/project.json - project_id: project-id - machine_type: n1-standard-1 - image: debian-7 + service_account_file: /home/my_account.json + project: my-project + auth_kind: serviceaccount + scopes: + - www.googleapis.com/auth/compute tasks: - - name: Launch instances - gce: - instance_names: dev - machine_type: "{{ machine_type }}" - image: "{{ image }}" - service_account_email: "{{ service_account_email }}" - credentials_file: "{{ credentials_file }}" - project_id: "{{ project_id }}" - -When running Ansible inside a GCE VM you can use the service account credentials from the local metadata server by -setting both ``service_account_email`` and ``credentials_file`` to a blank string. - -Configuring Modules with secrets.py -``````````````````````````````````` - -Create a file ``secrets.py`` looking like following, and put it in some folder which is in your ``$PYTHONPATH``: - -.. code-block:: python - - GCE_PARAMS = ('i...@project.googleusercontent.com', '/path/to/project.json') - GCE_KEYWORD_PARAMS = {'project': 'project_id', 'datacenter': 'gce_zone'} - -Ensure to enter the email address from the created services account and not the one from your main account. - -Now the modules can be used as above, but the account information can be omitted. - -If you are running Ansible from inside a GCE VM with an authorized service account you can set the email address and -credentials path as follows so that get automatically picked up: - -.. code-block:: python - - GCE_PARAMS = ('', '') - GCE_KEYWORD_PARAMS = {'project': 'project_id', 'datacenter': ''} - -Configuring Modules with Environment Variables + - name: Allocate an IP Address + gcp_compute_address: + state: present + name: 'test-address1' + region: 'us-west1' + project: "{{ project }}" + auth_kind: "{{ auth_kind }}" + service_account_file: "{{ service_account_file }}" + scopes: "{{ scopes }}" + +Providing Credentials as Environment Variables `````````````````````````````````````````````` Set the following environment variables before running Ansible in order to configure your credentials: .. code-block:: bash - GCE_EMAIL - GCE_PROJECT - GCE_CREDENTIALS_FILE_PATH + GCP_AUTH_KIND + GCP_SERVICE_ACCOUNT_EMAIL + GCP_SERVICE_ACCOUNT_FILE + GCP_SCOPES GCE Dynamic Inventory --------------------- -The best way to interact with your hosts is to use the gce inventory plugin, which dynamically queries GCE and tells Ansible what nodes can be managed. - -Note that when using the inventory script ``gce.py``, you also need to populate the ``gce.ini`` file that you can find in the contrib/inventory directory of the ansible checkout. +The best way to interact with your hosts is to use the gcp_compute inventory plugin, which dynamically queries GCE and tells Ansible what nodes can be managed. -To use the GCE dynamic inventory script, copy ``gce.py`` from ``contrib/inventory`` into your inventory directory and make it executable. You can specify credentials for ``gce.py`` using the ``GCE_INI_PATH`` environment variable -- the default is to look for gce.ini in the same directory as the inventory script. +To use the gcp_compute inventory plugin, create a file that ends in .gcp.yml file in your root directory. The gcp_compute inventory script takes in the same authentication +information as any module. -Let's see if inventory is working: - -.. code-block:: bash - - $ ./gce.py --list - -You should see output describing the hosts you have, if any, running in Google Compute Engine. - -Now let's see if we can use the inventory script to talk to Google. - -.. code-block:: bash +Here's an example of a valid inventory file: - $ GCE_INI_PATH=~/.gce.ini ansible all -i gce.py -m setup - hostname | success >> { - "ansible_facts": { - "ansible_all_ipv4_addresses": [ - "x.x.x.x" - ], - -As with all dynamic inventory scripts in Ansible, you can configure the inventory path in ansible.cfg. The recommended way to use the inventory is to create an ``inventory`` directory, and place both the ``gce.py`` script and a file containing ``localhost`` in it. This can allow for cloud inventory to be used alongside local inventory (such as a physical datacenter) or machines running in different providers. - -Executing ``ansible`` or ``ansible-playbook`` and specifying the ``inventory`` directory instead of an individual file will cause ansible to evaluate each file in that directory for inventory. - -Let's once again use our inventory script to see if it can talk to Google Cloud: - -.. code-block:: bash - - $ ansible all -i inventory/ -m setup - hostname | success >> { - "ansible_facts": { - "ansible_all_ipv4_addresses": [ - "x.x.x.x" - ], - -The output should be similar to the previous command. If you're wanting less output and just want to check for SSH connectivity, use "-m" ping instead. - -Use Cases ---------- - -For the following use case, let's use this small shell script as a wrapper. - -.. code-block:: bash - - #!/usr/bin/env bash - PLAYBOOK="$1" - - if [[ -z $PLAYBOOK ]]; then - echo "You need to pass a playbook as argument to this script." - exit 1 - fi - - export SSL_CERT_FILE=$(pwd)/cacert.pem - export ANSIBLE_HOST_KEY_CHECKING=False +.. code-block:: yaml - if [[ ! -f "$SSL_CERT_FILE" ]]; then - curl -O http://curl.haxx.se/ca/cacert.pem - fi + plugin: gcp_compute + projects: + - google.com:graphite-playground + filters: + auth_kind: serviceaccount + service_account_file: /home/alexstephen/my_account.json - ansible-playbook -v -i inventory/ "$PLAYBOOK" +Executing ``ansible-inventory --list -i .gcp.yml`` will create a list of GCP instances that are ready to be configured using Ansible. Create an instance `````````````````` -The GCE module provides the ability to provision instances within Google Compute Engine. The provisioning task is typically performed from your Ansible control server against Google Cloud's API. +The full range of GCP modules provide the ability to create a wide variety of +GCP resources with the full support of the entire GCP API. -A playbook would looks like this: +The following playbook creates a GCE Instance. This instance relies on a GCP +network and a Disk. By creating the Disk and Network separately, we can give as +much detail as necessary about how we want the disk and network formatted. By +registering a Disk/Network to a variable, we can simply insert the variable +into the instance task. The gcp_compute_instance module will figure out the +rest. .. code-block:: yaml - - name: Create instance(s) + - name: Create an instance hosts: localhost gather_facts: no connection: local - vars: - machine_type: n1-standard-1 # default - image: debian-7 - service_account_email: unique-id@developer.gserviceaccount.com - credentials_file: /path/to/project.json - project_id: project-id + project: my-project + auth_kind: serviceaccount + service_account_file: /home/my_account.json + zone: "us-central1-a" + region: "us-central1" tasks: - - name: Launch instances - gce: - instance_names: dev - machine_type: "{{ machine_type }}" - image: "{{ image }}" - service_account_email: "{{ service_account_email }}" - credentials_file: "{{ credentials_file }}" - project_id: "{{ project_id }}" - tags: webserver - register: gce + - name: create a disk + gcp_compute_disk: + name: 'disk-instance' + size_gb: 50 + source_image: 'projects/ubuntu-os-cloud/global/images/family/ubuntu-1604-lts' + zone: "{{ zone }}" + project: "{{ gcp_project }}" + auth_kind: "{{ gcp_cred_kind }}" + service_account_file: "{{ gcp_cred_file }}" + scopes: + - https://www.googleapis.com/auth/compute + state: present + register: disk + - name: create a network + gcp_compute_network: + name: 'network-instance' + project: "{{ gcp_project }}" + auth_kind: "{{ gcp_cred_kind }}" + service_account_file: "{{ gcp_cred_file }}" + scopes: + - https://www.googleapis.com/auth/compute + state: present + register: network + - name: create a address + gcp_compute_address: + name: 'address-instance' + region: "{{ region }}" + project: "{{ gcp_project }}" + auth_kind: "{{ gcp_cred_kind }}" + service_account_file: "{{ gcp_cred_file }}" + scopes: + - https://www.googleapis.com/auth/compute + state: present + register: address + - name: create a instance + gcp_compute_instance: + state: present + name: test-vm + machine_type: n1-standard-1 + disks: + - auto_delete: true + boot: true + source: "{{ disk }}" + network_interfaces: + - network: "{{ network }}" + access_configs: + - name: 'External NAT' + nat_ip: "{{ address }}" + type: 'ONE_TO_ONE_NAT' + zone: "{{ zone }}" + project: "{{ gcp_project }}" + auth_kind: "{{ gcp_cred_kind }}" + service_account_file: "{{ gcp_cred_file }}" + scopes: + - https://www.googleapis.com/auth/compute + register: instance - name: Wait for SSH to come up - wait_for: host={{ item.public_ip }} port=22 delay=10 timeout=60 - loop: "{{ gce.instance_data }}" + wait_for: host={{ instance.address }} port=22 delay=10 timeout=60 - name: Add host to groupname - add_host: hostname={{ item.public_ip }} groupname=new_instances - loop: "{{ gce.instance_data }}" + add_host: hostname={{ instance.address }} groupname=new_instances + - name: Manage new instances hosts: new_instances @@ -230,45 +240,4 @@ A playbook would looks like this: Note that use of the "add_host" module above creates a temporary, in-memory group. This means that a play in the same playbook can then manage machines in the 'new_instances' group, if so desired. Any sort of arbitrary configuration is possible at this point. -Configuring instances in a group -```````````````````````````````` - -All of the created instances in GCE are grouped by tag. Since this is a cloud, it's probably best to ignore hostnames and just focus on group management. - -Normally we'd also use roles here, but the following example is a simple one. Here we will also use the "gce_net" module to open up access to port 80 on -these nodes. - -The variables in the 'vars' section could also be kept in a 'vars_files' file or something encrypted with Ansible-vault, if you so choose. This is just -a basic example of what is possible:: - - - name: Setup web servers - hosts: tag_webserver - gather_facts: no - - vars: - machine_type: n1-standard-1 # default - image: debian-7 - service_account_email: unique-id@developer.gserviceaccount.com - credentials_file: /path/to/project.json - project_id: project-id - - roles: - - - name: Install lighttpd - apt: pkg=lighttpd state=installed - sudo: True - - - name: Allow HTTP - local_action: gce_net - args: - fwname: "all-http" - name: "default" - allowed: "tcp:80" - state: "present" - service_account_email: "{{ service_account_email }}" - credentials_file: "{{ credentials_file }}" - project_id: "{{ project_id }}" - -By pointing your browser to the IP of the server, you should see a page welcoming you. - -Upgrades to this documentation are welcome, hit the github link at the top right of this page if you would like to make additions! +For more information about Google Cloud, please visit the `Google Cloud website `