Ansible-playbook is not accepting JSON args properly

[nmaludy]

Problem

When running the ansible.playbook action and passing in "structure" data for extra_vars, Ansible is not interpreting them properly.

    ansible_playbook_linux:
      action: ansible.playbook
      input:
        playbook: "{{ _.playbook }}"
        inventory_file: "{{ _.server }},"
        extra_vars:
          - wait_timeout: 2
            wait_sleep: 2
            wait_connect_timeout: 2
            ansible_user: "user"
            ansible_ssh_pass: "xxx"
            ansible_ssh_extra_args: "-o StrictHostKeyChecking=no"

Looking at the output of ps -aef | grep ansible i see the following:

root     12711  2520  0 10:02 pts/0    00:00:00 sudo -E -- bash -c /opt/stackstorm/packs/ansible/actions/ansible_playbook.py --extra_vars='[{u'"'"'wait_sleep'"'"': 2, u'"'"'ansible_ssh_pass'"'"': u'"'"'xxx'"'"', u'"'"'ansible_user'"'"': u'"'"'user'"'"', u'"'"'wait_timeout'"'"': 2, u'"'"'wait_connect_timeout'"'"': 2}]' --inventory_file=host.domain.tld, -vvv /opt/encore/ansible/playbooks/wait_for_connection.yaml
root     12712 12711  0 10:02 pts/0    00:00:00 python /opt/stackstorm/packs/ansible/actions/ansible_playbook.py --extra_vars=[{u'wait_sleep': 2, u'ansible_ssh_pass': u'xxx', u'ansible_user': u'user', u'wait_timeout': 2, u'wait_connect_timeout': 2}] --inventory_file=host.domain.tld, -vvv /opt/encore/ansible/playbooks/wait_for_connection.yaml
root     12713 12712 12 10:02 pts/0    00:00:00 /opt/stackstorm/virtualenvs/ansible/bin/python /opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook --inventory-file=host.domain.tld, -vvv /opt/encore/ansible/playbooks/wait_for_connection.yaml --extra-vars='{"wait_timeout": 2, "wait_connect_timeout": 2, "wait_sleep": 2, "ansible_user": "user", "ansible_ssh_pass": "xxx"}'
root     12726 12713  2 10:02 pts/0    00:00:00 /opt/stackstorm/virtualenvs/ansible/bin/python /opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook --inventory-file=host.domain.tld, -vvv /opt/encore/ansible/playbooks/wait_for_connection.yaml --extra-vars='{"wait_timeout": 2, "wait_connect_timeout": 2, "wait_sleep": 2, "ansible_user": "user", "ansible_ssh_pass": "xxx"}'

This ansible command times out after the default 10 minutes for the playbook.

System Details

$ ansible --version
ansible 2.3.2.0
  config file = 
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

$ st2 --version
st2 2.4.0

$ st2 pack get ansible
+-------------+--------------------------------------------------+
| Property    | Value                                            |
+-------------+--------------------------------------------------+
| name        | ansible                                          |
| version     | 0.5.2                                            |
| author      | StackStorm, Inc.                                 |
| email       | info@stackstorm.com                              |
| keywords    | [                                                |
|             |     "ansible",                                   |
|             |     "cfg management",                            |
|             |     "configuration management"                   |
|             | ]                                                |
| description | st2 content pack containing ansible integrations |
+-------------+--------------------------------------------------+

Debugging

Manually running the following command executes just, ansible logs in immediately and no timeouts.

 /opt/stackstorm/virtualenvs/ansible/bin/python /opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook --inventory-file=host.domain.tld, -vvv /opt/encore/ansible/playbooks/wait_for_connection.yaml --extra-vars='{"wait_timeout": 2, "wait_connect_timeout": 2, "wait_sleep": 2, "ansible_user": "user", "ansible_ssh_pass": "xxx"}'

However, running the following does NOT work (waits forever until a timeout).

sudo -E -- bash -c /opt/stackstorm/packs/ansible/actions/ansible_playbook.py --extra_vars='[{u'"'"'wait_sleep'"'"': 2, u'"'"'ansible_ssh_pass'"'"': u'"'"'xxx'"'"', u'"'"'ansible_user'"'"': u'"'"'user'"'"', u'"'"'wait_timeout'"'"': 2, u'"'"'wait_connect_timeout'"'"': 2}]' --inventory_file=host.domain.tld, -vvv /opt/encore/ansible/playbooks/wait_for_connection.yaml

I next added a print statement to my playbook

  pre_tasks:
    - name: Display all variables/facts
      debug:
        msg: "{{ vars | to_nice_json(indent=4) }}"

Now i compared the outputs from the first command to the second command.

Output from the first command that invokes ansible_playbook.py:

TASK [Display all variables/facts] ************************************************************************************
task path: /opt/encore/ansible/playbooks/wait_for_connection.yaml:13
ok: [host.domain.tld] => {
    "msg": {
        "_raw_params": "'{\"wait_timeout\": 2, \"wait_connect_timeout\": 2, \"wait_sleep\": 2, \"ansible_user\": \"user\", \"ansible_ssh_pass\": \"xxx\"}'",                                                                             
        "ansible_check_mode": false, 
        "ansible_play_batch": [
            "host.domain.tld"
        ], 
        "ansible_play_hosts": [
            "host.domain.tld"
        ],
....
 
        "wait_connect_timeout": 5, 
        "wait_sleep": 1, 
        "wait_timeout": 600
    }
}

Output from the second command that invoking ansible directly:

TASK [Display all variables/facts] ************************************************************************************
ok: [host.domain.tld] => {
    "msg": {
        "ansible_check_mode": false, 
        "ansible_play_batch": [
            "host.domain.tld"
        ], 
        "ansible_play_hosts": [
            "host.domain.tld"
        ], 
        "ansible_play_hosts_all": [
            "host.domain.tld"
        ], 
        "ansible_playbook_python": "/opt/stackstorm/virtualenvs/ansible/bin/python", 
        "ansible_ssh_pass": "xxx", 
        "ansible_user": "user", 
....
        "wait_connect_timeout": 2, 
        "wait_sleep": 2, 
        "wait_timeout": 2
    }
}

As you can see, when invoking ansible_playbook.py the variables are are not merged properly into the vars dict, causing the connection to fail repeatedly until the timeout of 10 minutes (default) occurs.

Conversely, when we invoke it from shell, the --extra-vars is merged properly with the vars dict and the connection works as expected.

Proposed Solution

The ansible-playbook is where the command is being executed here https://github.com/StackStorm-Exchange/stackstorm-ansible/blob/master/actions/lib/ansible_base.py#L105 .

If we change that line to use a shell, and convert the command from a list to a string (see below), then everything works as expected.

exit_code = subprocess.call(' '.join(self.cmd), env=os.environ.copy(), shell=True)

[nmaludy]

Little more debugging (a lot more)...

Let me walk you through the call stack here:

Entry Point: /opt/stackstorm/virtualenv/ansible/bin/ansible-playbook

https://github.com/ansible/ansible/blob/stable-2.3/bin/ansible#L109 - There is some magic here, but it basically translates to ansible.cli.playbook.PlaybookCLI.run() (--extra-vars is parsed in run() not in parse() ... go figure)

https://github.com/ansible/ansible/blob/stable-2.3/lib/ansible/cli/playbook.py#L123 - This invokes load_extra_vars() to parse out the --extra-vars.

https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/vars.py#L122-L143 - This does the actual parsing.

According to my debugging the JSON parsing is failing because our argument string is --extra-vars='{0}' defined here https://github.com/StackStorm-Exchange/stackstorm-ansible/blob/master/actions/lib/ansible_base.py#L67

Now when looking at the load_extra_vars() code it's comparing the first character in extra_vars_opt to see if it's an @ in this case it loads from file, else if it's an [ or { then it tries to parse it as JSON/YAML.

In our case, from my debugging, our first char is a ' causing this logic to fall to the 3rd case of the key=value format which then results in bad parsing and variables being put into _raw_params.

Soo... if we remove the wrapping 's from https://github.com/StackStorm-Exchange/stackstorm-ansible/blob/master/actions/lib/ansible_base.py#L67 then this also solves our problem without having to invoke subprocess.call(... shell=True).

This is probably the best option with regards to security.

[nmaludy]

@cognifloyd Any thoughts?

[arm4b]

Yeah, definitely it would be good to avoid calling subprocess with shell=True

[cognifloyd]

Wow. That's surprising. I wonder why I didn't hit the same issue. I haven't tested this extra vars stuff with ST2 2.4 yet, though I did just install it last week. It's working for me with ansible 2.3.2 and ST2 2.3.something. maybe that's a fluke though.

If you're running from a shell you need the quotes, but it makes sense to remove them for this subprocess command. We may need to adjust the tests... Hmm.

[cognifloyd]

Are you using redhat or centos? What version?

I've been running this on a vagrant box with Ubuntu xenial (iirc). I will test this when i can using both the vagrant box with older ST2, and the centos7 box I just setup with ST2 2.4. both have ansible 2.3.2.

[nmaludy]

@cognifloyd i'm seeing this on CentOS 7

[root@stackstorm ~]# uname -a
Linux stackstorm.domain.tld 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@stackstorm ~]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 

[cognifloyd]

#20 looks like an excellent fix to me.

The reason I haven't been experiencing this, is because I've been using a fork that includes unmerged PRs: #11 and #14. #11 includes 8c2d845 which fixes this bug in exactly the same way as @nmaludy did in #20.

I'm sorry, I should have submitted that as a separate PR, but, instead, I forgot about it.

Please merge #20, probably as v0.5.3.