METRON-2162 Ambari client exception occurred: No JSON object could be…

[nickwallen]

When deploying Metron using the CentOS 7 development image on the feature/METRON-2088-support-hdp-3.1 feature branch, the following error occurs.

TASK [ambari_config : Deploy cluster with Ambari; http://node1:8080] ***********
...
"msg": "Ambari client exception occurred: No JSON object could be decoded"
}
to retry, use: --limit @/Users/nallen/Development/metron/metron-deployment/development/centos7/ansible/playbook.retry

PLAY RECAP *********************************************************************
node1 : ok=83 changed=16 unreachable=0 failed=1

[WARNING]: Module did not set no_log for password

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

• This PR is for the HDP 3.1 feature branch, not master.

Background

I have seen this issue previously and it was caused by the python-requests library. The CentOS 7 node is running Python Requests 2.6.1, which is the last known working version of the library at least on CentOS 6. This was not a problem when the original PR went in for METRON-2097 as #1397 . There might have been a change with the underlying CentOS 7 VM image.

Querying Ambari using curl shows Ambari is responding correctly.

[root@node1 ~]# curl -u admin:admin -H "X-Requested-By: ambari" -i -X GET -k http://node1:8080/api/v1/blueprints
HTTP/1.1 200 OK
Date: Tue, 25 Jun 2019 13:56:19 GMT
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-store
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=node0jsb8b6skdylu71zzy5l1qfv2.node0;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
User: admin
Content-Type: text/plain;charset=utf-8
X-Content-Type-Options: nosniff
Vary: Accept-Encoding, User-Agent
Transfer-Encoding: chunked

{
"href" : "http://node1:8080/api/v1/blueprints",
"items" : [ ]
}

Querying Ambari using python-requests 2.6.1 on CentOS 7 highlights the problem.

>>> import requests
>>> print requests.__version__
2.6.1
>>> r = requests.get("http://node1:8080/api/v1/blueprints", auth=("admin","admin"))
>>> r
<Response [200]>
>>> r.encoding
'utf-8'
>>> r.content
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\xab\xe6RPP\xca(JMSR\xb0\x022JJ\n\xac\xf4\xf5\xf3\xf2SR\r\xad,\x0c,\x0c\xf4\x13\x0b2\xf5\xcb\x0c\xf5\x93rJS\x0b\x8a2\xf3J\x8a\x95t@:2KRs\x8bAZ\xa2\x15b\xb9j\x01OR\xd0\x9bE\x00\x00\x00'
>>> r.text
u'\x1f\ufffd\x08\x00\x00\x00\x00\x00\x00\x00\ufffd\ufffdRPP\ufffd(JMSR\ufffd\x022JJ\n\ufffd\ufffd\ufffd\ufffd\ufffdSR\r\ufffd,\x0c,\x0c\ufffd\x13\x0b2\ufffd\ufffd\x0c\ufffd\ufffdrJS\x0b\ufffd2\ufffdJ\ufffd\ufffdt@:2KRs\ufffdAZ\ufffd\x15b\ufffdj\x01OR\u041bE\x00\x00\x00'
>>> r.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/requests/models.py", line 819, in json
return json.loads(self.text, **kwargs)
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Testing

Deploy Metron on CentOS 7.

cd metron-deployment/development/centos7
vagrant up

Pull Request Checklist

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?
• Have you included steps to reproduce the behavior or problem that is being changed or addressed?
• Have you included steps or a guide to how the change may be verified and tested manually?
• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
• Have you written or updated unit tests and or integration tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

[merrimanr]

I tested this in full dev and the issue is resolved. +1

[nickwallen]

This has been merged into the feature branch.