Skip to content

"TypeError: 'str' does not support the buffer interface" from async queries with redis backend on python3 #2079

@rumbin

Description

@rumbin

Make sure these boxes are checked before submitting your issue - thank you!

  • I have checked the superset logs for python stacktraces and included it here as text if any
  • I have reproduced the issue with at least the latest released version of superset
  • I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

0.15.4 on python3.4.5

Expected results

Async query succeeds; fetch results succeeds and shows result table.

Actual results

Async queries result in an error in the superset worker:
"TypeError: 'str' does not support the buffer interface"

This seems to be caused by python3 which requires the payload to be encoded.
For testing, I did the following adjustments which seem to do the job:

File sql_lab.py:

#results_backend.set(key, zlib.compress(payload))  
results_backend.set(key, zlib.compress(payload.encode('utf-8')))  

File views.py:

#json_payload = zlib.decompress(blob)
json_payload = zlib.decompress(blob).decode('utf-8')

Steps to reproduce

  1. Have a redis server running.
  2. Install superset on python3 with celery[redis] and redis,
  3. activate them in the config:
class CeleryConfig(object):
  BROKER_URL = 'redis://localhost:6379/0'
  CELERY_IMPORTS = ('superset.sql_lab', )
  CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
  CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
CELERY_CONFIG = CeleryConfig

from werkzeug.contrib.cache import RedisCache
RESULTS_BACKEND = RedisCache(host='localhost', port=6379, password=None, db=0, default_timeout=300, key_prefix='cel_result')
  1. Start superset server and worker
  2. Run async query in SQL-Lab and fetch results

Questions

Supposing, my adjustments for python3 are the right means for fixing the issue, I got the following questions:
a. Are there more than these two locations where we need to encode a string? Can anyone point me there?
b. What is the right way to fix this python3-specific issue so that we do not compromise python2 installations? (I have absolutely no experience in this regard…)
c. I'm also not experienced with github and pull requests. Should I still try to fix it myself? I'm perfectly fine if someone else wants to deal with this issue. However, if needed, I can try to make a pull request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions