Implement backend for the progress bar for the presto / hive queries

[bkyryliuk]

[bkyryliuk]

Presto API description and library:
https://github.com/easydatawarehousing/prestoclient

SQLAlchemy implementation:
https://github.com/dropbox/PyHive

[bkyryliuk]

Dropbox response (dropbox/PyHive#76):

You should be able to call the poll() method right after the execute method. We actually do this ourselves in our internal dashboarding tool.

For SQLAlchemy you need to get underneath SQLAlchemy's wrapper to the raw pyhive cursor:

result_proxy = engine.execute(query)
cursor = result_proxy.cursor
cursor.poll()

[jefffeng]

yay for progress bars!

On Fri, Aug 5, 2016 at 11:32 AM, Bogdan notifications@github.com wrote:

Dropbox response (dropbox/PyHive#76
dropbox/PyHive#76):

You should be able to call the poll() method right after the execute
method. We actually do this ourselves in our internal dashboarding tool.

For SQLAlchemy you need to get underneath SQLAlchemy's wrapper to the raw
pyhive cursor:

result_proxy = engine.execute(query)
cursor = result_proxy.cursor
cursor.poll()

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#886 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARcz4R7tWy-plm5-yavrYN2VGgMgHHQIks5qc4GqgaJpZM4Jd6wT
.

Jeff Feng
Product Manager
m: (949)-610-5108
twitter: @jtfeng

[bkyryliuk]

Testing:

>> > from sqlalchemy import *
>> > from sqlalchemy.engine import create_engine
>> > from sqlalchemy.schema import *
>> > engine = create_engine('presto://localhost:${port_number}/hive/default')

result_proxy = engine.execute('SELECT COUNT(1) FROM ${presto_table}')
>> > result_proxy.cursor.poll()
{u'stats': {u'scheduled': False, u'totalSplits': 813, u'runningSplits': 192,
            u'cpuTimeMillis': 0, u'queuedSplits': 600,
            u'processedBytes': 771751936,
            u'rootStage': {u'totalSplits': 1, u'runningSplits': 1,
                           u'subStages': [
                               {u'totalSplits': 812, u'runningSplits': 191,
                                u'subStages': [], u'cpuTimeMillis': 0,
                                u'queuedSplits': 600,
                                u'processedBytes': 771751936,
                                u'userTimeMillis': 0, u'state': u'SCHEDULING',
                                u'completedSplits': 21, u'done': False,
                                u'stageId': u'1', u'nodes': 60,
                                u'processedRows': 23158770,
                                u'wallTimeMillis': 672480}],
                           u'cpuTimeMillis': 0, u'queuedSplits': 0,
                           u'processedBytes': 90, u'userTimeMillis': 0,
                           u'state': u'RUNNING', u'completedSplits': 0,
                           u'done': False, u'stageId': u'0', u'nodes': 1,
                           u'processedRows': 10, u'wallTimeMillis': 0},
            u'userTimeMillis': 0, u'state': u'RUNNING', u'completedSplits': 21,
            u'nodes': 60, u'processedRows': 23158770,
            u'wallTimeMillis': 672480},
 u'infoUri': u'http://localhost:3506/v1/query/20160805_190005_42088_rv9qe',
 u'partialCancelUri': u'http://10.186.94.154:3506/v1/stage/20160805_190005_42088_rv9qe.1',
 u'nextUri': u'http://localhost:3506/v1/statement/20160805_190005_42088_rv9qe/3',
 u'id': u'20160805_190005_42088_rv9qe', u'columns': [{u'typeSignature': {
    u'typeArguments': [], u'rawType': u'bigint', u'literalArguments': []},
                                                      u'type': u'bigint',
                                                      u'name': u'_col0'}]}
>> > result_proxy.cursor.poll()
{u'stats': {u'scheduled': False, u'totalSplits': 951, u'runningSplits': 244,
            u'cpuTimeMillis': 0, u'queuedSplits': 598,
            u'processedBytes': 5475506577,
            u'rootStage': {u'totalSplits': 1, u'runningSplits': 1,
                           u'subStages': [
                               {u'totalSplits': 950, u'runningSplits': 243,
                                u'subStages': [], u'cpuTimeMillis': 0,
                                u'queuedSplits': 598,
                                u'processedBytes': 5475506577,
                                u'userTimeMillis': 0, u'state': u'SCHEDULING',
                                u'completedSplits': 109, u'done': False,
                                u'stageId': u'1', u'nodes': 60,
                                u'processedRows': 167889753,
                                u'wallTimeMillis': 1383110}],
                           u'cpuTimeMillis': 0, u'queuedSplits': 0,
                           u'processedBytes': 189, u'userTimeMillis': 0,
                           u'state': u'RUNNING', u'completedSplits': 0,
                           u'done': False, u'stageId': u'0', u'nodes': 1,
                           u'processedRows': 21, u'wallTimeMillis': 1},
            u'userTimeMillis': 0, u'state': u'RUNNING', u'completedSplits': 109,
            u'nodes': 60, u'processedRows': 167889753,
            u'wallTimeMillis': 1383111},
 u'infoUri': u'http://localhost:3506/v1/query/20160805_190005_42088_rv9qe',
 u'partialCancelUri': u'http://10.186.94.154:3506/v1/stage/20160805_190005_42088_rv9qe.1',
 u'nextUri': u'http://localhost:3506/v1/statement/20160805_190005_42088_rv9qe/4',
 u'id': u'20160805_190005_42088_rv9qe', u'columns': [{u'typeSignature': {
    u'typeArguments': [], u'rawType': u'bigint', u'literalArguments': []},
                                                      u'type': u'bigint',
                                                      u'name': u'_col0'}]}
>> > result_proxy.cursor.poll()
{u'stats': {u'scheduled': False, u'totalSplits': 1234, u'runningSplits': 344,
            u'cpuTimeMillis': 0, u'queuedSplits': 600,
            u'processedBytes': 15279115452,
            u'rootStage': {u'totalSplits': 1, u'runningSplits': 1,
                           u'subStages': [
                               {u'totalSplits': 1233, u'runningSplits': 343,
                                u'subStages': [], u'cpuTimeMillis': 0,
                                u'queuedSplits': 600,
                                u'processedBytes': 15279115452,
                                u'userTimeMillis': 0, u'state': u'SCHEDULING',
                                u'completedSplits': 290, u'done': False,
                                u'stageId': u'1', u'nodes': 60,
                                u'processedRows': 467344442,
                                u'wallTimeMillis': 3075580}],
                           u'cpuTimeMillis': 0, u'queuedSplits': 0,
                           u'processedBytes': 2273, u'userTimeMillis': 0,
                           u'state': u'RUNNING', u'completedSplits': 0,
                           u'done': False, u'stageId': u'0', u'nodes': 1,
                           u'processedRows': 253, u'wallTimeMillis': 10},
            u'userTimeMillis': 0, u'state': u'RUNNING', u'completedSplits': 290,
            u'nodes': 60, u'processedRows': 467344442,
            u'wallTimeMillis': 3075590},
 u'infoUri': u'http://localhost:3506/v1/query/20160805_190005_42088_rv9qe',
 u'partialCancelUri': u'http://10.186.94.154:3506/v1/stage/20160805_190005_42088_rv9qe.1',
 u'nextUri': u'http://localhost:3506/v1/statement/20160805_190005_42088_rv9qe/5',
 u'id': u'20160805_190005_42088_rv9qe', u'columns': [{u'typeSignature': {
    u'typeArguments': [], u'rawType': u'bigint', u'literalArguments': []},
                                                      u'type': u'bigint',
                                                      u'name': u'_col0'}]}
>> > result_proxy.cursor.poll()
{u'stats': {u'scheduled': True, u'totalSplits': 28392, u'runningSplits': 0,
            u'cpuTimeMillis': 0, u'queuedSplits': 0,
            u'processedBytes': 1675541904069,
            u'rootStage': {u'totalSplits': 1, u'runningSplits': 0,
                           u'subStages': [
                               {u'totalSplits': 28391, u'runningSplits': 0,
                                u'subStages': [], u'cpuTimeMillis': 0,
                                u'queuedSplits': 0,
                                u'processedBytes': 1675541904069,
                                u'userTimeMillis': 0, u'state': u'FINISHED',
                                u'completedSplits': 28391, u'done': True,
                                u'stageId': u'1', u'nodes': 60,
                                u'processedRows': 81104678494,
                                u'wallTimeMillis': 205004400}],
                           u'cpuTimeMillis': 0, u'queuedSplits': 0,
                           u'processedBytes': 255518, u'userTimeMillis': 0,
                           u'state': u'FINISHED', u'completedSplits': 1,
                           u'done': True, u'stageId': u'0', u'nodes': 1,
                           u'processedRows': 28391, u'wallTimeMillis': 663},
            u'userTimeMillis': 0, u'state': u'FINISHED',
            u'completedSplits': 28392, u'nodes': 60,
            u'processedRows': 81104678494, u'wallTimeMillis': 205005063},
 u'data': [[81104678494]], u'id': u'20160805_190005_42088_rv9qe', 
 u'columns': [{u'typeSignature': 
                           {u'typeArguments': [],
                            u'rawType': u'bigint',
                            u'literalArguments': []},
                            u'type': u'bigint',
                           u'name': u'_col0'}],
 u'infoUri': u'http://localhost:3506/v1/query/20160805_190005_42088_rv9qe'}

Progress would be:
0.23616236162361623, 0.2565720294426919, 0.2787682333873582, 1

[santhavathi]

@bkyryliuk, I am unable to see the progress bar for hive datasources, whereas I am able to see the progress bar for presto datasources. My data source is hive://172.31.26.109:10000/staging_db