Some Python parsing errors are not propagated to the driver response

[juanjux]

For example, this file: https://github.com/damoeb/kalipo/blob/master/kalipo-ir/harvester/spiders/heise_spider.py produces a TabError/SyntaxError (Python 3/2) but the only error that the driver returns is "Could not determine Python version", which is true, but also incomplete; the original SyntaxError should be added to the driver response.

[bzz]

Trying latest and grated version of everything today with https://github.com/juanjux/bblfsh-dev-image got me here. That's the only issue stumbled up on so far, detailed logs below.

Details

Client

        Parsing file:kalipo-ir/harvester/spiders/heise_spider.py, size:2247
No error, but - ParseUAST status:fatal, error:[Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 136, in process_request
    raise Exception('Could not determine Python version')
Exception: Could not determine Python version
] for kalipo-ir/harvester/spiders/heise_spider.py
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x587f17]

Server

DEBU[0031] sending ParseUAST request: Filename:"kalipo-ir/harvester/spiders/heise_spider.py" Language:"python" Content:"import scrapy\nfrom scrapy.contrib.spiders
 import CrawlSpider, Rule\nfrom scrapy.contrib.linkextractors import LinkExtractor\nfrom scrapy.selector import Selector\n\nfrom harvester.items import Comment\ni
mport time\nimport calendar\nimport re\n\nclass HeiseSpider(CrawlSpider):\n    name = \"heise\"\n    allowed_domains = [\"www.heise.de\"]\n    start_urls = [\n
         \"http://www.heise.de/forum/Telepolis/Kommentare/Ohne-Vorratsdatenspeicherung-sterben-vermisste-Kinder-und-Suizidale/forum-242979/\"\n    ]\n\n    rules
= (\n        #Rule(LinkExtractor(allow=('/tp/foren/[^/]+/forum-[0-9]+/list'))),\n\tRule(LinkExtractor(allow=('/posting-[0-9]+/show')), callback='parse_item')\n
 )\n\n    def clean_str(self, val):\n\treturn val.replace(u'\\xa0', u' ').strip()\n\n    def to_str(self, arr):\n\treturn self.clean_str(''.join(arr))\n\n    def
parse_date(self, val):\n\tgrps = re.search('[0-9]+\\. ([A-Za-z]+) [0-9]{4} [0-9]{2}:[0-9]{2}', val)\n\n\tmnth = grps.group(1)\n\n   \tmonths = ['Januar', 'Februar
', 'M\\u00e4rz', 'April', 'Mai', 'Juni', 'Juli', 'August', 'September', 'Oktober', 'November', 'Dezember']\n\tfor index, item in enumerate(months):\n\t   if item.
lower() == mnth.lower():\n\t      val = val.replace(mnth, str(index))\n\t      break\n\n\treturn calendar.timegm(time.strptime(val, \"%d. %m %Y %H:%M\"))\n\n    d
ef parse_item(self, response):\n        sel = Selector(response)\n\n\n\tisRoot = len(response.xpath(\"//ul[@class='forum_navi'][2]/li\")) == 6\n\n\tif !isRoot:\n\
t   # find parent\n\t   parent = response.xpath(\"//span[@class='active_post']/../../../parent::ul[@class='nextlevel_line']/preceding-sibling::div[@class='hover_l
ine']\")\n\t   # get link\n\t   link = parent.xpath(\".//div[@class='thread_title']/a\")\n\t   # extract parent id from href\n\n\n\titem = Comment()\n\titem['text
'] = self.to_str(sel.xpath(\"//h3[@class='posting_subject']/text()\").extract()) + self.to_str(sel.xpath(\"//p[@class='posting_text']/text()\").extract())\n\titem
['url'] = response.url\n\titem['parent'] = 'unknown'\n\titem['level'] = 0\n\titem['thread'] = re.search('forum-([0-9]+)', response.url).group(1)\n\titem['author']
 = self.to_str(sel.xpath(\"//div[@class='user_info']/i//text()\").extract())\n\titem['date'] = self.parse_date(self.to_str(response.xpath(\"//div[@class='posting_
date']/text()\").extract()))\n        return item\n\n" Encoding:UTF8
ERRO[0031] driver bblfsh/python-driver:latest (01BKQ64795JC9FEPB2AFWSR2NJ) stderr: ERROR:root:Filepath: , Errors: ['Traceback (most recent call last):\n  File "/u
sr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 136, in process_request\n    raise Exception(\'Could not determine Python version\')\nExce
ption: Could not determine Python version\n']

[juanjux]

Yes, looks like it, if you enter the image with the build drivers you can actually check for the specific error while I fix it:

python3 -m pydetector -v3 boom.py

Also try to run that Python file, last time it was mixing tabs and spaces with would have produced an error with the interpreter quickly.

Edit: forgot the -v3 parameter.

[bzz]

It seems that pydetector is only inside python-driver image.
Do you know any simple way of executing it from within https://github.com/juanjux/bblfsh-dev-image container?

[juanjux]

pip3 install pydetector. For it to work at 100% it should also be installed for Python 2, but it's not installed on the image, so you should have to install that too (and pydetector again for python2 with pip2).

[bzz]

TabError: inconsistent use of tabs and spaces in indentation 🎉

Thanks for support and introduction of pydetector - it's awesome!

😳 https://pypi.python.org/pypi/pydetector/0.3.3

It it developer as a semi-independent part of
the Babelfish project.

[juanjux]

Mmmm I'll add another issue to at least try to automatically fix these files with mixed indentation, I think there is even a script included in Python for that.

[juanjux]

@bzz #31

[juanjux]

PR submitted. Should fix this as soon as is merged. Thanks for your report and patience!