Semantic objects #157

juanjux · 2018-06-19T16:14:43Z

Status:

Some identifiers don't have positions, because the don't in the native AST, but that could be fixed later (was that way in the version with the previous DSL too).

dennwc · 2018-06-26T18:48:25Z

.gitignore

 .cache/*
 __pycache__
 makebuild.sh
+.vscode/launch.json


.vscode/?

Done in the next commit.

dennwc · 2018-06-26T18:50:02Z

driver/fixtures/fixtures_test.go

 	},
 	BenchName: "issue_server101",
+	// XXX check this
+	//Semantic: fixtures.SemanticConfig{


Time to uncomment this?

Done in the next push.

dennwc · 2018-06-26T18:53:34Z

driver/normalizer/normalizer.go

+var Preprocessors = []Mapping{
+	ObjectToNode{
+		InternalTypeKey: "ast_type",
+		// FIXME: if the node doesn't have positions or is missing end positions


Should be fixed now

Done in the next push.

dennwc · 2018-06-26T18:55:19Z

driver/normalizer/normalizer.go

+	return n, nil
+}
+
+// FIXME: decorators? (annotations/tags)


For now, this can be solved by adding a separate property to the node with async.

Mmmm, I don't understand well you here.

dennwc · 2018-06-26T18:56:56Z

driver/normalizer/normalizer.go

+
+// FIXME: decorators? (annotations/tags)
+// FIXME: in Python, an argument being variadic or not depends on being on
+// args.[kwonlyargs|args] or plain "args" not on any property of the argument


If I understood this correctly, it can be solved by using Cases operator. Please check Java/JS/Go for an example.

It was pretty complicated. Forgot to remove that comment; I did the easiest option and changed the native structure to use the standard Arg(with properties)->Default argument instead of the multiple lists to you should have to match aligning to the right...

dennwc · 2018-06-26T18:58:15Z

fixtures/annotations.py.native

-        }
-    }
-}
+   'PY3AST': {


Please regenerate the fixtures as well

Odd. I deleted the fixtures, updated the SDK, did bblfsh-sdk update on the driver dir and regenerated the fixtures with go test and is still using that format.

dennwc · 2018-06-26T18:59:18Z

fixtures/annotations.py.sem.uast

+               async: false,
+            },
+            { '@type': "uast:Alias",
+               Name: "somefunc",


Should be an Identifier node

dennwc · 2018-06-26T18:59:52Z

fixtures/annotations.py.sem.uast

+                     },
+                  ],
+                  Type: { '@type': "uast:FunctionType",
+                     Arguments: { '@type': "arguments",


Arguments should be an array

dennwc · 2018-06-26T19:00:28Z

fixtures/annotations.py.sem.uast

+                              },
+                              Init: ~,
+                              MapVariadic: false,
+                              Name: "a",


Should be an Identifier node

juanjux · 2018-06-27T13:59:32Z

Did some fixes, still not finished here but at least now all the semantic objects are being generated 🎉 , which allows me to find some problems (other than the indicated here of Names being strings instead of uast:Identifier nodes).

juanjux · 2018-06-28T16:23:12Z

Outside of the problem of nodes with missing partial or total positional info not being processed, I think I've fixed all the issues, but that problem doesn't allow me to check if uast:Comment and uast:RuntimeImports are right.

juanjux · 2018-07-02T16:39:07Z

@dennwc as requested, updated so go test can be run after just a make build-native without installing anything locally.

Unfortunately, this breaks make test and make integration-test; I'll investigate why tomorrow. Ultimately I really thing go test should run the Makefile commands in the configured docker image to avoid these kludges and improve reproducibility.

dennwc · 2018-07-02T17:40:29Z

@juanjux It's expected that user will run make build-native before running tests, so you can still add directives to that makefile (except installing stuff globally). This will ensure that build directory is self-contained and we will be able to package it as-is in Docker, Snap or anything else.

juanjux · 2018-07-02T17:55:24Z

To make the directory self contained you've to investigate how to do it, how to install modules to a specific path, add the libs in that path to the language path, how to do the same with the binary. This for every language, while installing everything "globally" into a container fixes everything and avoid exotic running environment that could cause side effects (and I've found several of these as you know with this approach).

dennwc · 2018-07-02T18:07:05Z

driver/fixtures/fixtures_test.go

 }

+func SetPythonPath() {
+	os.Setenv("PYTHONPATH", filepath.Join(projectRoot,


We can make ./build/bin/native a bash script, set an env variable there and call python_driver from it.

Yes, it actually is already a bash script, I'll change it, I was just testing things to make this work, but I fear the environment variable solution won't work for parsing when the Python2 binary must be called, maybe with .pth files...

juanjux · 2018-07-03T10:44:53Z

@dennwc with the SDK's update and the comments fix all nodes are now processed (searching for "ast_type" in *uast* doesn't give any results, trough I manually checked RuntimeImports which also had skipped subnodes and now doesn't).

I still need to fix make test to make this mergeable, working on it right now.

juanjux · 2018-07-04T12:11:40Z

make test fixed.

@dennwc I kept the setting of the PYTHONPATH environment variable on the fixtures_test.go file because it's cleaner there, since the native file can be called from different paths depending on if it's a local test, or running inside the docker image, to make it work there I would had needed to add several different paths, which would be ugly.

I've also tested that it doesn't have problems on the tests that force the Python2 interpreter (like print.py).

PS: also rebased.

dennwc · 2018-07-04T12:39:16Z

driver/fixtures/fixtures_test.go

+			"LineComment",
+			"Block",
+			"ImportDeclaration",
+			"MethodDeclaration",


I can't find this type in the Python's native AST. Is it a list for Java?

dennwc · 2018-07-04T12:39:59Z

fixtures/u2_func_simple.py.sem.uast

+   '@pos': { '@type': "uast:Positions",
+   },
+   body: [
+      { '@type': "FunctionDef",


uast:FunctionGroup

dennwc · 2018-07-04T12:40:19Z

fixtures/u2_func_simple.py.sem.uast

+   },
+   body: [
+      { '@type': "FunctionDef",
+         '@token': "testfnc1",


uast:Identifier

dennwc · 2018-07-04T12:40:48Z

fixtures/u2_func_simple.py.sem.uast

+               },
+            ],
+         },
+         body: { '@type': "FunctionDef.body",


uast:Block

dennwc · 2018-07-04T12:43:15Z

fixtures/u2_import_path.py.sem.uast

+               Name: "a",
+            },
+         ],
+         value: { '@type': "Call",


We could convert an __import__() to RuntimeImport as well. Thoughts?

Unsure. There are actually tons of different ways to side-import things in python:

Read and eval and assign to a global name in the module table.

imp module.

importlib

...many others that I don't remember but I've seen in the past, some of those pretty exotic.

But is true that __import__ is a default function in Python so that case at least could be easy to handle.

After some consideration, I've decided that I'll not convert __import__ in this version:

The argument could be a string, but also could be a variable, a function call, etc so we don't have a real way to know the RuntimeImport.path.

It will be assigned to a returned value that could be different from the argument.

This would need pretty complex annotations or flow analysis in the native drivers and thus I think is better to leave it to future versions to don't delay this anymore.

dennwc · 2018-07-04T12:43:40Z

fixtures/u2_import_path.py.sem.uast

+         },
+         All: false,
+         Names: [
+            { '@type': "alias",


uast:Identifier or uast:Alias

dennwc · 2018-07-04T12:44:02Z

fixtures/u2_import_path.py.sem.uast

+               },
+            },
+         ],
+         Path: "c",


uast:String

dennwc · 2018-07-04T12:46:08Z

driver/fixtures/fixtures_test.go

 	NewDriver: func() driver.BaseDriver {
-		return driver.NewExecDriverAt(filepath.Join(projectRoot, "build/bin/native"))
+		return driver.NewExecDriverAt(filepath.Join(projectRoot,
+			"build/.local/bin/python_driver"))


Ideally, we shouldn't change the path to native driver (../../build/bin/native). That script should use relative paths so it doesn't matter if it's in /opt/bblfsh/driver or in the project directory.

juanjux · 2018-07-05T17:54:06Z

All checks from go test passing after the last push. Still need to handle the import levels (the dots) to prepend ../ to the path (or the dots themselves, I'm not sure) and handle __import__(string) as a RuntimeImport object.

dennwc · 2018-07-06T13:45:36Z

@juanjux Please update fixture files

juanjux · 2018-07-06T13:57:18Z

@dennwc they are updated :-?

dennwc · 2018-07-06T14:11:29Z

Here's the output I'm getting:

Details

``` ~/go/src/github.com/bblfsh/python-driver$ git pull Already up to date. ~/go/src/github.com/bblfsh/python-driver$ go test ./driver/fixtures/ --- FAIL: TestPythonDriver (8.92s) --- FAIL: TestPythonDriver/native (0.85s) --- FAIL: TestPythonDriver/native/funcdef_kwonlyargs (0.00s) fixtures.go:154: Error Trace: fixtures.go:154 Error: Not equal: expected: "{\n 'PY3AST': {\n 'ast_type': \"Module\",\n... Test: TestPythonDriver/native/funcdef_kwonlyargs --- FAIL: TestPythonDriver/native/issue_server101 (0.47s) fixtures.go:154: Error Trace: fixtures.go:154 Error: Not equal: Test: TestPythonDriver/native/issue_server101 --- FAIL: TestPythonDriver/native/u2_func_params_kwonlyargs (0.00s) fixtures.go:154: Error Trace: fixtures.go:154 Error: Not equal: expected: "{\n 'PY3AST': {\n 'ast_type': \"Module\",\n... Test: TestPythonDriver/native/u2_func_params_kwonlyargs --- FAIL: TestPythonDriver/native/u2_func_params_kwonlyargs_default (0.00s) fixtures.go:154: Error Trace: fixtures.go:154 Error: Not equal: expected: "{\n 'PY3AST': {\n 'ast_type': \"Module\",\n body: [\n... Test: TestPythonDriver/native/u2_func_params_kwonlyargs_default --- FAIL: TestPythonDriver/native/u2_func_params_variadic_list (0.00s) fixtures.go:154: Error Trace: fixtures.go:154 Error: Not equal: expected: "{\n 'PY3AST': {\n 'ast_type': \"Module\",\n body: [\n {\n... Test: TestPythonDriver/native/u2_func_params_variadic_list --- FAIL: TestPythonDriver/native/u2_func_params_variadic_map (0.00s) fixtures.go:154: Error Trace: fixtures.go:154 Error: Not equal: expected: "{\n 'PY3AST': {\n 'ast_type': \"Module\",\n body: [\n {\n... Test: TestPythonDriver/native/u2_func_params_variadic_map --- FAIL: TestPythonDriver/uast (3.69s) --- FAIL: TestPythonDriver/uast/funcdef_kwonlyargs (0.01s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: expected: "{ '@type': \"Module\",\n '@ROLE': [File, Module],\n '@pos': { ... Test: TestPythonDriver/uast/funcdef_kwonlyargs --- FAIL: TestPythonDriver/uast/issue_server101 (2.53s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: Test: TestPythonDriver/uast/issue_server101 --- FAIL: TestPythonDriver/uast/u2_func_params_kwonlyargs (0.01s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: expected: "{ '@type': \"Module\",\n '@ROLE': [File, Module],\n '@pos': { ... Test: TestPythonDriver/uast/u2_func_params_kwonlyargs --- FAIL: TestPythonDriver/uast/u2_func_params_kwonlyargs_default (0.01s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: expected: "{ '@type': \"Module\",\n '@ROLE': [File, Module],\n '@pos': { ... Test: TestPythonDriver/uast/u2_func_params_kwonlyargs_default --- FAIL: TestPythonDriver/uast/u2_func_params_variadic_list (0.00s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: expected: "{ '@type': \"Module\",\n '@ROLE': [File, Module],\n '@pos': { ... Test: TestPythonDriver/uast/u2_func_params_variadic_list --- FAIL: TestPythonDriver/uast/u2_func_params_variadic_map (0.00s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: expected: "{ '@type': \"Module\",\n '@ROLE': [File, Module],\n '@pos': { ... Test: TestPythonDriver/uast/u2_func_params_variadic_map --- FAIL: TestPythonDriver/semantic (4.38s) --- FAIL: TestPythonDriver/semantic/issue30 (0.01s) fixtures.go:213: blacklisted nodes of type "Import" (1) found in the tree --- FAIL: TestPythonDriver/semantic/issue62 (0.01s) fixtures.go:213: blacklisted nodes of type "Import" (2) found in the tree --- FAIL: TestPythonDriver/semantic/issue94 (0.07s) fixtures.go:213: blacklisted nodes of type "Import" (3) found in the tree --- FAIL: TestPythonDriver/semantic/issue96 (0.07s) fixtures.go:213: blacklisted nodes of type "Import" (3) found in the tree --- FAIL: TestPythonDriver/semantic/issue_server101 (2.96s) fixtures.go:213: blacklisted nodes of type "Import" (6) found in the tree fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:Function": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:230: type check failed for "uast:FunctionType": incorrect object type: "uast:Identifier", expected: "uast:Argument" fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: Test: TestPythonDriver/semantic/issue_server101 --- FAIL: TestPythonDriver/semantic/u2_func_params_variadic_list (0.01s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: expected: "{ '@type': \"Module\",\n '@ROLE': [File, Module],\n '@pos': { ... Test: TestPythonDriver/semantic/u2_func_params_variadic_list --- FAIL: TestPythonDriver/semantic/u2_func_params_variadic_map (0.01s) fixtures.go:254: Error Trace: fixtures.go:254 Error: Not equal: expected: "{ '@type': \"Module\",\n '@ROLE': [File, Module],\n '@pos': { ... Test: TestPythonDriver/semantic/u2_func_params_variadic_map --- FAIL: TestPythonDriver/semantic/u2_import_module_alias (0.00s) fixtures.go:213: blacklisted nodes of type "Import" (1) found in the tree --- FAIL: TestPythonDriver/semantic/u2_import_multiple (0.01s) fixtures.go:213: blacklisted nodes of type "Import" (1) found in the tree --- FAIL: TestPythonDriver/semantic/u2_import_rename (0.01s) fixtures.go:213: blacklisted nodes of type "Import" (1) found in the tree --- FAIL: TestPythonDriver/semantic/u2_import_simple (0.01s) fixtures.go:213: blacklisted nodes of type "Import" (2) found in the tree --- FAIL: TestPythonDriver/semantic/u2_import_subsymbol (0.01s) fixtures.go:213: blacklisted nodes of type "Import" (1) found in the tree --- FAIL: TestPythonDriver/semantic/u2_import_subsymbol_alias (0.01s) fixtures.go:213: blacklisted nodes of type "Import" (1) found in the tree FAIL FAIL github.com/bblfsh/python-driver/driver/fixtures 8.925s ```

Maybe wrong Python version? It's still not running tests in Docker, I guess.

juanjux · 2018-07-06T14:31:45Z

@dennwc added a missing file, could you try again?

dennwc · 2018-07-06T14:43:48Z

It's better now, but still getting this diff in fixtures:

juanjux · 2018-07-06T15:04:47Z

@dennwc good catch, should be fixed now.

juanjux · 2018-07-12T10:59:21Z

PR updated with some more positions added to some uast:Identifiers. Some (most notably alias and keywords and function names) won't have positions because they don't have it in the native AST or their position in their native AST is the one of the parent node (e.g. functions).

I was thinking of creating (in Go, I fear) a generic tokenizer pass that could, given a token string and a range of positions, get the token position by the name and the range. For example for a function argument called foo it could be called like: FixPositionOp("foo", argsNodeStartPos, argsNodeEndPos) Then we could use it in the UAST and Semantic UAST to fix most positions of most languages except some cases like several tokens with the same name in the range. But that is a project for another day...

- Added mappings for comments, imports and functions - Change uast.Import for RuntimeImport - Fix for comments and function arguments Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

…rovements - Normalized function arguments, updated annotations to the new format - Removed positions since they don't work when endpos is missing. Added new argument annotations - Remove two now unneeded workarounds (fix in Python ast module and fix in the SDK) - Use uast:Identifiers instead of str for Name fields in semantic objects - FunctionType.Arguments should be an Array - Un-wtfized python's qualified identifiers. Updated SDK. Regenerated tests - Update fixtures - Map normal arg as uast.Argument - Changes to run go test locally - Changes some files to "go test" can be run only running `make native-build`. Unfortunately, this breaks `make test` and `make integration-test`. WIP. - Add missing file - Correct the column of QualifiedIdentifier children - Fix comments - Fix make test Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech> rebase this Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Signed-off-by: Denys Smirnov <denys@sourced.tech>

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

juanjux · 2018-07-12T13:34:46Z

Checking why CI fails.

Signed-off-by: Denys Smirnov <denys@sourced.tech>

juanjux added the enhancement label Jun 19, 2018

juanjux self-assigned this Jun 19, 2018

juanjux requested a review from dennwc June 19, 2018 16:14

dennwc mentioned this pull request Jun 21, 2018

Implement semantic types in drivers bblfsh/sdk#261

Closed

6 tasks

dennwc suggested changes Jun 26, 2018

View reviewed changes

dennwc reviewed Jul 2, 2018

View reviewed changes

dennwc suggested changes Jul 4, 2018

View reviewed changes

dennwc reviewed Jul 4, 2018

View reviewed changes

Juanjo Alvarez added 7 commits July 12, 2018 15:34

High level semantic objects and follow SDK changes

2dea82e

- Added mappings for comments, imports and functions - Change uast.Import for RuntimeImport - Fix for comments and function arguments Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Fixed all findings by the go test checker

9ad5956

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Fix name and path inverter for RuntimeImport

a1781b7

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Add missing file

9e16214

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Fix too eager conversion of the arguments type

36e1194

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Set the boolean on variadic arguments

1994da2

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Denys Smirnov and others added 3 commits July 12, 2018 15:34

run fixture tests in docker

92a2646

Signed-off-by: Denys Smirnov <denys@sourced.tech>

Fix remaining tokens that were not identifiers but should

8c6c675

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

Create output dir in makefile

23bba1d

Signed-off-by: Juanjo Alvarez <juanjo@sourced.tech>

use the new build system

053b402

Signed-off-by: Denys Smirnov <denys@sourced.tech>

dennwc approved these changes Jul 20, 2018

View reviewed changes

don't use the custom base image

bfb5604

Signed-off-by: Denys Smirnov <denys@sourced.tech>

juanjux merged commit 295eb2f into bblfsh:master Jul 20, 2018

Semantic objects #157

Semantic objects #157

Uh oh!

Conversation

juanjux commented Jun 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juanjux commented Jun 27, 2018

Uh oh!

juanjux commented Jun 28, 2018

Uh oh!

juanjux commented Jul 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennwc commented Jul 2, 2018

Uh oh!

juanjux commented Jul 2, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juanjux commented Jul 3, 2018

Uh oh!

juanjux commented Jul 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juanjux Jul 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

juanjux commented Jun 19, 2018 •

edited

Loading

juanjux commented Jul 2, 2018 •

edited

Loading

juanjux commented Jul 4, 2018 •

edited

Loading

juanjux Jul 5, 2018 •

edited

Loading