From ec0c1dc70e5b04cb7fe5c7e067f376d5a940ec2a Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Fri, 7 Jun 2019 15:05:59 +0100 Subject: [PATCH 1/2] Python: Change note for new points-to implementation. --- change-notes/1.21/analysis-python.md | 29 ++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/change-notes/1.21/analysis-python.md b/change-notes/1.21/analysis-python.md index 80269f5f4e12..7c8fa1b161c3 100644 --- a/change-notes/1.21/analysis-python.md +++ b/change-notes/1.21/analysis-python.md @@ -3,8 +3,33 @@ ## General improvements -> Changes that affect alerts in many files or from many queries -> For example, changes to file classification +Points-to analysis has been re-implemented to support more language features and provide better reachability analysis. +The new implementation adds the following new features: + +* Non-local tracking of bound methods and instances of `super()` +* Superior analysis of conditionals and thus improved reachability analysis. +* Superior modelling of descriptors, for example, classmethods and staticmethods. +* Superior tracking of values through parameters, especially `*` arguments. + +A new object API has been provided to complement the new points-to implementation. +A new class `Value` replaces the old `Object` class. The `Value` class has a simpler and more consistent API compared to `Object`. +Some of the functionality of `FunctionObject` and `ClassObject` has been added to `Value` to reduce the number of casts to more specific classes. +For example, the QL to find calls to `os.path.open` has changed from +`ModuleObject::named("os").attr("path").(ModuleObject).attr("join").(FunctionObject).getACall()` +to +`Value::called("os.path.join").getACall()` + +The old API is now deprecated, but will be continued to be supported for at least another year. + +### Impact on existing queries. + +As points-to underpins many queries and provides the call-graph and reachability analysis required for taint-tracking, many queries will have additional results and some may have fewer results. + +New results are a result of the improved reachability analysis and non-local tracking of bound-methods. +Removed results are a result of more precise tracking of values through `*` arguments. +The expectation is that number of true positives will increase and the number of false negatives will decline. +However, this is new code and may still contain errors. + ## New queries | **Query** | **Tags** | **Purpose** | From b7141ad998c5c7f74ed05868ae214f0f35de2fee Mon Sep 17 00:00:00 2001 From: Mark Shannon Date: Mon, 10 Jun 2019 10:56:28 +0100 Subject: [PATCH 2/2] Python: Clarify change note. --- change-notes/1.21/analysis-python.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/change-notes/1.21/analysis-python.md b/change-notes/1.21/analysis-python.md index 7c8fa1b161c3..970b86976f42 100644 --- a/change-notes/1.21/analysis-python.md +++ b/change-notes/1.21/analysis-python.md @@ -23,12 +23,13 @@ The old API is now deprecated, but will be continued to be supported for at leas ### Impact on existing queries. -As points-to underpins many queries and provides the call-graph and reachability analysis required for taint-tracking, many queries will have additional results and some may have fewer results. +As points-to analysis underpins many queries, and provides the call-graph and reachability analysis required for taint-tracking, the results of many queries may change. -New results are a result of the improved reachability analysis and non-local tracking of bound-methods. -Removed results are a result of more precise tracking of values through `*` arguments. -The expectation is that number of true positives will increase and the number of false negatives will decline. -However, this is new code and may still contain errors. +The improved reachability analysis and non-local tracking of bound methods may identify new results. +The increased precision in tracking of values through `*` arguments may remove false positive results. + +Overall the number of true positive results should increase and the number false negative results should decline. +We welcome feedback on the new implementation, particularly any surprising changes in results. ## New queries