Merge pull request #161 from adityasharad/merge/1.18-master-050918

Merge rc/1.18 into master.

Author: jbj

change-notes/1.18/analysis-cpp.md

@@ -19,6 +19,7 @@
 | [Nested loops with same variable] | Fewer false positive results | Results where the loop variable is a member of a class or struct now account for the object. |
 | [For loop variable changed in body] | Fewer false positive results | Results where the loop variable is a member of a class or struct now account for the object. |
 | [Local variable hides global variable] | Fewer false positive results | Results for parameters are now only reported if the name of the global variable is the same as the name of the parameter as used in the function definition (not just a function declaration). |
+| [Memory may not be freed] | More correct results | This query now models calls to `realloc` more accurately. |
 | Wrong number of arguments to formatting function | Fewer false positive results | Some false positives related to custom printf-like functions have been fixed. |
 | Wrong number of arguments to formatting function | Clear separation between results of high and low severity | This query has been split into two queries: a high-severity query named [Too few arguments to formatting function] and a low-severity query named [Too many arguments to formatting function]. |
 | [Too few arguments to formatting function] | More correct and fewer false positives results | This query now understands positional format arguments as supported by some libraries. |
@@ -34,5 +35,6 @@
 ## Changes to QL libraries
 
 * Fixes for aggregate initializers using designators:
-** `ClassAggregateLiteral.getFieldExpr()` previously assumed initializer expressions appeared in the same order as the declaration order of the fields, causing it to associate the expressions with the wrong fields when using designated initializers. This has been fixed.
-** `ArrayAggregateLiteral.getElementExpr()` previously assumed initializer expressions appeared in the same order as the corresponding array elements, causing it to associate the expressions with the wrong array elements when using designated initializers. This has been fixed.
+  * `ClassAggregateLiteral.getFieldExpr()` previously assumed initializer expressions appeared in the same order as the declaration order of the fields, causing it to associate the expressions with the wrong fields when using designated initializers. This has been fixed.
+  * `ArrayAggregateLiteral.getElementExpr()` previously assumed initializer expressions appeared in the same order as the corresponding array elements, causing it to associate the expressions with the wrong array elements when using designated initializers. This has been fixed.
+* `Element.getEnclosingElement()` no longer includes macro accesses in its results. To explore parents and children of macro accesses, use the relevant member predicates on `MacroAccess` or `MacroInvocation`.

change-notes/1.18/analysis-javascript.md

@@ -10,7 +10,7 @@
 
 * Modelling of re-export declarations has been improved. This may result in fewer false-positive results for a variety of queries.
 
-* Modelling of taint flow through the array operations `map` and `join` has been improved. This may give additional results for the security queries.
+* Modelling of taint flow through array operations has been improved. This may give additional results for the security queries.
 
 * The taint tracking library recognizes more ways in which taint propagates. In particular, some flow through string formatters is now recognized. This may give additional results for the security queries.
 
@@ -85,6 +85,8 @@
   - [xss](https://github.com/leizongmin/js-xss)
   - [xtend](https://github.com/Raynos/xtend)
 
+* Handling of ambient TypeScript code has been improved. As a result, fewer false positives will be reported in `.d.ts` files.
+
 ## New queries
 
 | **Query**                   | **Tags**  | **Purpose**                                                        |
@@ -103,7 +105,7 @@
 | Comparison between inconvertible types | Lower severity | The severity of this rule has been revised to "warning". |
 | CORS misconfiguration for credentials transfer | More true-positive results | This rule now treats header names case-insensitively. |
 | Hard-coded credentials | More true-positive results | This rule now recognizes secret cryptographic keys. |
-| Incomplete sanitization | More true-positive results | This rule now recognizes incomplete URL encoding and decoding. |
+| Incomplete string escaping or encoding | Better name, more true-positive results | This rule has been renamed to more clearly reflect its purpose. Also, it now recognizes incomplete URL encoding and decoding. |
 | Insecure randomness | More true-positive results | This rule now recognizes secret cryptographic keys. |
 | Missing rate limiting | More true-positive results, fewer false-positive results | This rule now recognizes additional rate limiters and expensive route handlers. | 
 | Missing X-Frame-Options HTTP header | Fewer false-positive results | This rule now treats header names case-insensitively. |
@@ -122,6 +124,6 @@
 
 * HTTP and HTTPS requests made using the Node.js `http.request` and `https.request` APIs and the Electron `Electron.net.request` and `Electron.ClientRequest` APIs are modeled as `RemoteFlowSources`.
 * HTTP header names are now always normalized to lower case to reflect the fact that they are case insensitive. In particular, the result of `HeaderDefinition.getAHeaderName`, and the first parameter of `HeaderDefinition.defines`, `ExplicitHeaderDefinition.definesExplicitly` and `RouteHandler.getAResponseHeader` is now always a lower-case string.
-* New AST nodes for TypeScript 2.9 features have been added.
+* New AST nodes have been added for TypeScript 2.9 and 3.0 features.
 * The class `JsonParseCall` has been deprecated. Use `JsonParserCall` instead.
 * The handling of spread arguments in the data flow library has been changed: `DataFlow::InvokeNode.getArgument(i)` is now only defined when there is no spread argument at or before argument position `i`, and similarly `InvokeNode.getNumArgument` is only defined for invocations without spread arguments.

config/identical-files.json

@@ -1,58 +1,58 @@
 {
     "C++ IR Instruction": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/Instruction.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/Instruction.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/Instruction.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/Instruction.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/Instruction.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/Instruction.qll"
     ],
     "C++ IR IRBlock": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/IRBlock.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/IRBlock.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/IRBlock.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/IRBlock.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/IRBlock.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/IRBlock.qll"
     ],
     "C++ IR IRVariable": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/IRVariable.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/IRVariable.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/IRVariable.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/IRVariable.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/IRVariable.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/IRVariable.qll"
     ],
     "C++ IR FunctionIR": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/FunctionIR.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/FunctionIR.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/FunctionIR.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/FunctionIR.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/FunctionIR.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/FunctionIR.qll"
     ],
     "C++ IR OperandTag": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/OperandTag.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/OperandTag.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/OperandTag.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/OperandTag.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/OperandTag.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/OperandTag.qll"
     ],
     "C++ IR IRImpl": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/IRImpl.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/IRImpl.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/IRImpl.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/IR.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/IR.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/IR.qll"
     ],
     "C++ IR IRSanityImpl": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/IRSanityImpl.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/IRSanityImpl.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/IRSanityImpl.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/IRSanity.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/IRSanity.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/IRSanity.qll"
     ],
     "C++ IR PrintIRImpl": [
-        "cpp/ql/src/semmle/code/cpp/ir/internal/PrintIRImpl.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/PrintIRImpl.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/PrintIRImpl.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/raw/PrintIR.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/PrintIR.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/PrintIR.qll"
     ],
     "C++ SSA AliasAnalysis": [
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/AliasAnalysis.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/AliasAnalysis.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/AliasAnalysis.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/AliasAnalysis.qll"
     ],
     "C++ SSA SimpleSSA": [
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/SimpleSSA.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/SimpleSSA.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SimpleSSA.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SimpleSSA.qll"
     ],
     "C++ SSA IRBlockConstruction": [
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/IRBlockConstruction.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/IRBlockConstruction.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/IRBlockConstruction.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/IRBlockConstruction.qll"
     ],
     "C++ SSA SSAConstruction": [
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/ssa/SSAConstruction.qll",
-        "cpp/ql/src/semmle/code/cpp/ssa/internal/aliased_ssa/SSAConstruction.qll"
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SSAConstruction.qll",
+        "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConstruction.qll"
     ]
 }

cpp/ql/src/Critical/MemoryMayNotBeFreed.ql

@@ -62,8 +62,14 @@ predicate verifiedRealloc(FunctionCall reallocCall, Variable v, ControlFlowNode
       // a realloc followed by a null check at 'node' (return the non-null
       // successor, i.e. where the realloc is confirmed to have succeeded)
       newV.getAnAssignedValue() = reallocCall and
-      node.(AnalysedExpr).getNonNullSuccessor(newV) = verified
+      node.(AnalysedExpr).getNonNullSuccessor(newV) = verified and
       // note: this case uses naive flow logic (getAnAssignedValue).
+
+      // special case: if the result of the 'realloc' is assigned to the
+      // same variable, we don't descriminate properly between the old
+      // and the new allocation; better to not consider this a free at
+      // all in that case.
+      newV != v
     ) or (
       // a realloc(ptr, 0), which always succeeds and frees
       // (return the realloc itself)

cpp/ql/src/semmle/code/cpp/Class.qll

@@ -566,7 +566,7 @@ class Class extends UserType {
    * The alignment of this type in bytes (on the machine where facts were
    * extracted).
    */
-  int getAlignment() { usertypesize(underlyingElement(this),_,result) }
+  override int getAlignment() { usertypesize(underlyingElement(this),_,result) }
 
   /**
    * Holds if this class is constructed from another class as a result of

cpp/ql/src/semmle/code/cpp/File.qll

@@ -162,7 +162,7 @@ abstract class Container extends Locatable, @container {
    *
    * This is the absolute path of the container.
    */
-  string toString() {
+  override string toString() {
     result = getAbsolutePath()
   }
 }

cpp/ql/src/semmle/code/cpp/controlflow/internal/PrimitiveBasicBlocks.qll

@@ -43,15 +43,28 @@ private cached module Cached {
     (not successors_extended(_, node) and successors_extended(node, _))
   }
 
+  /** Holds if `n2` follows `n1` in a `PrimitiveBasicBlock`. */
+  private predicate member_step(Node n1, Node n2) {
+    successors_extended(n1, n2) and
+    not n2 instanceof PrimitiveBasicBlock
+  }
+
+  /** Returns the index of `node` in its `PrimitiveBasicBlock`. */
+  private int getMemberIndex(Node node) {
+    primitive_basic_block_entry_node(node) and
+    result = 0
+    or
+    exists(Node prev |
+      member_step(prev, node) and
+      result = getMemberIndex(prev) + 1
+    )
+  }
+
   /** Holds if `node` is the `pos`th control-flow node in primitive basic block `bb`. */
   cached
   predicate primitive_basic_block_member(Node node, PrimitiveBasicBlock bb, int pos) {
-    (node = bb and pos = 0)
-    or
-    (not (node instanceof PrimitiveBasicBlock) and
-     exists (Node pred
-     | successors_extended(pred, node)
-     | primitive_basic_block_member(pred, bb, pos - 1)))
+    pos = getMemberIndex(node) and
+    member_step*(bb, node)
   }
 
   /** Gets the number of control-flow nodes in the primitive basic block `bb`. */

cpp/ql/src/semmle/code/cpp/dataflow/internal/DataFlowPrivate.qll

@@ -128,6 +128,7 @@ predicate readStep(Node node1, Content f, Node node2) {
  * numeric conversions, and otherwise the erasure is used.
  */
 RefType getErasedRepr(Type t) {
+  suppressUnusedType(t) and
   result instanceof VoidType // stub implementation
 }
 
@@ -140,6 +141,8 @@ predicate compatibleTypes(Type t1, Type t2) {
   any() // stub implementation
 }
 
+private predicate suppressUnusedType(Type t) { any() }
+
 //////////////////////////////////////////////////////////////////////////////
 // Java QL library compatibility wrappers
 //////////////////////////////////////////////////////////////////////////////

cpp/ql/src/semmle/code/cpp/ir/IR.qll

@@ -1 +1,3 @@
-import internal.IRImpl
+// Most queries should operate on the aliased SSA IR, so that's what we expose
+// publically as the "IR".
+import implementation.aliased_ssa.IR

cpp/ql/src/semmle/code/cpp/ir/IRSanity.ql

@@ -5,4 +5,4 @@
  * @id cpp/ir-sanity-check
  */
 
-import internal.IRSanityImpl
+import implementation.aliased_ssa.IRSanity