package commons

  1. Overview
  2. Docs
Yet another set of common utilities

Install

Dune Dependency

semgrep.dev Readme LGPL-2.1-only License Edit opam file Versions (2)

Authors

  1. Y
    Yoann Padioleau <pad@r2c.dev>

Maintainers

  1. Y
    Yoann Padioleau <pad@r2c.dev>

Sources

1.5.5.tar.gz
md5=3536f6162fec35b13072e8fd84c4e985
sha512=89cdda71fe2036b94ae6142f55f1a55d8d142ec3c9e49206b5a91f5f217984fd5b7564a165011dd845f5b50f32a29d3b969dc404c41d6e140a81cb7106c1ac7b

CHANGELOG.md.html

CHANGELOG.md

Changelog

This project adheres to Semantic Versioning.

1.5.1 - 2023-01-20

Added

  • Python: Constant propagation will now recognize the idiom cond and X or Y, as well as True and X and False or X. So e.g. cond and "a" or "b" will be identified as a constant string. (gh-6079)

  • Julia: Julia is now experimental (pa-2366)

Changed

  • DeepSemgrep is now Semgrep PRO! To install the Semgrep PRO engine run: semgrep install-semgrep-pro. This engine is still invoked using the --deep flag, but please expect changes to the CLI in the near future. The new Semgrep PRO engine adds support for Apex! (pa-2389)

Fixed

  • New 'transform:' field in extract mode rules, with 'concat_json_string_array' option useful to extract python code from jupyter notebooks. Thanks to Jose Selvi for his contribution! (gh-4477)

  • Java: Fixed regression introduced in 0.123.0 that could cause a private class attribute to be incorrectly regarded as a constant. (gh-6793)

  • Make $F(x) match eval(x). Previously, eval was special-cased and metavariable function call patterns would not match it. (gh-6877)

  • DeepSemgrep: Enabled --dataflow-traces by default when --deep is specified (pa-2274)

  • In rare situations, mainly in DeepSemgrep and related to naming bugs, the use of symbolic propagation could make Semgrep fall into an infinite loop during matching. This has been fixed by bounding the number of times that Semgrep can follow symbolically-propagated values. (pa-2324)

  • CLI: Made an error message for when two autofix matches overlap have a more helpful message, as well as be displayed as a debug message. (pa-2393)

  • CLI: Made the warning message when using Semgrep Pro more friendly when logged in (pa-2396)

1.3.0 - 2023-01-04

Changed

  • Removed the poor support for reading dependencies from pom.xml files. Instead semgrep will try to read dependencies from a maven_dep_tree.txt file, which can be generated using the command mvn dependency:tree -DoutputFile=maven_dep_tree.txt (sc-pom)

Fixed

  • Use the GitHub REST API when possible to compute the merge base for semgrep ci, improving performance on shallow clones of large repositories. (gha-mergebase)

  • YAML: Fixed a bug where metavariables matching YAML double-quoted strings would not capture the entire range of the string, and would not contain the double-quotes. Also added the ability to properly use patterns like "$FOO", which will unpack the contents of the matched string. (pa-2332)

  • Fixed a race condition related to the parsing cache that could lead to internal errors (pa-2335)

  • YAML: Fixed a bug where literal or folded blocks would not be parsed properly.

    So for instance, in:

    key: |
  string goes here

    A metavariable matching the contents of the string value might not be correct. (pa-2347)

  • Julia: Greatly improved parsing support (pa-2362)

1.2.1 - 2022-12-16

Fixed

  • Go: fix NoTokenLocation for metavariables matching function type without an argument (e.g. func()) (gh-6715)

  • typed-metavariables: handle ternary expressions so we can type correctly expressions like foo(cond ? new A() : this.a) (pa-2328)

  • Reverted a change which caused findings with different sources (but the same sink) to be deduplicated. This would cause findings which looked identical in range and data, but had different taint traces. (pa-2336)

1.2.0 - 2022-12-15

Fixed

  • Fixed rare crash that could occur due to stale file caches when temp file names overlap (cache-invalidation)

  • PHP: support metavariables in string (as in foo("$VAR")) (gh-6311)

  • Java: support static block patterns (gh-6366)

  • Rust: parse correctly scoped identifier in constructor (gh-6594)

  • Java: support super(...) patterns (gh-6638)

  • C#: bugfix on bad interaction between -fast and the internal code generated for LINQ queries (gh-6666)

  • Java: support the Java 10 'var' keyword by not using 'var' as a valid type when using typed metavariables. (gh-6672)

  • When matching module specifiers in imports, prevent metavariables from capturing quotes. (gh-6674)

  • Swift: support complex expression in switch case (gh-6704)

  • Constant propagation: Fixed a bug where constant propagation would only run within functions. Now, it runs on the top-level of the program as well. (pa-1656)

  • DeepSemgrep: Fixed a bug where imports which reached type names (among other things) would not resolve properly (pa-2260)

  • DeepSemgrep: Fixed a bug which caused Java functions with interfaces to wipe taint (instead of propagating taint by default) (pa-2265)

  • Fix matching of qualified names. For example, new $X.Foo() will now match new a.b.Foo(). (pa-2296)

  • DeepSemgrep: Fix regression in taint-mode introduced by Semgrep v1.1 that caused some findings to be missed. Also, DeepSemgrep will assume, for now, that a method call on a tainted object is always tainted. (pa-2304)

  • Improved matching behavior on JS require calls (require-match)

1.1.0 - 2022-12-05

Added

  • JSON output: Added a max_memory_bytes field to the semgrep --time output which corresponds to the amount of memory allocated during the OCaml phase of Semgrep. This is useful for telemetry purposes. (pa-2075)

Changed

  • Don't print out summary of blocking rules unless running with semgrep ci subcommand (gh-6651)

Fixed

  • taint-mode: In 0.94.0 we made that when a pattern-source (or pattern-sanitizer) matched a variable exactly, this was understood as that variable being tainted (sanitized, resp.) by side-effect. For example, given tainted(x) we would taint x by side-effect, and subsequent occurrences of x were also considered tainted. This allowed to write rules like c.lang.security.use-after-free.use-after-free in a very succint way, and it also addressed some limitations of the workarounds that were being used to simulate this until then.

    This worked well initially, or so we thought, until in 0.113.0 we added field-sensitivity to taint-mode, and in subsequent versions we made sources and sanitizers apply by side-effect to more kinds of l-values than just simple variables. It was then that we started to see regressions that were fairly unintuitive for users. For example, if $_GET['foo'] was a taint source, this would make $_GET itself to be tainted by side-effect, and a subsequent expression like $_GET['bar'] was also considered tainted.

    We now correct the situation by adding the by-side-effect option to sources and sanitizers, and requiring this option to be explicitly enabled (that is, by-side-effect: true) in order to apply the source or the sanitizer by side-effect. Otherwise, the default is that sources and sanitizers matching l-values apply only to the precise occurrences that they match. (pa-1629)

  • taint-mode: Fixed matching of pattern-sinks to be more precise, so that e.g. it will no longer report sink(ok1 if tainted else ok2) as a tainted sink, as the expression passed to the sink is actually not tainted. (pa-2142)

  • CLI: Separated experimental rules from normal rules in semgrep --debug output. (pa-2159)

  • Taint: Fixed an issue where findings with the same sink would be identified as the same, and cause only one of them to be reported, even if they had different sources. (pa-2208)

  • DeepSemgrep: When the "DeepSemgrep" setting is enabled in Semgrep App, semgrep ci will try to run the analysis using the DeepSemgrep engine. But if this engine was not installed, semgrep ci failed. Now semgrep ci will automatically try to install DeepSemgrep if it is not already present. Note that, if DeepSemgrep is already installed, semgrep ci does not attempt to upgrade it to a newer version. (pa-2226)

  • CLI: Made the number of jobs when using semgrep --deep default to 1. (pa-2231)

  • Autofix: If multiple autofixes are targeting an overlapping range, then one of them is picked arbitrarily to occur, to prevent autofixes which may produce incorrect code. (pa-2276)

  • DeepSemgrep: Time data now outputs properly when running semgrep --deep --time (pa-2280)

  • DeepSemgrep: Added a message which suggests that users update their version of DeepSemgrep, if the DeepSemgrep binary crashes (pa-2283)

  • Yarn 2 parse failure on versions like @storybook/react-docgen-typescript-plugin@canary. This is only present as some kind special version range specifier and never appears as a concrete version. It would only be used to check if the dependency was in the manifest file, so we just parse the version as "canary" Yarn 2 parse failure on versions like @types/ol-ext@npm:@siedlerchr/types-ol-ext@3.0.6 Yarn 2 parse failure on versions like resolve@patch:resolve@^1.1.7#~builtin<compat/resolve>. These are now just ignored, as they appear to always come with a non-patch version as well. (sc-406)

1.0.0 - 2022-12-01

Added

  • DeepSemgrep: If you have a Team tier account in Semgrep App, and you enable the DeepSemgrep setting, then semgrep ci will automatically run the DeepSemgrep engine instead of the OSS engine on full scans (but not in PR scans). (pa-2226)

0.123.0 - 2022-11-29

Added

  • CLI: Added deep traces to --dataflow-traces (pa-2116)

Changed

  • Reachable Supply Chain findings will no longer block pull requests when using semgrep ci. Note that unreachable findings have been non-blocking already. (sca-nonblocking)

Fixed

  • Fix matching issue related to JS imports with multiple imported values (gh-5305)

  • DeepSemgrep: Keep only the shortest trace originating from a taint source. This also prevents falling into infinite loops when inferring taint signatures for mutually recursive procedures. (pa-2224)

  • taint-mode: Improved taint tracking for array-like accesses. Previously, if x.a.b[i].c got tainted, Semgrep would track x.a.b as tainted, and thus x.a.b[i].d would be incorrectly considered as tainted too. Now Semgrep will do the right thing and track x.a.b[*].c as tainted, and x.a.b[i].d will not be considered tainted. (pa-2225)

  • Java: private, singly-assigned class variables now permit constant propagation (pa-2230)

  • JS/TS: Allow dependencies to @Injectable and @Component classes in Angular JS to be visible outside the scope of the constructor. (pa-2233)

  • Fix matching issue related to Python imports with multiple imported values (python-imports)

  • Supply Chain findings from a yarn.lock lockfile were marked as 'transitive' when we couldn't find the matching package.json file. These findings will now be marked as having 'unknown' transitivity. (sc-425)

  • Make $X(...) match this() and super(). (this-match)

0.122.0 - 2022-11-16

Fixed

  • DeepSemgrep: Added installation path for DeepSemgrep on M1 machines (pa-2153)

  • Correctly handle optional name field in yaml1 lockfiles (parse_yaml)

0.121.2 - 2022-11-10

Fixed

  • Fix yaml excessive mapping captures (gh-5698)

  • Rule syntax: Allow pattern-not, pattern-inside, and pattern-not-inside to take in arbitrary patterns (such as patterns, pattern-either, and friends) (pa-1723)

  • Kotlin: Fixed bug where constructor invocation with a trailing comma would not parse (pa-1988)

  • Constant propagation: Type casts and parenthesized expressions (in Go) can now be symbolically propagated. (pa-2054)

  • CLI: Added a fix preventing findings from experimental rules being displayed. Experimental rules also now no longer run silently. (pa-2121)

0.121.1 - 2022-11-08

No significant changes.

0.121.0 - 2022-11-07

Changed

  • taint-mode: Semgrep can now track taint through l-values of the form this.x. It will also be more precise when tracking taint on l-values involving an array access, previously if x.a[i] was tainted, then x itself was tainted; now only x.a will be considered tainted. (pa-2086)

Fixed

  • JS: correctly parse exported function pattern (gh-6218)

  • Terraform: Can now analyze constant-propagated values of variables with metavariable-comparison and friends (pa-2088)

0.120.0 - 2022-11-02

Added

  • Fail gracefully and print error message when running in unsupported Linux aarch64/arm64 environment (arm-fail)

  • Added 'fingerprints' field to results in sarif output (gh-5729)

  • Add dataflow traces as 'codeFlows' object in SARIF output (gh-6367)

  • Fail immediately if semgrep tries to run a supply chain rule with an invalid version range specifier (ssc-fail-early)

  • Added Supply Chain support for requirements.txt lockfiles (with requirement.in manifest files) and Yarn 2/3 lockfiles (ssc-yarn-req)

Changed

  • Change default behavior of CircleCI configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2434)

  • Change default behavior of Buildkite CI configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2435)

  • Change default behavior of bitbucket CI configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2436)

Fixed

  • Handle unexpected exceptions when performing AST-based autofix. (autofix-exception)

  • Fix an issue that could cause AST-based autofix to fail on autofixes containing statements. (autofix-statement-exception)

  • Fix an issue preventing AST-based autofix from running in the presence of focus-metavariable. (focus-metavariable-autofix)

  • Implement string literal metavariables in Python AST-based autofix (gh-3648)

  • Go: parse correctly generic types from other packages (gh-5799)

  • Fix parsing of dot access ellipsis in PHP patterns (e.g. $x-> ... ->bar()). (gh-6183)

  • JS/TS: Allowed parsing of patterns using the optional chaining "?." syntax. (gh-6201)

  • Dockerfile language: Add support for RUN options such as RUN --mount=type=$TYPE,target=$TARGET .... (gh-6353)

  • taint-mode: Fixed a bug in the experimental taint-labels feature that caused labels to be incorrectly applied to dot accesses. For instance, if a pattern-source that requires label A and adds label B matches a dot-access expression like x.a, the field a will get the label B even if it does not carry label A as required. (gh-6355)

  • Use AST-based autofix when possible for JS autofixes. This is more likely to lead to correct output, especially for complicated fixes. (js-autofix)

  • taint-mode: Fixed regression in 0.113.0, after field sensitivity support was added, that broke existing behavior when a prefix in a chain of dot-accesses such as x.a.b was specified as a source/sanitizer/sink. For example, if x had been previously tainted, then we encountered sink(x.a.b) where x.a matched a sanitizer, there was a finding reported because x.a.b was incorrectly considered tainted. (pa-1928)

  • JS/TS: Fixed a parsing bug where special identifiers were parsed differently in patterns (pa-2030)

  • Language server now appropriately applies regex fixes (vscode-regex)

0.119.0 - Skipped

Version 0.119.0 of Semgrep was intentionally skipped. Version 0.120.0 immediately follows version 0.118.0.

0.118.0 - 2022-10-19

Added

  • Taint mode will now track taint coming from the default values of function parameters. For example, given def test(url = "http://example.com"):, if "http://example.com" is a taint source (due to not using TLS), then url will be marked as tainted during the analysis of test. (gh-6298)

  • taint-mode: Added two new rule options that help minimizing false positives.

    First one is taint_assume_safe_indexes, which makes Semgrep assume that an array-access expression is safe even if the index expression is tainted. Otherwise Semgrep assumes that e.g. a[i] is tainted if i is tainted, even if a is not. Enabling this option is recommended for high-signal rules, whereas disabling it may be preferred for audit rules. Currently, it is disabled by default for pure backwards compatibility reasons, but this may change in the near future after some evaluation.

    The other one is taint_assume_safe_functions, which makes Semgrep assume that function calls do NOT propagate taint from their arguments to their output. Otherwise, Semgrep always assumes that functions may propagate taint. This is intended to replace not conflicting sanitizers (added in v0.69.0) in the future. This option is still experimental and needs to be complemented by other changes to be made in future releases. (pa-1541)

Changed

  • Ignore the .npm/ directory by default in Semgrep scans (gh-6315)

  • The --scan-unknown-extensions option is now set to false by default. This means that from now on --skip-unknown-extensions is the default. This is an important change that prevents many errors when using Semgrep in a pre-commit context or in CI. (pa-1932)

Fixed

  • Add autodetection for pull request numbers for Azure Pipelines. If SEMGREP_PR_ID is set, override the autodetection. (app-2083)

  • Fixed an autofix regression that caused Semgrep to fail to replace metavariables in string literals, e.g. foo("xyz $X"). (autofix-string-metavar)

0.117.0 - 2022-10-12

Added

  • taint-mode: It is now possible to use pattern-propagators to propagate taint through higher-order iterators such as forEach in Java. For example:

      pattern-propagators:
    - pattern: $X.forEach(($Y) -> ...)
      from: $X
      to: $Y
``` (gh-5971)

Changed

  • Change default behavior of Travis CI configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT, SEMGREP_PR_ID), use it before falling back on autodetection.

Fixed

  • Scala: Fixed a bug where generators would not parse if newlines were present, in certain cases (pa-1902)

  • Fixed bug where nested dependencies in package-lock.json files were not detected (sc-247)

  • Removed Gradle as a separate supply chain ecosystem. Maven rules now work on Gradle projects (sc-256)

  • Lockfiles are no longer subject to size filtering during file targetting, so very large lockfiles can now generate unreachable findings (sc-293)

0.116.0 - 2022-10-06

Added

  • Added support for named arguments in taint tracking. This is only relevant for DeepSemgrep users. (pa-1886)

Changed

  • Change default behavior of Jenkins CI configurations. If a user manually sets their environment variables (i.e. SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2432)

  • Change default behavior of Azure Pipelines configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2433)

  • taint-mode: Removed experimental poor-man's support for wrapper functions around taint sources. This was an early experiment to make Semgrep inter-procedural, but it was later abandoned in favor of DeepSemgrep. (pa-1838)

  • Disabled Bloom filter optimization by default, due to undesired interactions with constant and symbolic propagation, while it appears to not provide a net major performance benefit (nowadays). If you do notice a significant drop in performance after this change, please let us know. (pa-1927)

  • Semgrep-core will no longer accept a rule file containing only one rule object, rules must be given in an array unde the rules: key. This change does not affect Semgrep CLI which never accepted that relaxed format. (pa-1931)

  • Changed command line flag for supply chain scans from --sca to --supply-chain. Correspondinly changed --config sca to --config supply-chain (sca-ssc)

Fixed

  • Change default behavior of Jenkins CI configurations. If the SEMGREP_REPO_URL is set, use it. Otherwise, default to autodetection. (app-2406)

  • fix: Ensure the docker image uses the latest base packages (docker)

  • Fixed symbolic propagation of the new operator, that had been broken since version 0.98.0. You can again e.g. use the pattern new A().foo() to match a.foo(), with a = new A(). (gh-6161)

  • Some crypto code like hashing algorithms can lead to a very large amount of symbolically propageted values, which previously caused Semgrep's Bloom filter to hang. (pa-1920)

  • taint-mode: It is now possible for this or this.x to be a source of taint. (pa-1929)

  • taint-mode: Fixed a bug that made Semgrep miss taint findings when the sink was located inside an if condition or a throw (aka raise) expression/statement. (pa-1933)

  • Start to use the AST to render autofixes in some specific cases. As this is extended, autofix will become more correct and more powerful. (ast-autofix)

  • Fixed a parser error in some package-lock.json files (sca-parse-error)

0.115.0 - 2022-09-27

Added

  • Adds support for a .semgrepconfig file. Users can add metadata (such as a list of tags) to the .semgrepconfig YAML file which will automatically be assigned to the project. (app-2112)

  • Modify the CLI output to separate non-blocking and blocking findings and show a list of the blocking rules that fired. (app-2306)

Changed

  • generic mode: allow text input without human-readable indentation up to 500 bytes. This value is subject to change. This relaxing is intended to facilitate testing where someone might copy-paste a long line without a trailing newline. Semgrep users should not expect files that are not human-readable to be processed in semgrep's generic mode, or in any mode for the matter. (gh-6071)

  • Changed behavior for renamed files on diff scans (scans in which a baseline ref is provided). Semgrep will not show old issues to developers when they rename a file now. (gh-6157)

Fixed

  • Fixed nondeterministic failure of test_api test due to invalid settings file by configuring home directory to temporary directory. (app-2166)

  • Change default behavior of Jenkins CI configurations. If the SEMGREP_REPO_NAME environment variable is set, use it. Otherwise, default autodetection. (app-2331)

  • Dockerfile mode: Fix failure to match where image name and image alias should be the same. The problem was due to some names and identifiers being fragmented due to parsing rules and not pieced back together. (gh-5229)

  • Scala: add support for ellipsis in match body (e.g., $X match { ... }) (gh-6131)

  • Added a fix for a bug involving parsing of TS imports, where they were not allowed to appear as patterns to a rule. (pa-1910)

0.114.0 - 2022-09-19

Added

  • Add functionality to exclude rules by id passing it by cli flag --exclude-rule (cli-2530)

  • Fixes https://github.com/returntocorp/semgrep/issues/5686. You can now have multiple metavariables under focus-metavariable, which allows Semgrep to highlight the values matched by multiple metavariables more easily in certain circumstances. See the gist in the description of the original issue for an example. (gh-5686)

Fixed

  • C++: support ellipsis in right-hand-side of an assignment (gh-1923)

  • Rust: support ellipsis in struct declarations (gh-3759)

  • Fixed incorrect stripping of '$' (literal dollar sign) in regexps used in the context of metavariable-regex. (gh-5987)

  • Solidity: support constructor and modifier patterns (gh-6053)

  • C#: support for metavariable ellipsis (e.g., $...ARGS) in arguments (gh-6065)

  • Rust: support ellipsis inside module mody (gh-6066)

  • Hold references to NamedTemporaryFile objects while their corresponding temporary files are still in use by the core runner. Failure to explicitly hold references to these objects on some Python implementations, such as PyPy, results in them sometimes being garbage-collected during processing. This, in turn, triggers removal of the temp files while they are still in use by the core runner or the worker subprocesses, resulting in various crashes and processing failures. (gh-6100)

  • Swift: Fix parsing of statement ellipsis without a preceding semicolon (pa-1809)

0.113.0 - 2022-09-15

Added

  • Adds backwards-compatibility with older versions of semgrep-app. Only relevant for customers with on-prem versions of the app. (gh-6098)

  • taint-mode: Experimental support for basic field-sensitive taint tracking. Semgrep can now track x.a and x.b separately, so that e.g. x.a can be tainted at the same time as x.b is clean, hence sink(x.a) would produce a finding but sink(x.b) would not. It is also possible for x to be tainted while x.a is clean. We expect this to have an net positive effect by reducing false positives. (pa-1278)

  • Update the supply chain API to include information about the transitivity of matched dependencies (sca-199)

Fixed

  • Resolve imported classes when there are multiple chained classes. For example, if you import world.Hello, and create a new Hello.internal_class(), you can match that with new world.Hello.internal_class(). (gh-6001)

  • Java: Correctly parse ellipsis in the body of top-level constructor patterns with privacy modifiers (e.g. public Foo() { ... }) (gh-6051)

  • semgrep --test now fails when encountering a parsing error in target code. (gh-6068)

  • Allowed metavariable-comparison to make use of the not in operator. (gh-6072)

  • Fixed parsing of Rust impl definitions (gh-6078)

  • Fixed a TypeError: unbound method set.intersection() needs an argument crash that occurred when all of a scan's rules were multilang (regex or generic). (gh-6093)

0.112.1 - 2022-09-08

Fixed

  • Fixed a regression introduced with the previous release, involving a bug with pattern-inside. (gh-6059)

0.112.0 - 2022-09-07

Added

  • JS/TS: Allow standalone switch cases as patterns (e.g. case 5: ...) (pa-1788)

  • Symbolic propagation: Added propagation of all variables which are assigned to a single time. This will affect global variables and class attributes, in particular. (pa-1821)

Changed

  • Use new semgrep-app endpoint that combines the two POST requests to upload findings and ignored findings into one POST request. (app-1446)

Fixed

  • Update git url parser to support optional "org" after hostname. Example: https://some.enterprise.scm/myorg/owner/repo. (app-2202)

  • Fix an incorrect autofix application when the fix includes Python f strings (gh-2995)

  • Fix matching and autofix with bare tuples in Python array accesses (e.g. x[1, 2]) (gh-3387)

  • Ruby: A pattern of the form /.../ can now match any regexp, including regexp templates such as /hello #{name}/. (gh-5147)

  • Fix parsing of Java constructors with privacy modifiers as patterns, e.g. the pattern public Foo() { } (gh-5558)

  • Java: correctly parse class literal patterns (gh-6002)

  • Solidity: correctly parse try statements (gh-6031)

  • Python: Now support match statements (pa-1739)

  • Fixed bug in constant propagation that caused incorrect constants to be inferred in the presence of subtraction. (pa-1846)

0.111.1 - 2022-08-23

Changed

  • Previously, the following error message appears when metrics are not uploaded within the set timeout timeframe:

    Error in send: HTTPSConnectionPool(host='metrics.semgrep.dev', port=443): Read timed out. (read timeout=3)

    As this causes users confusion when running the CLI, the log level of the message is reduced to appear for development and debugging purposes only. Note that metrics are still successfully uploaded, but the success status is not sent in time for the curent timeout set. (app-1398)

Fixed

  • taint-mode: Fixed the translation from Generic to IL for expressions like "some string".concat(x). Previously, when x was tainted, the concat expression was not recognized as tainted and this caused false negatives. (pa-1787)

0.111.0 - 2022-08-22

Added

  • Introduced experimental support for Swift (gh-2232)

  • Add configuration options for using a tree-sitter library installed anywhere on the system. (gh-5944)

  • Updated the supply chain finding API:

    • The API is now typed and defined entirely in semgrep_output_v0.atd

    • Supply chain findings now have only one dependency match, not a list, and only one resolved url

    • Supply chain findings now have a field called reachable and reachability_rule, which indicate if the finding is reachable, and whether or not it was generated by a reachability rule (rule that had a semgrep pattern)

    • Supply chain findings now include a schema version

    • The complete finding information sent to semgrep app now includes a mapping from lockfile paths to the number of dependencies that were present in that lockfile (sca-197)

Fixed

  • When a YAML rule file had a string that contained an ISO timestamp, that would be parsed as a datetime object, which would then be rejected by Semgrep's rule schema validator. This is now fixed by keeping strings that contain an ISO timestamp as strings. (app-2157)

  • When parsing PHP with tree-sitter, parse $this similar to pfff, as an IdSpecial. This makes it possible to match $this when the pattern is parsed with pfff and the program with tree-sitter. (gh-5594)

  • Parse die() as exit() in tree-sitter PHP. This makes pfff and tree-sitter parse die() in the same way. (gh-5880)

  • All: Applied a fix so that qualified identifiers can unify with metavariables. Notably, this affected Python decorators, among others. (pa-1700)

  • Fixed a regression in DeepSemgrep after the experimental taint labels feature was introduced in 0.106.0. This prevented DeepSemgrep from reporting taint findings when e.g. the sink was wrapped by another function. (pa-1750)

  • Fixed metavariable unification in JSON when one of the patterns is a single field. (pa-1763)

  • Changed symbolic propagation such that "redundant" matches are no longer reported as findings. For instance:

    def foo():
  x = g(5)
  f(x)

    If we are looking for the pattern g(5), we should not match on line 3, since we will match on line 2 anyways, and this is just repeating information that we already know.

    This patch changes it so that we do not match on line 3 anymore. (pa-1772)

  • Semgrep now passes -j to DeepSemgrep engine so --deep became noticeably faster. (pa-1776)

  • taint-mode: Due to a mistake in the instantiation of a visitor, named function definitions were being analyzed twice! This is now fixed and you may observe significant speed ups in some cases. (pa-1778)

  • Extract mode: fixed a possible exception in normal usage introduced due to changes in handling of search/taint rules. (pa-1786)

  • Changed the fail-open message body (pm-194)

Infra/Release Changes

  • GHA Runner macos-12 is unreliable and has begun failing without a clear explanation as to why: this downgrades to macos-11, since 10.15 is to be depracted ~10 from now. (devop-609)

  • Keep the tree-sitter library inside a local folder rather than requiring a global installation. (gh-2956)

0.110.0 - 2022-08-15

Changed

  • Parse several built-in PHP functions in the same way in pfff and tree-sitter. This makes it possible to match exit, eval, empty and isset, even if the pattern is parsed with pfff and the PHP file with tree-sitter. (gh-5382)

  • Use new semgrep-app endpoints that are pure GET with no side-effects and pure POST (app-2001)

Fixed

  • Skip fail-open for exit code 1 (app-2073)

0.109.0 - 2022-08-11

Changed

  • semgrep ci now defaults to fail open and will always exit with exit code 0, which is equivalent to passing --suppress-errors. To disable this behavior, you can pass --no-suppress-errors and semgrep will behave as it did previously, surfacing any exit codes that may result. (app-1951)

Fixed

  • taint-mode: Taint traces (--dataflow-traces) should no longer report "strange" intermediate variables when there are record accesses involved. This happened e.g. if foo was a tainted record and the code accessed some of its fields as in foo.bar.baz. This was related to the use of auxiliary variables in the Dataflow IL. These variables got tainted, but they had real tokens attached corresponding to the dot . operator. Now we do not include these variables in the taint trace. (pa-1672)

Infra/Release Changes

  • GHA runner-image macos-10.15 is deprecated and will be unsupported by 30AUG2022. We've tested and can upgrade to macos-12 to avoid issues with brownouts or end of support. (devop-586)

0.108.0 - 2022-08-03

Added

  • Metrics now include language-aggregated parse rates (files, bytes). The purpose of this is to help drive parsing improvements more intelligently. See PRIVACY.md for more details. (pa-1678)

Changed

  • Updated SCA finding generation so that the following hold:

    • One SCA finding per vulnerable dependency. If one rule matches multiple dependencies in one lockfile, that will produce multiple findings. This still needs to be codified in the typed interface

    • No findings in files that were not targeted. If foo.py depends on Pipfile.lock, and foo.py is targeted but Pipfile.lock is not, then we can produce reachable findings in foo.py but not non-reachable findings in Pipfile.lock. If Pipfile.lock is included in our targets then we can produce non-reachable findings inside of it

    • No massive single scan for lockfiles. (sca-127)

Fixed

  • Fixed issue when scan fails due to pending changes in submodule. (cli-272)

  • Semgrep CI now accepts more formats of git url for metadata provided to semgrep.dev and lets the user provide a fallback for repo name (SEMGREP_REPO_NAME) and repo url (SEMGREP_REPO_URL) if they are undefined by CI. (cli-280)

  • Fixed a crash that occurred when reporting results when join mode and taint mode were used together (gh-5839)

  • JS: Allowed decorators to appear in Semgrep patterns for class methods and fields. (pa-1677)

  • Quick fix for a regression introduced in 0.107.0 (presumably by taint labels) that could cause some taint rules to crash Semgrep with:

    Invalid_argument "output_value: abstract value (Custom)" (pa-1724)

  • Increase timeout for network calls to semgrep.dev from 30s to 60s (timeout-1)

0.107.0 - 2022-07-29

Added

  • Added metadata in App-connected scans to report extensions of files that do not match the language of any enabled rules in order to enable more effective language prioritization while developing new rules. (app-1354)

  • Support fail-open in CI: adds --suppress-errors/--no-suppress-errors (defaults to --no-suppress-errors) (cli-254)

  • New language Elixir with experimental support. (gh-3698)

  • Kotlin: support for ellipsis in field access (e.g., obj. ... .bar()) (gh-5819)

  • Changed semgrep-core so that it can now be run with -rules on .yaml files which do not have a top-level rules: ... key. This means you can now copy paste from the playground editor directly into a .yaml file for use with semgrep-core. (implicit-rules-sc-core)

  • Add experimental support for taint labels, that is the ability to attach labels to different kinds of taint. Both sources and sinks can retrict what labels are present in the data that passes through them in order to apply. This allows to write more complex taint rules that previously required ugly workarounds. Taint labels are also useful for writing certain classes of typestate analyses (e.g., check that a file descriptor is not used after being closed). (pa-1362)

  • Introduced the --dataflow-traces flag, which directs the Semgrep CLI to explain how non-local values lead to a finding. Currently, this only applies to taint mode findings and it will trace the path from the taint source to the taint sink. (pa-1599)

Changed

  • Made breaking changes to the dataflow_trace JSON output to make it more easily consumable by the App. Added content for taint_source and intermediate_vars, and collapsed the multile taint_source locations into one. (dataflow-trace-json)

  • Removed the unique_id field from the semgrep (and semgrep-core) JSON output for metavariables. (unique-id-json)

Fixed

  • Fixed format of repository urls so links to findings can be properly displayed on semgrep.dev (cli-278)

  • Scala: Allow metavariables in import patterns (gh-5219)

  • Rules reported for LSP metrics now are hashed before sending (lsp-metrixs)

  • -filter_irrelevant_rules was incorrectly skipping files when the PCRE engine threw an error, while trying to match a regex that determines whether a rule is relevant for a file. This has been fixed so that, in case of a PCRE error, we assume that the rule could be relevant and we do run it on the file. (pa-1635)

0.106.0 - 2022-07-21

Changed

  • metavariable-comparison: The metavariable field is now optional, except if strip: true. When strip: false (the default) the metavaraible field has no use so it was pointless to require it. (metavariable-comparison-metavariable)

  • metavariable-comparison now also works on metavariables that cannot be evaluated to simple literals. In such cases, we take the string representation of the code bound by the metavariable. The way to access this string representation is via str($MVAR). For example:

    - metavariable-comparison:
    metavariable: $X
    comparison: str($X) == str($Y)

    Here $X and $Y may bind to two different code variables, and we check whether these two code variables have the same name (e.g. two different variables but both named x). (pa-1659)

  • When running an SCA scan with semgrep ci --sca, SCA findings will no longer be considered blocking if they are unreachable. (sca-128)

Fixed

  • Fixed a regression in name resolution that occurred with metavariable patterns (gh-5690)

  • Rust: Fixed a bug with matching for scoped identifiers

    Basically, scoped identifiers were only looking at the last identifier. So something like A::B::C would result in something like C. (gh-5717)

0.105.0 - 2022-07-20

Added

  • Experimental LSP support for: metavariable inlay hints, hot reloading, App integation, scan commands, and much much more (cli-235)

  • Added a concatenation reduction for extract mode, so that disjoint snippets within a file can be treated as one unified file. (extract-reductions)

  • Extract mode: taint trace positions are now updated to correspond to the source file. (extract-taint-traces)

  • Extract mode: generic is now permitted as a languages value (pa-1648)

  • Added handling for EXPERIMENT severity to allow the ability to test rules as experiments. (rules-680)

Changed

  • Do not store time or skipped targets output unless requested. This reduces memory consumption significantly in large repos (pa-1618)

  • Reduce memory consumption of semgrep by passing the targets in a more condensed structure. Previously, we told semgrep which rules to run on which target by listing out all the rule_ids each target should run. Now, we have a separate rule_id list and for each target we only ilst the rule_id indices. On large repos, particularly when run with multiple processes, this has a significant impact. (pa-1618-2)

Fixed

  • C#: Improved error message when function parameters are declared with var (gh-5068)

  • Scala/others: Added a fix allowing percolation of name information from class parameters

    For example, classes which take in arguments like the following in Scala:

    class ExampleClass(val x: TypeName) {
}

    do not properly enter the context. So in our analysis, we would not know that the identifier x has type TypeName, within the body of ExampleClass. (gh-5506)

  • Fixed the logged message describing the endpoint where rules are fetched from when SEMGREP_URL is set (gh-5753)

  • Fixed what data was used for indexing match results to used match based id data (index)

0.104.0 - 2022-07-13

Added

  • semgrep ci will now not block builds on triage ignored issues (cli-162)

  • Add support to cli/scripts/compare.py for podman environments (compare-script-podman)

  • A new experimental 'extract' mode. This mode runs a Semgrep rule on a codebase and "extracts" code from matches, treating it as a different language. This allows users to supplement an existing set of rules, e.g., for JavaScript, by writing additional rules to find JavaScript in non-JavaScript files, e.g., JavaScript contained in HTML or template files. While this is somewhat possible with metavariable-pattern, this reduces the work from an M * N problem to an M + N one. (gh-4478)

  • Added taint traces as part of Semgrep's JSON output. This helps explain how the sink became tainted. (pa-1271)

Changed

  • Metavariable-pattern now uses the same metavariable context as its parent. This will potentially cause breaking changes for rules that reuse metavariables in the pattern. For example, consider the following formula:

    - patterns:
   - pattern-either:
       - pattern-inside: $OBJ.output($RESP)
   - pattern: $RESP
   - metavariable-pattern:
       metavariable: $RESP
       pattern: `...{ $OBJ }...`

    Previously, the $OBJ in the metavariable-pattern would be a new metavariable. The formula would behave the same if that $OBJ was $A instead. Now, $OBJ will try to unify with the value bound by $OBJ in the pattern-inside. (gh-5060)

  • The semgrep test output used to produce expected lines and reported lines which is difficult to read and interpret. This change introduces missed lines and incorrect lines to make it easier for the users to pinpoint the differences in output. (gh-5600)

  • Separator lines are no longer drawn between findings that have no source code snippet. (sca-ui)

  • Using ellipses in XML/HTML elements is now more permissive of whitespace. Previously, in order to have a element with an ellipsis no leading/trailing whitespace was permitted in the element contents, i.e., <tag>...</tag> was the only permitted form. Now, leading or trailing whitespace is ignored when the substantive content of the element is only an ellipsis. (xml-permissive-ellipsis)

Fixed

  • Semgrep App's links to repositories, code snippets, and pull requests are no longer broken for projects that run Semgrep in Circle CI, Bitbucket, and Buildkite. (cli-267)

  • When running Semgrep via Docker for Mac, all volume-mounted files were considered executable and thus a candidate for scanning Bash code, making target collection take longer, and sometimes targeting more files than intended. This was fixed by using os.stat instead of os.access to determine if a file is executable. (gh-5560)

  • Semgrep used to crash when trying to print findings that match only whitespace, such as when a rule disallows two newlines at the end of a file. This crash is now fixed. (gh-5633)

  • Memory usage improvement: don't save skipped targets when --debug isn't passed since it isn't read unless --debug is used (pa-1618)

Infra/Release Changes

  • Updated the GH Release creation to publish the release instead of drafting one - this avoids a failure of validation jobs on release. (cli-243)

  • Using some building blocks from release, set up a workflow to test semgrep ci e2e. (cli-253)

  • Updated our changelog management system to towncrier to avoid merge conflicts in changelog on release (cli-77)

  • The opening of PRs in other repos is automated with semgrep release - there is still a manual review required to build confidence. (cli-84)

Unreleased

0.103.0 - 2022-07-05

Added

  • Expression statement patterns (e.g. foo();) used to also match when they were a bit deeper in the expression (e.g., x = foo();). This can now be disabled via rule options: with implicit_deep_exprstmt: false (#5472)

  • The timeout for git commands Semgrep runs is now configurable by setting the SEMGREP_GIT_COMMAND_TIMEOUT environment variable. The unit used is seconds. The default value is 300.

Fixed

  • Error messages in JSON output used to contain ANSI color codes, this was fixed so they are now uncolored (and readable!) plain text.

  • Fixed config parsing bug to cut CI scan speeds down significantly

0.102.0 - 2022-06-30

Added

  • Scala: ellipsis are now allowed in for loop headers, so you can write patterns like for (...; $X <- $Y if $COND; ...) { ... } to match nested for loops. (#5650)

  • The SEMGREP_GHA_MIN_FETCH_DEPTH environment variable which lets you set how many commits semgrep ci fetches from the remote at the minimum when calculating the merge-base in GitHub Actions. Having more commits available helps Semgrep determine what changes came from the current pull request, fixing issues where Semgrep would report findings that weren't touched in a given pull request. This value is set to 0 by default (#5664)

Fixed

  • taint-mode: In some scenarios some statements were not being included in the CFG used by taint tracking, and as a result some expected findings were not being reported (i.e. false negatives). This affected mainly languages like Scala where traditional control-flow constructs are expressions rather than statements (or, seen in a different way, every statement returns a value). (#5652)

  • Yaml: location information is fixed for unicode characters (#5660)

Changed

  • --verbose no longer toggles the display of timing information, use --verbose --time to display this information.

  • Change timeout for git operations from 100s to 500s

0.101.1 - 2022-06-28

Fixed

  • semgrep ci: CI runs in GitHub Actions failed to checkout the commit assoociated with the head branch, and is fixed here.

0.101.0 - 2022-06-27

Added

  • Bash: Support for subshell syntax i.e. commands in parentheses (#5629)

Changed

Fixed

  • semgrep ci: CI runs were failing to checkout the PR head in GitHub Actions, which is corrected here.

  • TS: fixed the parsing of type predicates and typeof queries

  • Deep expression matching now works on HTML in JavaScript

  • taint-mode: Taint propagation via pattern-propagators now works correclty when the from or to metavariables match a function call. For example, given sqlBuilder.append(page.getOrderBy()), we can now propagate taint from page.getOrderBy() to sqlBuilder.

  • Will no longer print "files were not tracked by git" if not in a git repo

  • Will no longer print "Some files were skipped" if no files were skipped

  • Fixed bug where semgrep would crash in nonexistent directory (#4785)

  • taint-mode: Correctly propagate taint in for-each loops with typed iteration variables (as in Java or C#). If the iterator object is tainted, that taint will now be propagated to the iteration variable. This should fix some false negatives (i.e., findings not being reported) in the presence of for-each loops. (#5590)

0.100.0 - 2022-06-22

Added

  • taint-mode: New experimental pattern-propagators feature that allows to specify arbitrary patterns for the propagation of taint by side-effect. In particular, this allows to specify how taint propagates through side-effectful function calls. For example, you can specify that when tainted data is added to an array then the array itself becomes tainted. (#4509)

Changed

  • --config auto no longer sends the name of the repository being scanned to the Semgrep Registry. As of June 21st, this data is not recorded by the Semgrep Registry backend, even if an old Semgrep version sends it. Also as of June 21st, none of the previously collected repository names are retained by the Semgrep team; any historical data has been wiped.

  • Gitlab SAST output is now v14.1.2 compliant

  • Removed the following deprecated semgrep scan options: --json-stats, --json-time, --debugging-json, --save-test-output-tar, --synthesize-patterns, --generate-config/-g, --dangerously-allow-arbitrary-code-execution-from-rules, and --apply (which was an easter egg for job applications, not the same as --autofix)

  • PHP: switch to GA maturity! Thanks a lot to Sjoerd Langkemper for most of the heavy work

Fixed

  • Inline join mode rules can now run taint-mode rules

  • Python: correctly handle with context expressions where the value is not bound (#5513)

  • Solidity: update to a more recent tree-sitter-solidity to fix certain parsing errors (#4957)

0.99.0 - Skipped

Version 0.99.0 of Semgrep was intentionally skipped. Version 0.100.0 immediately follows version 0.98.0.

0.98.0 - 2022-06-15

Added

  • New language R with experimental support (#2360) Thanks to Zythosec for some contributions.

  • Autodetection of CI env now supports Azure Pipelines, Bitbucket, Buildkite, Circle CI, Jenkins, and Travis CI in addition to GitHub and GitLab

  • You can now disable version checks with an environment variable by setting SEMGREP_ENABLE_VERSION_CHECK=0

  • Dataflow: spread operators in record expressions (e.g. {...foo}) are now translated into the Dataflow IL

  • An experimental LSP daemon mode for semgrep. Try it with semgrep lsp --config auto!

Changed

  • Rules are now downloaded from the Semgrep Registry in JSON format instead of YAML. This speeds up rule parsing in the Semgrep CLI, making a semgrep --config auto run on the semgrep Python package in 14s instead of 16s.

Fixed

  • Fixed a bug where --disable-version-check would still send a request when a scan resulted in zero findings.

  • Fixed a regression in 0.97 where the Docker image's working directory changed from /src without notice. This also could cause permission issues when running the image.

  • Go: single pattern field can now match toplevel fields in a composite literal (#5452)

  • PHP: metavariable-pattern: works again when used with language: php (#5443)

  • PHP: booleans are propagated by constant propagation (#5509)

  • PHP: named arguments work in patterns (#5508)

  • Fixed a non-deterministic crash when matching a large number of regexes (#5277)

  • Fixed issue when running in GithubActions that caused semgrep to report on files not changed in the PR (#5453)

  • JS/TS: $X() no longer matches new Foo(), for consistency with other languages (#5510)

  • JS/TS: Typed metavariables now match constructor calls (e.g. ($X: C) matches new C(). (#5540)

0.97.0 - 2022-06-08

Added

  • Dataflow: XML elements (e.g. JSX elements) have now a basic translation to the Dataflow IL, meaning that dataflow analysis (constant propagation, taint tracking) can now operate inside these elements (#5115)

  • Java: you can now use a metavariable in a package directive (#5420), for example, package $X, which is useful to bind the package name and use it in the error message.

Fixed

  • The output of semgrep ci should be clear it is exiting with error code 0 when there are findings but none of them being blockers

  • Java: support for Sealed classes and Text Blocks via tree-sitter-java (#3787, #4644)

  • The JUnit XML output should serialize the failure messages as a single string instead of a python list of strings.

  • Typescript: update to latest tree-sitter-typescript, with support for 'abstract' modifier in more places

  • Scala: stop parsing parenthesized expressions as unary tuples

  • yarn.lock files with no depenencies, and with dependencies that lack URLs, now parse

  • Scala: fixed bug where typed patterns inside classes caused an exception during name resolution

  • metavariable-regex: patterns are now unanchored as specified by the documentation (#4807)

  • When a logged in CI scan encounters a Git failure, we now print a helpful error message instead of a traceback.

0.96.0 - 2022-06-03

Added

  • Generic mode: new option generic_ellipsis_max_span for controlling how many lines an ellipsis can match (#5211)

  • Generic mode: new option generic_comment_style for ignoring comments that follow the specified syntax (C style, C++ style, or Shell style) (#3428)

  • Metrics now include a list of features used during an execution. Examples of such features are: languages scanned, CLI options passed, keys used in rules, or certain code paths reached, such as using an :include instruction in a .semgrepignore file. These strings will NOT include user data or specific settings. As an example, with semgrep scan --output=secret.txt we might send "option/output" but will NOT send "option/output=secret.txt".

Changed

  • The output summarizing a scan's results has been simplified.

0.95.0 - 2022-06-02

Added

  • Sarif output format now includes fixes section

  • --test flag will now seach for code files with .fixed suffix and use these to test the behaviour of autofixes of the rules.

  • Rust: added support for method chaining patterns.

  • r2c-internal-project-depends-on: support for poetry and gradle lockfiles

  • M1 Mac support added to PyPi

  • Accept SEMGREP_BASELINE_REF as alias for SEMGREP_BASELINE_COMMIT

  • r2c-internal-project-depends-on:

    • pretty printing for SCA results

    • support for poetry and gradle lockfiles

  • taint-mode: Taint tracking will now analyze lambdas in their surrounding context. Previously, if a variable became tainted outside a lambda, and this variable was used inside the lambda causing the taint to reach a sink, this was not being detected because any nested lambdas were "opaque" to the analysis. (Taint tracking looked at lambdas but as isolated functions.) Now lambas are simply analyzed as if they were statement blocks. However, taint tracking still does not follow the flow of taint through the lambda's arguments!

  • Metrics now include an anonymous Event ID. This is an ID generated at send-time and will be used to de-duplicate events that potentially get duplicated during transmission.

  • Metrics now include an anonymous User ID. This ID is stored in the ~/.semgrep/settings.yml file. If the ID disappears, the next run will generate a new one randomly. See the Anonymous User ID in PRIVACY.md for more details.

Fixed

  • M1 Mac installed via pip now links tree-sitter properly

  • Restore --sca

Changed

  • The ci CLI command will now include ignored matches in output formats that dictate they should always be included

  • Previously, you could use $X in a message to interpolate the variable captured by a metavariable named $X, but there was no way to access the underlying value. However, sometimes that value is more important than the captured variable. Now you can use the syntax value($X) to interpolate the underlying propagated value if it exists (if not, it will just use the variable name).

    Example:

    Take a target file that looks like

    x = 42
log(x)

    Now take a rule to find that log command:

    - id: example_log
  message: Logged $SECRET: value($SECRET)
  pattern: log(42)
  languages: [python]

    Before, this would have given you the message Logged x: value(x). Now, it will give the message Logged x: 42.

  • A parameter pattern without a default value can now match a parameter with a default value (#5021)

Fixed

  • Numerous improvements to PHP parsing by switching to tree-sitter-php to parse PHP target code. Huge shoutout to Sjoerd Langkemper for most of the heavy lifting work (#3941, #2648, #2650, #3590, #3588, #3587, #3576, #3848, #3978, #4589)

  • TS: support number and boolean typed metavariables (#5350)

  • When a rule from the registry fails to parse, suggest user upgrade to latest version of semgrep

  • Scala: correctly handle return for taint analysis (#4975)

  • PHP: correctly handle namespace use declarations when they don't rename the imported name (#3964)

  • Constant propagation is now faster and memory efficient when analyzing large functions with lots of variables.

0.94.0 - 2022-05-25

Added

  • metavariable-regex now supports an optional constant-propagation key. When this is set to true, information learned from constant propagation will be used when matching the metavariable against the regex. By default it is set to false

  • Dockerfile: constant propagation now works on variables declared with ENV

  • shouldafound - False Negative reporting via the CLI

Changed

  • taint-mode: Let's say that e.g. taint(x) makes x tainted by side-effect. Previously, we had to rely on a trick that declared that any occurrence of x inside taint(x); ... was as taint source. If x was overwritten with safe data, this was not recognized by the taint engine. Also, if taint(x) occurred inside e.g. an if block, any occurrence of x outside that block was not considered tainted. Now, if you specify that the code variable itself is a taint source (using focus-metavariable), the taint engine will handle this as expected, and it will not suffer from the aforementioned limitations. We believe that this change should not break existing taint rules, but please report any regressions that you may find.

  • taint-mode: Let's say that e.g. sanitize(x) sanitizes x by side-effect. Previously, we had to rely on a trick that declared that any occurrence of x inside sanitize(x); ... was sanitized. If x later overwritten with tainted data, the taint engine would still regard x as safe. Now, if you specify that the code variable itself is sanitized (using focus-metavariable), the taint engine will handle this as expected and it will not suffer from such limitation. We believe that this change should not break existing taint rules, but please report any regressions that you may find.

  • The dot access ellipsis now matches field accesses in addition to method calls.

  • pattern-regex, pattern-not-regex, metavariable-regex: ^ and $ now match at the beginning and end of each line, respectively, rather than previously just at the beginning and end of the input file. This corresponds to PCRE's multiline mode. To get the old behavior back, use \A instead of '^' and \Z instead of $. See the PCRE manual for details.

  • Made error message for resource exhausion (exit code -11/-9) more actionable

  • Made error message for rules with patterns missing positive terms more actionable (#5234)

  • In this version, we have made several performance improvements to the code that surrounds our source parsing and matching core. This includes file targeting, rule fetching, and similar parts of the codebase. Running semgrep scan --config auto on the semgrep repo itself went from 50-54 seconds to 28-30 seconds.

    • As part of these changes, we removed :include .gitignore and .git/ from the default .semgrepignore patterns. This should not cause any difference in which files are targeted as other parts of Semgrep ignore these files already.

    • A full breakdown of our performance updates, including some upcoming ones, can be found here https://github.com/returntocorp/semgrep/issues/5257#issuecomment-1133395694

  • If a metrics event request times out, we no longer retry the request. This avoids Semgrep waiting 10-20 seconds before exiting if these requests are slow.

  • The metrics collection timeout has been raised from 2 seconds to 3 seconds.

Fixed

  • TS: support for template literal types after upgrading to a more recent tree-sitter-typescript (Oct 2021)

  • TS: support for override keyword (#4220, #4798)

  • TS: better ASI (#4459) and accept code like (null)(foo) (#4468)

  • TS: parse correctly private properties (#5162)

  • Go: Support for ellipsis in multiple return values (e.g., func foo() (..., error, ...) {}) (#4896)

  • semgrep-core: you can use again rules stored in JSON instead of YAML (#5268)

  • Python: adds support for parentheses around with context expressions (e.g., with (open(x) as a, open(y) as b): pass) (#5092)

  • C++: we now parse correctly const declarations (#5300)

0.93.0 - 2022-05-17

Changed

  • Files where only some part of the code had to be skipped due to a parse failure will now be listed as "partially scanned" in the end-of-scan skip report.

  • Licensing: The ocaml-tree-sitter-core component is now distributed under the terms of the LGPL 2.1, rather than previously GPL 3.

  • A new field was added to metrics collection: isAuthenticated. This is a boolean flag which is true if you ran semgrep login.

Fixed

  • semgrep ci used to incorrectly report the base branch as a CI job's branch when running on a pull_request_target event in GitHub Actions. By fixing this, Semgrep App can now track issue status history with on: pull_request_target jobs.

  • Metrics events were missing timestamps even though PRIVACY.md had already documented a timestamp field.

0.92.1 - 2022-05-13

Added

  • Datafow: The dataflow engine now handles if-then-else expressions as in OCaml, Ruby, etc. Previously it only handled if-then-else statements. (#4965)

Fixed

  • Kotlin: support for ellispis in class parameters, e.g.. class Foo(...) {} (#5180)

  • JS/TS: allow ellipsis in binding_pattern (e.g., in arrow parameters) (#5230)

  • JS/TS: allow ellipsis in imports (e.g., import {..., Foo, ...} from 'Bar') (#5012)

  • fixed_lines is once again included in JSON output when running with --autofix --dryrun

0.92.0 - 2022-05-11

Added

  • The JSON output of semgrep scan is now fully specified using ATD (https://atd.readthedocs.io/) and jsonschema (https://json-schema.org/). See the semgrep-interfaces submodule under interfaces/ (e.g., interfaces/semgrep-interfaces/Semgrep_output_v0.atd for the ATD spec)

  • The JSON output of semgrep scan now contains a "version": field with the version of Semgrep used to generate the match results.

  • taint-mode: Previously, to declare a function parameteter as a taint source, we had to rely on a trick that declared that any occurence of the parameter was a taint source. If the parameter was overwritten with safe data, this was not recognized by the taint engine. Now, focus-metavariable can be used to precisely specify that a function parameter is a source of taint, and the taint engine will handle this as expected.

  • taint-mode: Add basic support for object destructuring in languages such as Javascript. For example, given let {x} = E, Semgrep will now infer that x is tainted if E is tainted.

Fixed

  • OCaml: Parenthesis in autofixed code will no longer leave dangling closing-paren. Thanks to Elliott Cable for his contribution (#5087)

  • When running the Semgrep Docker image, we now mark all directories as safe for use by Git, which prevents a crash when the current user does not own the source code directory.

  • C++: Ellipsis are now allowed in for loop header (#5164)

  • Java: typed metavariables now leverages the type of foreach variables (#5181)

0.91.0 - 2022-05-03

Added

  • --core-opts flag to send options to semgrep-core. For internal use: no guarantees made for semgrep-core options (#5111)

Changed

  • semgrep ci prints out all findings instead of hiding nonblocking findings (#5116)

0.90.0 - 2022-04-26

Added

  • Users can access the propagated value of a metavariable in the JSON output in the extra field

  • Join mode now supports inline rules via the rules: key underneath the join: key.

  • Added vendor.name field in gitlab sast output (#5077)

Changed

  • YAML parsing is more tolerant of {} appearing when it expects a scalar, allowing extensions of YAML that use {} to be parsed (#4849)

  • Turn off optimization that trades off memory for performance because the effect is minor (with current parameters)

Fixed

  • Keep only latest run logs in last.log file (#5070)

  • r2c-internal-project-depends-on:

    • Lockfiles that fail to parse will not crash semgrep

    • cargo.lock and Pipfile.lock dependencies that don't specify hashes now parse

    • go.sum files with a trailing newline now parse

0.89.0 - 2022-04-20

Added

  • Bash/Dockerfile: Add support for named ellipses such as in echo $...ARGS (#4887)

  • PHP: Constant propagation for static constants (#5022)

Changed

  • When running a baseline scan on a shallow-cloned git repository, Semgrep still needs enough git history available to reach the branch-off point between the baseline and current branch. Previously, Semgrep would try to gradually fetch more and more commits up to a thousand commits of history, before giving up and just fetching all commits from the remote git server. Now, Semgrep will keep trying smaller batches until up to a million commits. This change should reduce runtimes on large baseline scans on very large repositories.

  • Semgrep-core now logs the rule and file affected by a memory warning.

  • Improved error messages from semgrep-core (#5013)

  • Small changes to text output (#5008)

  • Various exit codes changed so that exit code 1 is only for blocking findings (#5039)

  • Subcommand is sent as part of user agent (#5051)

Fixed

  • Lockfiles scanning now respects .semgrepignore

  • Workaround for git safe.directory change in github action (#5044)

  • When a baseline scan diff showed that a path changed a symlink a proper file, Semgrep used incorrectly skip that path. This is now fixed.

  • Dockerfile support: handle image aliases correctly (#4881)

  • TS: Fixed matching of parameters with type annotations. E.g., it is now possible to match ({ params }: Request) => { } with ({$VAR} : $REQ) => {...}. (#5004)

0.88.0 - 2022-04-13

Added

  • Scala support is now officially GA

    • Ellipsis method chaining is now supported

    • Type metavariables are now supported

  • Ruby: Add basic support for lambdas in patterns. You can now write patterns of the form -> (P) {Q} where P and Q are sub-patterns. (#4950)

  • Experimental semgrep install-deep-semgrep command for DeepSemgrep beta (#4993)

Changed

  • Moved description of parse/internal errors to the "skipped" section of output

  • Since 0.77.0 semgrep-core logs a warning when a worker process is consuming above 400 MiB of memory. Now, it will also log an extra warning every time memory usage doubles. Again, this is meant to help diagnosing OOM-related crashes.

Fixed

  • Dockerfile: lang.json file not found error while building the docker image

  • Dockerfile: EXPOSE 12345 will now parse 12345 as an int instead of a string, allowing metavariable-comparison with integers (#4875)

  • Scala: unicode character literals now parse

  • Scala: multiple annotated type parameters now parse (def f[@an A, @an B](x : A, y : B) = ...)

  • Ruby: Allow 'unless' used as keyword argument or hash key (#4948)

  • Ruby: Fix regexp matching in the presence of escape characters (#4999)

  • r2c-internal-project-depends-on:

    • Generic mode rules work again

    • Semgrep will not fail on targets that contain no relevant lockfiles

    • package-lock.json parsing now defaults to dependencies instead of packages, and will not completely fail on dependencies with no version

    • yarn.lock parsing has been rewritten to fix a bug where sometimes large numbers of dependencies would be ignored

  • Go: parse multiline string literals

  • Handle utf-8 decoding errors without crashing (#5023)

0.87.0 - 2022-04-07

Added

  • New focus-metavariable operator that lets you focus (or "zoom in") the match on the code region delimited by a metavariable. This operator is useful for narrowing down the code matched by a rule, to focus on what really matters. (#4453)

  • semgrep ci uses "GITHUB_SERVER_URL" to generate urls if it is available

  • You can now set NO_COLOR=1 to force-disable colored output

Changed

  • taint-mode: We no longer force the unification of metavariables between sources and sinks by default. It is not clear that this is the most natural behavior; and we realized that, in fact, it was confusing even for experienced Semgrep users. Instead, each set of metavariables is now considered independent. The metavariables available to the rule message are all metavariables bound by pattern-sinks, plus the subset of metavariables bound by pattern-sources that do not collide with the ones bound by pattern-sinks. We do not expect this change to break many taint rules because source-sink metavariable unification had a bug (see #4464) that prevented metavariables bound by a pattern-inside to be unified, thus limiting the usefulness of the feature. Nonetheless, it is still possible to force metavariable unification by setting taint_unify_mvars: true in the rule's options.

  • r2c-internal-project-depends-on: this is now a rule key, and not part of the pattern language. The depends-on-either key can be used analgously to pattern-either

  • r2c-internal-project-depends-on: each rule with this key will now distinguish between reachable and unreachable findings. A reachable finding is one with both a dependency match and a pattern match: a vulnerable dependency was found and the vulnerable part of the dependency (according to the patterns in the rule) is used somewhere in code. An unreachable finding is one with only a dependency match. Reachable findings are reported as coming from the code that was pattern matched. Unreachable findings are reported as coming from the lockfile that was dependency matched. Both kinds of findings specify their kind, along with all matched dependencies, in the extra field of semgrep's JSON output, using the dependency_match_only and dependency_matches fields, respectively.

  • r2c-internal-project-depends-on: a finding will only be considered reachable if the file containing the pattern match actually depends on the dependencies in the lockfile containing the dependency match. A file depends on a lockfile if it is the nearest lockfile going up the directory tree.

  • The returntocorp/semgrep Docker image no longer sets semgrep as the entrypoint. This means that semgrep is no longer prepended automatically to any command you run in the image. This makes it possible to use the image in CI executors that run provisioning commands within the image.

Fixed

  • - is now parsed as a valid identifier in Scala

  • new $OBJECT(...) will now work properly as a taint sink (#4858)

  • JS/TS: ...{$X}... will no longer match str

  • taint-mode: Metavariables bound by a pattern-inside are now available to the rule message. (#4464)

  • parsing: fail fast on in semgrep-core if rules fail to validate (broken since 0.86.5)

  • Setting either SEMGREP_URL or SEMGREP_APP_URL now updates the URL used both for Semgrep App communication, and for fetching Semgrep Registry rules.

  • The pre-commit hook exposed from semgrep's repository no longer fails when trying to install with recent setuptools versions.

0.86.5 - 2022-03-28

Changed

  • Set minimum urllib3 version

0.86.4 - 2022-03-25

Changed

  • Increase rule fetch timeout from 20s to 30s

0.86.3 - 2022-03-25

Fixed

  • Network timeouts during rule download are now less likely.

0.86.2 - 2022-03-24

Fixed

  • Some finding fingerprints were not matching what semgrep-agent would return.

0.86.1 - 2022-03-24

Fixed

  • The fingerprint of findings ignored with # nosemgrep is supposed to be the same as if the ignore comment wasn't there. This has previously only worked for single-line findings, including in semgrep-agent. Now the fingerprint is consistent as expected for multiline findings as well.

Changed

  • --timeout-threshold default set to 3 instead of 0

0.86.0 - 2022-03-24

Added

  • Semgrep can now output findings in GitLab's SAST report and secret scanning report formats with --gitlab-sast and --gitlab-secrets.

  • JSON output now includes a fingerprint of each finding. This fingerprint remains consistent when matching code is just moved around or reindented.

  • Go: use latest tree-sitter-go with support for Go 1.18 generics (#4823)

  • Terraform: basic support for constant propagation of locals (#1147) and variables (#4816)

  • HTML: you can now use metavariable ellipsis inside